GCP Capability Parity
Status: STABLE_SUPPORTED_SURFACE
Adapter target: contractforge-gcp / gcp_bigquery
Reference maturity target: Databricks adapter stable behavior, then AWS/Snowflake/Fabric parity gates where applicable.
Purpose
This document records what the GCP adapter can claim today, what Google Cloud can natively support, and what remains outside the scoped stable-supported surface.
The central rule is strict: a Google Cloud service having a feature is not enough. ContractForge marks a capability as stable only after the adapter preserves the contract semantics and records comparable evidence.
Research Basis
This parity pass used:
- local adapter code:
adapters/gcp/src/contractforge_gcp; - local parity references:
docs/specs/platform-contract-parity.md,docs/specs/evidence-mapping-matrix.md,docs/platform-parity-tests.md; - Context7 BigQuery documentation for load jobs, MERGE/DML, job statistics and security metadata;
- Context7 Google Cloud SDK / Secret Manager documentation for
gcloud secrets versions accesscommand planning; - official Google Cloud documentation linked in the source list.
Status Legend
| Status | Meaning |
|---|---|
IMPLEMENTED | The adapter renders or plans this behavior today. |
PLATFORM_NATIVE | Google Cloud has a relevant native primitive, but ContractForge has not fully mapped it. |
REVIEW_REQUIRED | Equivalence depends on design choices, permissions, data layout, runtime or test evidence. |
UNSUPPORTED | The adapter must block or avoid the behavior for now. |
E2E_REQUIRED | Code may exist, but real-account bronze-to-gold execution and evidence are still required. |
Current Adapter Claim
The current GCP adapter is stable-final for the documented BigQuery batch surface.
| Area | Current claim | Reason |
|---|---|---|
| Planning | IMPLEMENTED | plan_gcp_contract normalizes contracts and runs the core planner against gcp_bigquery capabilities. |
| Rendering | IMPLEMENTED | render_gcp_contract emits BigQuery review, capability, source-support, redacted source-review JSON/Markdown with non-JDBC source-family promotion paths, source-family promotion-plan JSON, SQL, load-job, public REST/HTTP source materialization, authenticated REST/HTTP Secret Manager review/runtime resolution artifacts, schema-policy planning, advanced write-mode review, Dataplex data-quality create plus execution/readback planning, governance ledger and reconciliation planning, evidence DDL, deployment manifest and bundle manifest artifacts. Project deployment dry-runs also emit a Google Workflows source plan, execution-plan artifact, evidence-readback artifact, bounded BigQuery job polling, connector retry planning and optional readback command metadata for review. |
| Runtime execution | IMPLEMENTED bronze-to-gold smoke | Real GCS, SQL and MERGE BigQuery smokes have been captured, including a compact bronze-to-gold contract set. The adapter also exposes sequential project smoke execution through the same contract-only runtime. |
| Evidence writes | IMPLEMENTED for smoke paths | The smoke workflows persist run, quality, annotation, neutral OpenLineage and governance-ledger evidence from BigQuery jobs; failed-run evidence is validated. Governance evidence write/readback is validated for declared governance intent, and non-mutating governance reconciliation readback is included through the adapter command surface. |
| Row access policy enforcement | IMPLEMENTED for BigQuery row access smoke | The live smoke applies a policy, reads it back and verifies restricted-principal enforcement. |
| Direct column data masking | IMPLEMENTED for BigQuery V2 data policy smoke | The live smoke creates a direct column data policy, attaches it to a column and verifies masked reads through a restricted service account. |
| Policy-tag column access | IMPLEMENTED for BigQuery policy-tag smoke | The live smoke creates a regional Data Catalog taxonomy/policy tag, attaches it to a column, verifies denied access before grant and allowed access after categoryFineGrainedReader. |
| Annotations | IMPLEMENTED for BigQuery description smoke | The adapter renders table/column description DDL and annotation evidence SQL; the live smoke applies/read backs descriptions through INFORMATION_SCHEMA and persists annotation evidence rows. Tags/aspects remain review-required. |
| Failed run evidence | IMPLEMENTED for live negative smoke | A SQL-source contract with a missing table fails natively and persists a FAILED run evidence row with the BigQuery error message. |
| Data Catalog policy tags | IMPLEMENTED for column access | Policy-tag column access passed. Policy-tag-backed masking remains outside first stable scope. |
| Stable-final status | STABLE_SUPPORTED_SURFACE | gcp_stabilization_report() returns stable_final: true for the documented BigQuery batch surface with explicit exclusions. |
The first live smoke is recorded in docs/reports/gcp-bigquery-csv-smoke.json. It validates one small GCS CSV load into BigQuery, persists run and quality evidence rows, and runs a not-null quality query.
The file-format smoke is recorded in docs/reports/gcp-bigquery-file-formats-smoke.json. It validates CSV, NDJSON, Parquet, Avro and ORC GCS load contracts with three output rows each, zero not-null failures and persisted run/quality evidence rows. These close the GCS load-format slice, not bronze-to-gold parity.
The upsert smoke is recorded in docs/reports/gcp-bigquery-upsert-smoke.json. It validates explicit-column BigQuery MERGE rendering and execution with one updated row, one inserted row, three final target rows, zero not-null failures and persisted run/quality evidence rows.
The bronze-to-gold smoke is recorded in docs/reports/gcp-bigquery-bronze-to-gold-smoke.json. It validates an ordered GCP contract set: GCS bronze load, SQL silver curation and SQL gold aggregation, with row-count checks, quality checks and run evidence rows for every layer.
The row access policy smoke is recorded in docs/reports/gcp-bigquery-row-access-policy-smoke.json. It validates BigQuery row access policy apply/readback and restricted-principal enforcement on the silver table from the bronze-to-gold smoke.
The error evidence smoke is recorded in docs/reports/gcp-bigquery-error-evidence-smoke.json. It validates that a failed SQL-source contract writes a failed run evidence row and preserves the native BigQuery error message.
The direct column data masking smoke is recorded in docs/reports/gcp-bigquery-data-masking-smoke.json. It validates BigQuery V2 data policy creation, column attachment and restricted-principal masked read enforcement. The earlier blocker in docs/reports/gcp-bigquery-data-masking-blocker.json remains as historical evidence for a non-organization project.
The policy-tag smoke is recorded in docs/reports/gcp-bigquery-policy-tags-smoke.json. It validates Data Catalog taxonomy/policy tag creation, BigQuery schema attachment, INFORMATION_SCHEMA.COLUMN_FIELD_PATHS readback, denied protected-column reads before fine-grained access is granted and successful protected-column reads after roles/datacatalog.categoryFineGrainedReader is granted. It also records that policy tags must be regional-compatible with the BigQuery dataset.
The BigLake Iceberg smoke is recorded in docs/reports/gcp-biglake-iceberg-smoke.json. It validates a BigLake managed Iceberg table backed by Cloud Storage, created through a BigQuery Cloud Resource connection, with append, MERGE, query readback, biglakeConfiguration readback and observed Iceberg metadata/ plus Parquet data/ objects. The raw Iceberg registration smoke is recorded in docs/reports/gcp-raw-iceberg-registration-smoke.json; it validates contractforge-gcp source-promotion --execute --readback for a raw gs:// Iceberg path with explicit schema, BigLake registration and metadata/query readback. The stable source claim remains registered BigQuery/BigLake Iceberg table reads; direct raw-path query execution without registration remains excluded.
The authenticated REST Secret Manager smoke is recorded in docs/reports/gcp-authenticated-rest-secret-manager-smoke.json. It validates a normal contract with a {{ secret:scope/key }} password placeholder, resolves the value from Google Secret Manager only at smoke runtime, reads the protected REST endpoint through the shared core reader and loads the materialized records into BigQuery. Generated reports do not contain the secret value. The authenticated REST/HTTP variants smoke is recorded in docs/reports/gcp-auth-rest-http-secret-manager-variants-smoke.json; it validates contract-only rest_api bearer-token, rest_api API-key, http_json bearer-token and http_json API-key sources, all resolved from Google Secret Manager at runtime and loaded into BigQuery with run, lineage and quality evidence. The earlier blocker in docs/reports/gcp-auth-rest-http-secret-manager-variants-blocker.json is historical and superseded.
The public HTTP sources BigQuery smoke is recorded in docs/reports/gcp-http-sources-bigquery-smoke.json. It validates a contract-only GCP project with http_text, generic http_file format=text, generic http_file format=json and generic http_file format=csv, all through the shared core HTTP reader, temporary local materialization, BigQuery local load jobs and run/lineage/quality evidence where declared. Target readback after the run showed 195 text rows, 195 generic text rows, 1 JSON row and 194 CSV rows. The HTTP binary file smoke is recorded in docs/reports/gcp-http-file-binary-bigquery-smoke.json; it validates declared http_file Avro, ORC and Parquet pass-through loads against BigQuery with three rows per target, quality evidence and lineage evidence. The previous http_text BigQuery blocker in docs/reports/gcp-http-text-bigquery-smoke-blocker.json is now superseded.
The streaming scope decision is recorded in docs/reports/gcp-streaming-scope-decision.json. Kafka/Event Hubs bounded and available-now sources, Dataflow Kafka runtime and direct Pub/Sub BigQuery subscriptions remain outside the first stable GCP default surface. Pub/Sub BigQuery subscriptions are native, but they do not prove ContractForge kafka_available_now semantics. Review-required Kafka/Event Hubs contracts emit a deterministic Dataflow Kafka-to-BigQuery promotion plan with template parameters, Secret Manager auth hooks, consumer group, checkpoint location, offset evidence requirements, streaming launch options and non-claims for continuous streaming. The adapter-owned source-promotion --execute --readback path can launch that Dataflow template and read back Dataflow job plus BigQuery target and DLQ metadata. A live Confluent smoke reached JOB_TYPE_STREAMING from a ContractForge kafka_available_now contract, wrote 12 target rows including a live 3-row batch, kept DLQ rows at 0 and proved no-input replay with the same consumer group/checkpoint by keeping the target count unchanged at 12; see docs/reports/gcp-confluent-kafka-dataflow-source-promotion-smoke.json. This closes GCP-BQ-13A for the Confluent/Dataflow provider path; broader Pub/Sub/Event Hubs and production streaming operations remain review-scoped.
The write-mode scope decision is recorded in docs/reports/gcp-write-mode-scope-decision.json. The first stable GCP surface is limited to append, overwrite and explicit-column upsert. hash_diff_upsert, historical and snapshot_reconcile_soft_delete remain review-required by default, but the hash-diff production-parity decision is now accepted using GCP, AWS, Snowflake and Fabric evidence. Historical and snapshot modes still need cross-adapter production-sized parity beyond the recorded GCP/Fabric evidence before stable promotion. Rendered bundles include deterministic advanced write-mode review artifacts with candidate BigQuery SQL, blockers and promotion evidence requirements, but those artifacts are not default executable stable runtime claims.
Schema-policy planning is in scope for the first stable surface. Rendered bundles include deterministic BigQuery preflight queries, apply hints and contractforge_schema_evidence DDL for strict, additive_only and permissive policies. The smoke runtime now has an opt-in --enforce-schema-policy hook for BigQuery table/view/SQL sources and declared-schema GCS load sources: it reads source and target INFORMATION_SCHEMA.COLUMNS, applies additive nullable columns for additive_only and permissive, blocks strict drift and writes schema evidence. The additive nullable path passed live BigQuery validation in docs/reports/gcp-bigquery-schema-policy-smoke.json; the strict negative path passed live validation in docs/reports/gcp-bigquery-schema-policy-strict-smoke.json; the permissive nullable path passed live validation in docs/reports/gcp-bigquery-schema-policy-permissive-smoke.json; the destructive type-change negative path passed live validation in docs/reports/gcp-bigquery-schema-policy-type-change-smoke.json; the SQL-source path passed live validation in docs/reports/gcp-bigquery-schema-policy-sql-source-smoke.json; the GCS/load-source path passed live validation in docs/reports/gcp-bigquery-schema-policy-gcs-source-smoke.json. The type-mutation decision is recorded in docs/reports/gcp-schema-policy-type-mutation-decision.json: automatic BigQuery type widening or mutation remains review-required and blocked by the stable runtime path.
The schema-policy smoke is recorded in docs/reports/gcp-bigquery-schema-policy-smoke.json. It validates source and target INFORMATION_SCHEMA.COLUMNS inspection, ALTER TABLE ADD COLUMN amount FLOAT64, append after schema synchronization, target schema readback and schema evidence readback.
The strict schema-policy smoke is recorded in docs/reports/gcp-bigquery-schema-policy-strict-smoke.json. It validates source and target INFORMATION_SCHEMA.COLUMNS inspection, blocks an additive drift under schema_policy: strict before write execution and reads back a FAILED schema evidence row with the native strict violation message.
The permissive schema-policy smoke is recorded in docs/reports/gcp-bigquery-schema-policy-permissive-smoke.json. It validates source and target INFORMATION_SCHEMA.COLUMNS inspection, ALTER TABLE ADD COLUMN amount FLOAT64, append after schema synchronization and successful schema evidence readback under schema_policy: permissive.
The type-change schema-policy smoke is recorded in docs/reports/gcp-bigquery-schema-policy-type-change-smoke.json. It validates source and target INFORMATION_SCHEMA.COLUMNS inspection, blocks an incompatible amount STRING to amount FLOAT64 drift under schema_policy: permissive, keeps the target unwritten and reads back failed type_changes_json evidence.
The SQL-source schema-policy smoke is recorded in docs/reports/gcp-bigquery-schema-policy-sql-source-smoke.json. It validates SQL source schema inspection through a zero-row evidence-dataset probe table, probe cleanup, nullable column application, append after schema synchronization and successful schema evidence readback.
The GCS-source schema-policy smoke is recorded in docs/reports/gcp-bigquery-schema-policy-gcs-source-smoke.json. It validates declared source columns on a GCS CSV load contract, schema inspection through an evidence-dataset load probe, probe cleanup, nullable column application, append load after schema synchronization and successful schema evidence readback.
The schema-policy type-mutation decision is recorded in docs/reports/gcp-schema-policy-type-mutation-decision.json. It promotes the validated GCP schema-policy runtime surface while keeping automatic type widening or type mutation outside the stable path. Type changes are recorded and blocked with schema evidence until a future reviewed DDL/rewrite path is separately certified.
The deployment/orchestration scope decision is recorded in docs/reports/gcp-deployment-orchestration-scope-decision.json. The stable GCP surface includes render artifacts, deterministic single-contract deployment manifests, dry-run project bundle materialization, generated Google Workflows source planning with bounded BigQuery job polling, connector retry blocks and deterministic broad/execution-scoped evidence-readback plus cleanup queries, an adapter-owned deploy-project --deploy-orchestration/--run-orchestration/--wait-orchestration/--readback-orchestration/--reset-orchestration-data/--cleanup-orchestration/--cleanup-orchestration-data command surface, live generated-runner deploy/execute, command-path readback, runner-side run/quality/schema evidence, quality failed-row semantics, execution-scoped evidence ids, workflow cleanup, write-failure run evidence, target/evidence cleanup and repeated full-project rerun execution/readback. Google Workflows is the certified GCP project deployment runner for the documented BigQuery batch surface. Cloud Run Jobs, Composer DAGs and BigQuery scheduled queries remain future deployment surfaces.
The Dataplex lineage and data-quality scope decision is recorded in docs/reports/gcp-dataplex-lineage-dq-scope-decision.json. SQL quality checks and BigQuery control-table evidence, including neutral OpenLineage event payloads, are in scope for the first stable surface. Rendered bundles include deterministic Dataplex DataScan create-request artifacts, DataScan command/readback metadata, native Data Lineage publication/readback planning and Dataplex aspect taxonomy/apply/readback planning. The adapter-owned dataplex-quality --execute --wait --readback command passed a live native DataScan run over a 10,000-row BigQuery target and read back seven exported Dataplex rule-result rows. The adapter-owned dataplex-lineage-aspects command has live native lineage event readback and Knowledge Catalog/Dataplex aspect modifyEntry/readback evidence in docs/reports/gcp-dataplex-lineage-aspects-smoke.json. The runtime decision is recorded in docs/reports/gcp-dataplex-lineage-aspect-runtime-decision.json: native Dataplex lineage/aspect mutation is stable-supported through the explicit command surface, while automatic native mutation during every contract run remains outside the scoped claim.
The governance stable-scope decision is recorded in docs/reports/gcp-governance-stable-scope-decision.json. Row access policy enforcement, direct column data masking, policy-tag column access, table/column descriptions, governance ledger planning artifacts, governance reconciliation planning artifacts, governance evidence write/readback for declared governance intent, non-mutating governance reconciliation readback and run/quality/annotation/failed-run evidence are in scope. The governance ledger evidence smoke is recorded in docs/reports/gcp-governance-ledger-evidence-smoke.json, the reconciliation planning decision is recorded in docs/reports/gcp-governance-ledger-reconciliation-plan.json, and the reconciliation command smoke is recorded in docs/reports/gcp-governance-ledger-reconciliation-smoke.json. Tag-based masking, policy-tag-backed masking, automatic repair/delete and overwrite-retention behavior remain excluded from the first stable GCP surface.
The stable-surface evidence manifest is recorded in docs/reports/gcp-stable-surface-evidence.json. It is the source of truth for the scoped stable_final=true claim and links the live smoke reports plus the accepted review boundaries.
The annotation smoke is recorded in docs/reports/gcp-bigquery-annotations-smoke.json. It validates BigQuery table and column descriptions using native ALTER TABLE ... SET OPTIONS and ALTER COLUMN ... SET OPTIONS, reads table metadata through INFORMATION_SCHEMA.TABLE_OPTIONS, reads column metadata through INFORMATION_SCHEMA.COLUMN_FIELD_PATHS, and persists annotation audit rows to contractforge_annotation_evidence.
The Workflows runner smoke is recorded in docs/reports/gcp-workflows-runner-smoke.json. It validates API enablement, Workflows service identity creation, generated YAML deployment without manual edits, live execution, location-aware BigQuery job polling, bronze-to-gold table readback and evidence-table DDL readback. The command-path readback smoke is recorded in docs/reports/gcp-workflows-command-readback-smoke.json; it validates the adapter-owned deploy/run/wait/readback CLI path, platform executable resolution, single-line bq SQL execution on Windows, --readback-location, target-count readback and evidence-table presence readback. The generated readback plan also includes execution-scoped run/quality/schema evidence templates; the runtime uses them when --workflow-execution-id is supplied or when the current execution id is known. The runner evidence smoke is recorded in docs/reports/gcp-workflows-runner-evidence-smoke.json; it validates runner-side run evidence for write operations and quality evidence for quality query operations. The quality semantics smoke is recorded in docs/reports/gcp-workflows-quality-semantics-smoke.json; it validates quality result readback, PASSED evidence for zero failed rows and FAILED evidence plus a raised Workflows execution for non-zero failed rows. The execution run-id smoke is recorded in docs/reports/gcp-workflows-execution-runid-smoke.json; it validates GOOGLE_CLOUD_WORKFLOW_EXECUTION_ID based run ids for successful run evidence, passed quality evidence and failed quality evidence. The schema evidence smoke is recorded in docs/reports/gcp-workflows-schema-evidence-smoke.json; it validates Workflows-side schema evidence rows from the rendered schema-policy plan with execution-scoped ids. The cleanup command smoke is recorded in docs/reports/gcp-workflows-cleanup-command-smoke.json; it validates --cleanup-orchestration, generated gcloud workflows delete metadata, live workflow-resource deletion, missing-workflow idempotency and email redaction. The write-failure evidence smoke is recorded in docs/reports/gcp-workflows-write-failure-evidence-smoke.json; it validates a controlled failed BigQuery write path, execution-scoped FAILED run evidence, non-null failed error_message, workflow raise after evidence persistence and workflow-resource cleanup. The target/evidence cleanup smoke is recorded in docs/reports/gcp-workflows-target-evidence-cleanup-smoke.json; it validates deployment/gcp_workflows_cleanup_plan.json, --cleanup-orchestration-data, scoped target/evidence cleanup SQL and live before/after evidence readback. The certified runner smoke is recorded in docs/reports/gcp-workflows-certified-runner-smoke.json; it validates the adapter-owned reset/deploy/run/wait/readback CLI path twice against the same ordered project, with execution-scoped target, run, quality and schema evidence readback for both runs.
Capability Matrix
The adapter-owned source classifier is the source of truth for source status. Every explicitly review-required
source family is also published as source.<name> in review_required_semantics, so planning, rendered source
review artifacts and public capabilities stay aligned.
| ContractForge capability | Google Cloud native surface | Adapter status | Parity decision |
|---|---|---|---|
source.type: table / view / sql | BigQuery tables, views and GoogleSQL queries | IMPLEMENTED with live SQL smoke | SQL contracts participate in the bronze-to-gold smoke; broader shared-contract parity remains pending. |
| GCS files: CSV, JSON/JSONL/NDJSON, Parquet, Avro, ORC | BigQuery batch load jobs from Cloud Storage | IMPLEMENTED with live GCS load smoke | File-format load counts and quality evidence pass; bronze-to-gold schema evolution still pending. |
| XML files | No direct BigQuery load-job format in current adapter scope | UNSUPPORTED | Use Dataflow/Dataproc or staged transformation later. |
mode: append | BigQuery INSERT or load job WRITE_APPEND | IMPLEMENTED render path | E2E must validate row counts and evidence. |
mode: overwrite | BigQuery CREATE OR REPLACE TABLE or load job WRITE_TRUNCATE | IMPLEMENTED render path | Must validate schema, policy tag and row-policy retention rules before governance parity. |
mode: upsert | BigQuery MERGE | IMPLEMENTED for explicit-column rendering and live smoke | BigQuery requires explicit update assignments; contracts without select_columns or source.read.columns remain review-required for executable SQL. Bronze-to-gold replay parity is still pending. |
mode: hash_diff_upsert | BigQuery SQL hash functions plus MERGE | REVIEW_REQUIRED by default; production parity accepted | Compact real-account first-run, changed-row replay, no-change replay and null/duplicate merge-key preflight failures passed. GCP production-sized overlap evidence plus AWS, Snowflake and Fabric hash-diff evidence close GCP-BQ-12C; flipping default stable execution remains a product decision. |
mode: historical | BigQuery DML, historical access, temporal table design | REVIEW_REQUIRED outside first stable surface; deterministic review artifact and opt-in smoke SQL rendered | Compact real-account first-run, no-change replay, delete-expression expiration and late-arriving reject failure passed. Broader late-arriving apply/ignore variants, concurrency and production parity tests remain before stable promotion. |
mode: snapshot_reconcile_soft_delete | BigQuery MERGE synchronization design | REVIEW_REQUIRED outside first stable surface; deterministic review artifact and opt-in smoke SQL rendered | Compact complete-source reconciliation, no-change replay, incomplete-source blocking, same-hash reactivation and missing-row tombstone passed. Concurrency and production parity remain before stable promotion. |
| Required columns, not-null, unique key, expression quality | BigQuery SQL and INFORMATION_SCHEMA; Dataplex data quality for governed scans | IMPLEMENTED SQL render for common checks, Dataplex DataScan create plus execution/readback planning artifacts, and native DQ execution/export readback smoke | Stable adapter quality relies on validated SQL checks. The native Dataplex path is an opt-in command surface with live DataScan job and BigQuery export readback evidence. |
| Schema policy | BigQuery INFORMATION_SCHEMA.COLUMNS, load-job schema fields, reviewed ALTER TABLE ADD COLUMN | IMPLEMENTED artifact planning, opt-in BigQuery table/view/SQL and declared-schema GCS-source runtime hook, additive nullable E2E smoke, strict negative E2E smoke, permissive nullable E2E smoke, destructive type-change negative smoke, SQL-source smoke, GCS/load-source smoke and type-mutation decision | The adapter renders preflight queries, apply hints and schema evidence DDL. The opt-in smoke hook validates BigQuery table/view/SQL-source and declared-schema GCS load-source drift, applies additive/permissive nullable columns, blocks strict drift and blocks destructive type changes with failed schema evidence. Automatic type widening or mutation remains review-required outside the stable runtime path. |
| Run evidence | BigQuery job API/CLI statistics and audit metadata | IMPLEMENTED for smoke paths | Bronze-to-gold smoke proves run evidence rows for every layer; error, governance evidence write/readback and non-mutating governance reconciliation paths are validated. |
| Cost signal | total_slot_ms, total_bytes_billed, reservations | IMPLEMENTED for run-evidence SQL reports | Smoke evidence captures bytes and slot fields; contractforge-gcp cost-report renders grouped operational cost SQL. Billing export parity remains outside the first stable surface. |
| Row filters | BigQuery row access policies | IMPLEMENTED for live smoke plus governance ledger/reconciliation planning, command readback and evidence write-readback | Apply, readback and restricted-principal enforcement passed for a simple status filter. Declared row-filter intent writes governance evidence rows in the smoke runtime and governance-reconcile --execute reads native row policy state. Copy/overwrite retention and automatic repair remain outside first stable scope. |
| Column masks / PII | BigQuery direct column data policies; policy tags and data masking | IMPLEMENTED for direct data masking, policy-tag column access, governance ledger/reconciliation planning and evidence write-readback; tag-based masking excluded from first stable scope | Direct V2 column policies passed with ALWAYS_NULL masking. Data Catalog policy tags passed for column-level access. Policy-tag-backed masking remains a distinct future design if masked values are required. |
| Annotations | BigQuery table/column descriptions; Knowledge Catalog/Dataplex aspects/tags | IMPLEMENTED for descriptions, evidence, governance ledger/reconciliation planning, evidence write-readback and explicit native aspect command execution/readback | Native BigQuery table/column descriptions render, pass live readback and persist annotation evidence. Richer aliases, tags, PII and operations metadata render Dataplex aspect plans, write governance-intent evidence rows and passed explicit dataplex-lineage-aspects --execute --readback validation with AspectType/modifyEntry/readback evidence. |
| JDBC / relational sources | BigQuery federated queries, Dataflow, Dataproc, Cloud SQL connectors | REVIEW_REQUIRED | Avoid JDBC dialect expansion for now; use table/GCS/HTTP contracts first. |
| Public REST / HTTP JSON/CSV/text sources | Shared ContractForge core bounded readers plus BigQuery local load jobs | IMPLEMENTED for public/no-auth bounded rest_api, http_json, http_csv, line-oriented http_text and declared-format http_file materialization in the smoke runtime | The adapter emits a deterministic source materialization plan, uses the shared core reader, writes a temporary local NDJSON/CSV or pass-through Avro/ORC/Parquet file, loads it into the declared BigQuery target and persists the same run/lineage evidence used by GCS load jobs. http_text and generic http_file format=text map each text line to a declared string column. Local runtime evidence is recorded in docs/reports/gcp-http-text-materialization-local-smoke.json and docs/reports/gcp-http-file-materialization-local-smoke.json; live BigQuery project evidence for http_text, generic http_file text/JSON/CSV and generic http_file Avro/ORC/Parquet is recorded in docs/reports/gcp-http-sources-bigquery-smoke.json and docs/reports/gcp-http-file-binary-bigquery-smoke.json. |
| Authenticated REST / HTTP JSON/CSV/text sources | Shared core readers plus Secret Manager-bound credentials | IMPLEMENTED for placeholder-backed bounded reads | Contracts using {{ secret:scope/key }} credentials render Secret Manager resource, IAM and access-command metadata, resolve the secret at runtime and passed live authenticated REST, REST API-key, HTTP JSON bearer-token and HTTP JSON API-key execution through the shared core readers. Inline credentials remain review-required for REST, HTTP subtypes and generic http_file. |
| Kafka bounded / available-now | Dataflow KafkaIO, Pub/Sub, BigQuery Storage Write API | REVIEW_REQUIRED outside first stable surface | Streaming is a separate maturity track. It needs checkpoint semantics, offset evidence and network/security validation before promotion. |
| Delta and Delta Sharing sources | BigLake/external table handoff, Dataproc/Spark or staged copy | REVIEW_REQUIRED | Canonical delta, delta_table and delta_share connector names are classified explicitly. Review-required contracts emit a Dataproc Serverless Spark materialization plan with dependency set, landing prefix, post-materialization table source and evidence requirements, but no GCP runtime path is claimed until the governed handoff is validated. |
| Iceberg | BigLake/Iceberg managed tables and Spark/Iceberg access | IMPLEMENTED for registered BigQuery/BigLake table reads and write-mode smoke; raw path registration command PASS_WITH_REVIEW_BOUNDARY | BigLake managed Iceberg table creation, append, MERGE and readback passed. The adapter renders registered table references as BigQuery sources; raw gs:// Iceberg paths emit and can execute a BigLake registration plan with connection, storage URI, explicit schema and metadata readback. Direct raw-path query execution without registration remains excluded. |
| Lineage | BigQuery control-table OpenLineage payloads; Dataplex/Data Catalog lineage | IMPLEMENTED for neutral control-table evidence and explicit native Data Lineage command execution/readback | Executed load/write smoke operations persist OpenLineage-shaped event payloads to BigQuery evidence tables. Native Dataplex lineage plans render from the same contract and passed explicit dataplex-lineage-aspects --execute --readback validation. The stable runtime decision keeps native Dataplex mutation explicit rather than automatic. |
| Deployment/orchestration | Workflows, Composer, Cloud Run Jobs, Dataflow Flex Templates, BigQuery scheduled queries | IMPLEMENTED for Google Workflows; other runners remain excluded | The adapter renders single-contract deployment manifests, dry-run project bundles, Google Workflows source plans, an execution-plan artifact with adapter-owned deploy/execute/wait/describe/readback/reset/cleanup command metadata, evidence-readback queries for target counts plus run/quality/schema evidence, and scoped target/evidence cleanup queries. The generated Workflows YAML includes retry blocks for BigQuery connector calls: non-idempotent retry for job submission and idempotent retry for job polling. It writes run evidence for successful write/load operations, FAILED run evidence for a validated failed write/load path, quality evidence for both zero-failure and failed-row quality outcomes, and schema evidence from the rendered schema-policy plan, with execution-scoped run ids from GOOGLE_CLOUD_WORKFLOW_EXECUTION_ID. It can run opt-in pre-run bq reset statements for planned targets/evidence, delete the generated workflow resource, treat missing-workflow cleanup as idempotent, and run opt-in post-run bq cleanup statements. The Workflows runner passed repeated full-project rerun execution/readback; Composer, Cloud Run Jobs and scheduled-query runners remain non-claims. |
Native Evidence Mapping
BigQuery gives enough telemetry to make ContractForge evidence comparable to Databricks/AWS/Snowflake/Fabric, but the adapter must implement the runtime bridge.
| ContractForge evidence field | BigQuery source |
|---|---|
run_id | BigQuery job_id. |
status | Job state and error_result. |
started_at_utc, finished_at_utc | Job start/end timestamps. |
rows_inserted, rows_updated, rows_deleted | dml_statistics.inserted_row_count, updated_row_count, deleted_row_count. |
total_bytes_processed, total_bytes_billed | BigQuery job statistics / INFORMATION_SCHEMA.JOBS*. |
total_slot_ms | BigQuery job statistics / INFORMATION_SCHEMA.JOBS*. |
row_security_applied | Job statistics row-level security metadata where present. |
data_masking_applied | Job statistics data-masking metadata where present. |
The first runtime implementation writes evidence immediately after each BigQuery job using the job id and statistics returned by the API/CLI. A later production-grade pass can reconcile the same rows against region-qualified INFORMATION_SCHEMA.JOBS_BY_PROJECT views when additional audit fields are needed.
Stability Gates
These are the concrete gates to bring GCP toward Databricks-level maturity.
| Gate | Scope | Acceptance |
|---|---|---|
GCP-BQ-01 | Plan/render parity | contractforge-gcp plan/render emits deterministic artifacts for table, SQL and GCS load contracts, including redacted source-review JSON/Markdown artifacts with structured non-JDBC source-family promotion paths. |
GCP-BQ-02 | GCS file E2E | PASS for overwrite contracts loading CSV, NDJSON, Parquet, Avro and ORC from gs:// into BigQuery with evidence writes. Append/replay belongs to bronze-to-gold parity. |
GCP-BQ-03 | Bronze-to-gold medallion E2E | PASS for a compact GCP BigQuery medallion using GCS bronze, SQL silver and SQL gold contracts with evidence writes. Cross-adapter shared-contract parity remains a broader platform parity gate. |
GCP-BQ-04 | Upsert E2E | PASS for explicit-column one-step seed/change MERGE with DML stats and evidence writes. Idempotent replay belongs to bronze-to-gold parity. |
GCP-BQ-05 | Evidence | Run and quality evidence writes pass for single-contract, file-format, upsert, bronze-to-gold and failed-run smokes. Governance evidence write/readback passes for declared governance intent. Full governance-ledger reconciliation is excluded from the first stable GCP surface. |
GCP-BQ-06 | Quality | Not-null, required columns, unique key, row-count and expression checks fail/pass consistently with Databricks. |
GCP-BQ-07A | Schema policy artifact planning | PASS: render deterministic BigQuery schema-policy plans and schema evidence DDL for strict, additive_only and permissive. |
GCP-BQ-07B | Schema policy runtime hook | PASS: opt-in smoke execution reads BigQuery table/view/SQL source schemas, declared-schema GCS load-source schemas and target INFORMATION_SCHEMA.COLUMNS, applies additive nullable columns and writes schema evidence. |
GCP-BQ-07C | Schema policy additive nullable E2E | PASS: live BigQuery smoke applies amount FLOAT64, appends rows and reads back schema evidence. |
GCP-BQ-07D | Schema policy strict negative E2E | PASS: live BigQuery smoke blocks strict additive drift before write execution and persists failed schema evidence. |
GCP-BQ-07E | Schema policy permissive nullable E2E | PASS: live BigQuery smoke applies amount FLOAT64, appends rows and reads back permissive schema evidence. |
GCP-BQ-07F | Schema policy destructive type-change negative E2E | PASS: live BigQuery smoke blocks amount STRING to amount FLOAT64 drift before write execution and persists failed type-change evidence. |
GCP-BQ-07G | Schema policy SQL source E2E | PASS: live BigQuery smoke inspects an inline SQL source through a zero-row probe, applies status STRING, appends rows and reads back schema evidence. |
GCP-BQ-07H | Schema policy GCS load source E2E | PASS: live BigQuery smoke inspects a declared-schema GCS CSV source through a load probe, applies status STRING, loads rows and reads back schema evidence. |
GCP-BQ-07I | Schema policy type mutation decision | PASS: automatic BigQuery type widening or mutation is review-required outside the stable runtime path. |
GCP-BQ-07 | Schema policy live enforcement | PASS_WITH_REVIEW_BOUNDARY: validated for strict/additive/permissive nullable additions, type-change blocking, table, SQL-source and declared-schema GCS-source inspection. Automatic type mutation is excluded. |
GCP-BQ-08 | Governance smoke | PASS for BigQuery row access policy apply/readback/enforcement, direct column data masking enforcement and Data Catalog policy-tag column access. Tag-based masking remains outside first stable scope. |
GCP-BQ-09 | Annotation smoke | PASS for BigQuery table and column descriptions plus annotation evidence rows. Tags/aspects remain review-required until the GCP governance taxonomy scope is decided. |
GCP-BQ-10 | Operational cost | PASS: total_slot_ms and bytes billed/processed are captured into evidence, and cost-report renders grouped cost SQL with operator-supplied rates. |
GCP-BQ-11 | Native lineage | PASS_WITH_REVIEW_BOUNDARY: explicit native Dataplex lineage publication/readback passed for the bronze-to-gold target through dataplex-lineage-aspects; automatic emission during every contract run is not claimed. |
GCP-BQ-11A | Neutral lineage evidence | PASS: executed load/write smoke operations persist OpenLineage-shaped event payloads into BigQuery control tables. |
GCP-BQ-12 | Write-mode decision | DECIDED: keep advanced modes review-gated by default; hash_diff_upsert production parity is accepted, while historical and snapshot_reconcile_soft_delete still need cross-adapter production parity before stable promotion. |
GCP-BQ-12B | Advanced write-mode review artifact planning | PASS: rendered bundles include deterministic BigQuery candidate SQL, blockers and promotion evidence requirements for hash_diff_upsert, historical and snapshot_reconcile_soft_delete, while deployment manifests keep execution blocked. |
GCP-BQ-12A | Advanced write-mode opt-in replay smoke | PASS_WITH_REVIEW_BOUNDARY: compact BigQuery smokes passed first-run plus no-change replay for hash_diff_upsert, historical and snapshot_reconcile_soft_delete through explicit --allow-review-required execution. Stable promotion still requires negative, late-arriving, concurrency and production parity cases. |
GCP-BQ-12C1 | Hash-diff changed-wave and source-key preflight smoke | PASS_WITH_REVIEW_BOUNDARY: hash_diff_upsert changed-row replay, no-change replay after change, null merge-key preflight failure and duplicate merge-key preflight failure passed in BigQuery with failed evidence. |
GCP-BQ-12C2 | Historical delete-expression and late-arriving reject smoke | PASS_WITH_REVIEW_BOUNDARY: historical expired a delete-classified source row without inserting a replacement current row, rejected a stale changed row before mutation, persisted failed evidence and read back the unchanged target state after the negative run. |
GCP-BQ-12C3 | Snapshot complete-source reactivation and tombstone smoke | PASS_WITH_REVIEW_BOUNDARY: snapshot_reconcile_soft_delete blocks incomplete-source contracts, reactivates an inactive same-hash target row when the complete snapshot includes it, soft-deletes a missing active target row and replays with zero DML changes. |
GCP-BQ-12C4 | Hash-diff production benchmark and overlap smoke | PASS: a 10,000-row hash_diff_upsert BigQuery benchmark passed initial load, no-change replay, 250-row changed wave, null/duplicate key failed evidence and two overlapping contract executions where BigQuery serialized the first 200-row update and second zero-row replay. Evidence: docs/reports/gcp-bigquery-hashdiff-production-benchmark.json. |
GCP-BQ-12C | Hash-diff cross-adapter production parity | PASS: GCP, AWS and Snowflake production-sized hash_diff_upsert benchmark reports exist, and Fabric stable-surface evidence covers seed/no-op/change waves. Evidence: docs/reports/gcp-hashdiff-cross-adapter-production-parity.json. |
GCP-BQ-12C5 | Historical and snapshot production benchmark | PASS_WITH_REVIEW_BOUNDARY: 10,000-row BigQuery contracts passed historical initial load, no-change replay, changed-row wave, delete-expression expiration and late-arriving rejection, plus snapshot_reconcile_soft_delete initial load, no-change replay, tombstone replay and reactivation. Evidence: docs/reports/gcp-bigquery-advanced-write-production-benchmark.json. |
GCP-BQ-13 | Stream decision | DECIDED: exclude Kafka/Event Hubs/Dataflow/Pub/Sub streaming from the first stable GCP surface. Keep it as a separate maturity track. |
GCP-BQ-13A0 | Dataflow Kafka source-promotion command surface | PASS: source-promotion --execute --readback can launch the rendered Dataflow Kafka-to-BigQuery Flex Template command, pass streaming launch options through gcloud, read back Dataflow job metadata and read back BigQuery output/DLQ table counts. |
GCP-BQ-13A | Streaming provider parity | PASS_WITH_REVIEW_BOUNDARY: the Confluent/Dataflow provider path passed row ingestion, zero-DLQ reconciliation and same-consumer-group no-input replay in a real GCP project. Broader Pub/Sub/Event Hubs coverage and production streaming operations remain review-scoped. |
GCP-BQ-14 | BigLake Iceberg smoke | PASS for registered BigLake managed Iceberg table create, append, MERGE and BigQuery table-source rendering. Raw Iceberg paths remain review-required until registration is explicit. |
GCP-BQ-15A | Workflows project-runner artifact planning | PASS: deploy-project emits a deterministic Google Workflows source plan, manifest, execution-plan artifact and evidence-readback artifact from the ordered project contracts, including bounded jobs.get polling and connector retry blocks after each submitted BigQuery job. Single-contract manifests expose execution_ready and keep apply_order empty for review-required or blocked contracts. |
GCP-BQ-15B | Workflows runner live deploy/execute smoke | PASS: a generated Workflows runner deployed and executed in a real GCP project, preserved bronze-to-gold order, polled BigQuery jobs with job location and read back target/evidence tables. |
GCP-BQ-15 | Deployment/orchestration decision | PASS_WITH_NON_WORKFLOWS_EXCLUSIONS: include the adapter-owned Google Workflows deployment runner for the stable BigQuery batch surface; exclude Cloud Run Jobs, Composer DAGs and scheduled-query deployment runners until separately certified. |
GCP-BQ-15C0 | Workflows adapter command surface | PASS: deploy-project exposes --render-orchestration, --deploy-orchestration, --run-orchestration, --wait-orchestration, --readback-orchestration, --reset-orchestration-data, --cleanup-orchestration and --cleanup-orchestration-data for the generated Workflows runner. |
GCP-BQ-15C1 | Workflows connector retry planning | PASS: generated YAML wraps jobs.insert with http.default_retry_predicate_non_idempotent and jobs.get polling with http.default_retry_predicate; the certified runner smoke exercised the same generated command path twice against live Workflows executions. |
GCP-BQ-15C2 | Workflows evidence readback planning | PASS: deploy-project emits deterministic BigQuery readback queries for target counts, run evidence, quality evidence, schema evidence and evidence-table presence; certified rerun evidence was execution-scoped for both executions. |
GCP-BQ-15C3 | Workflows evidence readback command | PASS: --readback-orchestration runs generated broad or execution-scoped readback queries with bq --format=json; --readback-location can override stale environment location. Live command-path and certified rerun smokes passed for target counts, evidence-table presence and execution-scoped run/quality/schema evidence. |
GCP-BQ-15C4 | Workflows runner evidence persistence | PASS: generated Workflows persists run evidence after successful write/load operations and quality evidence after successful quality query operations. Live runner and certified rerun smokes read back succeeded run rows per target and passed quality rows per contract. |
GCP-BQ-15C5 | Workflows quality failed-row semantics | PASS: generated Workflows reads quality query results with googleapis.bigquery.v2.jobs.getQueryResults, records PASSED quality evidence for zero failed rows, records FAILED evidence with the native failed-row count when failures exist, then raises the Workflows execution. A live negative smoke persisted failed_rows=1. |
GCP-BQ-15C6 | Workflows execution-scoped evidence ids | PASS: generated Workflows initializes contractforge_run_id from sys.get_env("GOOGLE_CLOUD_WORKFLOW_EXECUTION_ID") and binds it as a BigQuery query parameter for run and quality evidence. Live success and failure smokes produced distinguishable workflows:<execution_id>:<step>:<operation> ids. |
GCP-BQ-15C7 | Workflows schema evidence persistence | PASS: generated Workflows persists schema evidence rows for schema-policy contracts from the rendered schema-policy plan with execution-scoped run ids. Live schema and certified rerun smokes read back bronze and gold SUCCEEDED schema evidence rows. |
GCP-BQ-15C8 | Workflows cleanup command surface | PASS: deploy-project --cleanup-orchestration invokes the generated gcloud workflows delete command. Live cleanup and certified rerun smokes deleted the generated workflow; missing-workflow cleanup remains idempotent. |
GCP-BQ-15C9 | Workflows write failure run evidence | PASS: generated Workflows persists FAILED run evidence for a controlled failed BigQuery write/load operation before raising. A live smoke read back an execution-scoped failed run_id and non-null error_message. |
GCP-BQ-15C10 | Workflows target and evidence cleanup/reset | PASS: deploy-project emits a scoped deployment/gcp_workflows_cleanup_plan.json; --cleanup-orchestration-data executes generated bq cleanup statements after a run, and --reset-orchestration-data executes the same generated cleanup statements before deploy/run/wait/readback. Live cleanup and certified rerun smokes validated scoped reset/cleanup. |
GCP-BQ-15C | Certified Workflows deployment runner | PASS: the adapter-owned reset/deploy/run/wait/readback CLI path ran the same ordered project twice in a real GCP project, with expected bronze/gold counts, execution-scoped run/quality/schema evidence readback for both executions and workflow-resource cleanup. |
GCP-BQ-16A | Dataplex data-quality artifact planning | PASS: rendered bundles include deterministic Dataplex DataScan create-request JSON for supported quality rules, with unsupported rules marked review-required inside the artifact. |
GCP-BQ-16A1 | Dataplex execution/readback artifact planning | PASS: rendered bundles include deterministic DataScan create/run/get/list-jobs REST metadata, curl templates and BigQuery export readback SQL for review. |
GCP-BQ-16A2 | Dataplex data-quality execution/readback smoke | PASS_WITH_REVIEW_BOUNDARY: a native Dataplex DataScan job succeeded against a 10,000-row BigQuery target and exported seven rule-result rows to BigQuery. Evidence: docs/reports/gcp-dataplex-data-quality-execution-smoke.json. |
GCP-BQ-16B0 | Dataplex lineage and aspect artifact planning | PASS: rendered bundles include deterministic native Data Lineage publication/readback plans and Dataplex aspect taxonomy/apply/readback plans from normal contracts. |
GCP-BQ-16B1 | Dataplex lineage and aspect command surface | PASS: dataplex-lineage-aspects renders non-mutating reports by default and can explicitly execute lineage publication, aspect apply and native API readback with injected-client coverage. |
GCP-BQ-16B2 | Dataplex lineage and aspect execution/readback smoke | PASS_WITH_REVIEW_BOUNDARY: explicit dataplex-lineage-aspects --execute --readback validation published native lineage, read back run-scoped lineage events, created/read an indexed AspectType schema, discovered the BigQuery entry through searchEntries, applied aspects with modifyEntry and read them back. Evidence: docs/reports/gcp-dataplex-lineage-aspects-smoke.json. |
GCP-BQ-16B3 | Dataplex lineage and aspect runtime decision | PASS: explicit dataplex-lineage-aspects execution remains the stable native Dataplex mutation surface. Automatic native lineage/aspect emission during ordinary contract runs remains excluded to avoid hidden platform metadata side effects, IAM/quota coupling and retry/idempotency changes. Evidence: docs/reports/gcp-dataplex-lineage-aspect-runtime-decision.json. |
GCP-BQ-16 | Dataplex lineage and aspects decision | PASS: SQL quality evidence, Dataplex create plus execution/readback planning artifacts, native Dataplex DQ execution/readback, native lineage/aspect planning artifacts, explicit native lineage/aspect command execution/readback and runtime-scope decision are in scope. Automatic native lineage/aspect emission during every contract run remains excluded. |
GCP-BQ-17A | Governance ledger artifact planning | PASS: rendered bundles include deterministic governance ledger planning JSON and governance evidence DDL for declared access/annotation intent. |
GCP-BQ-17B1 | Governance ledger evidence write/readback smoke | PASS_WITH_REVIEW_BOUNDARY: a governed BigQuery smoke persisted three governance evidence rows for declared row-filter, description and aspect/tag intent, read them back from contractforge_governance_evidence and verified principal redaction. Evidence: docs/reports/gcp-governance-ledger-evidence-smoke.json. |
GCP-BQ-17B2 | Governance ledger reconciliation artifact planning | PASS: rendered bundles with governance intent include a non-mutating reconciliation artifact with expected contract state, BigQuery INFORMATION_SCHEMA readback queries, row/data policy and IAM API readback metadata, reconciliation states and redacted principals. Evidence: docs/reports/gcp-governance-ledger-reconciliation-plan.json. |
GCP-BQ-17B | Governance ledger reconciliation command surface | PASS_WITH_REVIEW_BOUNDARY: governance-reconcile --execute performs non-mutating native BigQuery readback for row policies, data policies, policy tags, descriptions and ContractForge governance evidence, then compares readback to contract intent. Auto-repair/delete, overwrite-retention certification and enforcement loops remain review-scoped. Evidence: docs/reports/gcp-governance-ledger-reconciliation-smoke.json. |
GCP-BQ-17 | Governance stable-scope decision | DECIDED: include validated row policy, direct masking, policy-tag column access, description evidence, governance ledger/reconciliation planning, smoke-runtime governance evidence write/readback and non-mutating reconciliation readback; exclude tag-based masking, overwrite-retention and automatic governance repair/delete. |
GCP-BQ-18 | Stable-surface manifest | PASS: docs/reports/gcp-stable-surface-evidence.json records the included GCP BigQuery surface, validation reports and accepted review boundaries. |
GCP-BQ-19 | Sequential project smoke | PASS: contractforge-gcp run-project runs project execution-order contracts through the same BigQuery smoke runtime. This is not a reusable Google Cloud deployment runner. |
GCP-BQ-20A | Public REST/HTTP source materialization | PASS: public/no-auth bounded rest_api, http_json, http_csv, line-oriented http_text and generic http_file contracts declared as csv/json/text render source materialization plans and execute through core readers plus BigQuery local load jobs in the smoke runtime. Local materialization evidence is recorded in docs/reports/gcp-http-text-materialization-local-smoke.json and docs/reports/gcp-http-file-materialization-local-smoke.json; live BigQuery project evidence is recorded in docs/reports/gcp-http-sources-bigquery-smoke.json. |
GCP-BQ-20B | Authenticated REST/HTTP Secret Manager review planning | PASS: authenticated REST/HTTP contracts with {{ secret:scope/key }} placeholders render deterministic Secret Manager resource, IAM and access-command metadata. |
GCP-BQ-20D | Authenticated REST/HTTP Secret Manager execution smoke | PASS: live authenticated REST, REST API-key, HTTP JSON bearer-token and HTTP JSON API-key contracts resolved credentials from Google Secret Manager at runtime, read through the shared core readers and loaded materialized rows into BigQuery. Evidence: docs/reports/gcp-authenticated-rest-secret-manager-smoke.json and docs/reports/gcp-auth-rest-http-secret-manager-variants-smoke.json. The earlier gcloud reauthentication blocker in docs/reports/gcp-auth-rest-http-secret-manager-variants-blocker.json is superseded. |
GCP-BQ-20C | Non-JDBC source-family promotion artifact planning | PASS: raw Iceberg path, Delta/Delta Sharing, undeclared/unsupported HTTP file variants and streaming review-required sources emit deterministic promotion-plan JSON with required bindings, evidence and blockers while most execution remains review-required. Raw Iceberg paths emit a BigLake registration plan with bq mk flags, connection/IAM requirements, explicit schema, readback assertions and the post-registration source table shape. Delta/Delta Sharing sources emit Dataproc Serverless Spark materialization plans, and Kafka/Event Hubs sources emit Dataflow Kafka-to-BigQuery promotion plans with checkpoint, consumer-group and offset evidence requirements. Line-oriented http_text and declared-format http_file have moved to adapter-runtime materialization with live BigQuery evidence for Avro/CSV/JSON/ORC/Parquet/text. |
GCP-BQ-20E | Raw Iceberg BigLake registration command | PASS_WITH_REVIEW_BOUNDARY: contractforge-gcp source-promotion --execute --readback created a BigLake Iceberg table from a declared raw gs:// prefix with explicit schema, read back biglakeConfiguration, verified schema fields and queried the registered table. Evidence: docs/reports/gcp-raw-iceberg-registration-smoke.json. |
Recommended Next Implementation Order
- Run historical and snapshot production-sized parity contracts against AWS/Snowflake or a shared cross-adapter benchmark before promoting those advanced modes.
- Expand streaming beyond the validated Confluent/Dataflow provider path only if Pub/Sub, Event Hubs or long-running production streaming operations become a near-term product requirement.
- Execute and read back non-Workflows orchestration surfaces only if they become product requirements; Google Workflows is the certified runner for the stable GCP surface.
- Keep automatic type widening/mutation and automatic native Dataplex mutation as future reviewed extensions unless separate runtime paths are certified.
Future Promotion Gates
These gates are outside the current stable-supported surface, but are now tracked explicitly by contractforge-gcp stabilization-report.
| Gate | Scope | Required Evidence |
|---|---|---|
GCP-BQ-12D | Historical and snapshot production parity | Cross-adapter production-sized parity acceptance for historical and snapshot_reconcile_soft_delete contracts beyond the recorded GCP and Fabric evidence. |
GCP-BQ-20 | Expanded source families | Delta Sharing, direct raw Iceberg path execution without registration, undeclared/unsupported HTTP file variants and other non-JDBC source-family E2E validation without workaround code. |
Non-Claims
The current GCP adapter does not yet claim:
- Databricks-level full runtime parity beyond the scoped BigQuery batch and Workflows runner surface;
- JDBC, Kafka/Event Hubs streaming, direct raw Iceberg path execution without registration, Delta Sharing parity or inline authenticated REST/HTTP credentials;
- governance parity for tag-based masking, automatic Knowledge Catalog/Dataplex aspect emission, overwrite-retention behavior or automatic governance repair/delete;
- automatic BigQuery type widening or type mutation;
- stable historical or snapshot soft-delete support beyond the opt-in replay/preflight/production-benchmark evidence already recorded for review-required execution;
- automatic native Dataplex lineage/aspect emission during every contract run;
- live deployment/orchestration parity.
Sources
- BigQuery batch loading from Cloud Storage: https://docs.cloud.google.com/bigquery/docs/batch-loading-data
- BigQuery DML and
MERGE: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax - BigQuery Standard SQL hash functions: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hash_functions
- BigQuery historical data access: https://docs.cloud.google.com/bigquery/docs/access-historical-data
- BigQuery
INFORMATION_SCHEMA.JOBS: https://docs.cloud.google.com/bigquery/docs/information-schema-jobs - BigQuery table and column metadata: https://docs.cloud.google.com/bigquery/docs/tables
- BigQuery
INFORMATION_SCHEMA.COLUMN_FIELD_PATHS: https://docs.cloud.google.com/bigquery/docs/information-schema-column-field-paths - BigQuery row-level security: https://docs.cloud.google.com/bigquery/docs/row-level-security-intro
- BigQuery column-level access control and policy tags: https://docs.cloud.google.com/bigquery/docs/column-level-security-intro
- BigQuery access control and data policy roles: https://docs.cloud.google.com/bigquery/docs/access-control
- BigQuery Iceberg managed tables: https://docs.cloud.google.com/bigquery/docs/biglake-iceberg-tables-in-bigquery
- BigQuery Cloud SQL federated queries: https://docs.cloud.google.com/bigquery/docs/connect-to-sql
- Secret Manager access control: https://docs.cloud.google.com/secret-manager/docs/access-control
- Secret Manager secret version access: https://docs.cloud.google.com/secret-manager/docs/access-secret-version
gcloud secrets versions access: https://docs.cloud.google.com/sdk/gcloud/reference/secrets/versions/access- Dataflow Kafka reads: https://docs.cloud.google.com/dataflow/docs/guides/read-from-kafka
- Pub/Sub BigQuery subscriptions: https://docs.cloud.google.com/pubsub/docs/bigquery
- Google Cloud Workflows: https://cloud.google.com/workflows/docs
- Google Cloud Workflows built-in environment variables: https://docs.cloud.google.com/workflows/docs/reference/environment-variables
- Google Cloud Workflows BigQuery connector: https://cloud.google.com/workflows/docs/reference/googleapis/bigquery/Overview
- Google Cloud Workflows retry syntax: https://docs.cloud.google.com/workflows/docs/reference/syntax/retrying
- Dataplex REST API: https://docs.cloud.google.com/dataplex/docs/reference/rest
- Data Catalog lineage: https://docs.cloud.google.com/data-catalog/docs/data-lineage
- Knowledge Catalog / Dataplex data quality tasks: https://docs.cloud.google.com/dataplex/docs/check-data-quality
- Data Lineage REST API: https://cloud.google.com/data-catalog/docs/reference/data-lineage/rest
- Dataplex Aspect resource: https://docs.cloud.google.com/dataplex/docs/reference/rest/v1/Aspect
- Dataplex modifyEntry API: https://docs.cloud.google.com/dataplex/docs/reference/rest/v1/projects.locations/modifyEntry
- Dataplex AspectTypes API: https://docs.cloud.google.com/dataplex/docs/reference/rest/v1/projects.locations.aspectTypes