Skip to main content

AWS Capability Parity

This document records researched AWS-native capabilities that should inform the contractforge-aws adapter roadmap. It exists to avoid under-building the AWS adapter simply because a capability was not part of an earlier implementation pass.

The adapter should aim for the maximum ContractForge semantics that AWS can preserve natively, while still returning REVIEW_REQUIRED or UNSUPPORTED when semantic equivalence is not proven.

Primary Runtime Thesis​

The primary AWS runtime remains:

AWS Glue Spark
Apache Iceberg
AWS Glue Data Catalog
Amazon S3
AWS Lake Formation
Iceberg evidence tables

This target is capable of more than review artifacts:

  • Glue supports Apache Iceberg tables and DataFrame writes through the Glue Data Catalog.
  • Iceberg tables support append, create/replace and SQL/DataFrame merge patterns through Spark.
  • Glue Data Quality provides DQDL and a PySpark EvaluateDataQuality transform.
  • Lake Formation data filters support column-level, row-level and cell-level access controls for read operations.
  • Glue job bookmarks provide incremental source tracking for JDBC and selected S3 formats.
  • Glue native connectors and custom connectors provide a better native passthrough target than writing SaaS connector code inside ContractForge.

Runtime Parity Matrix​

ContractForge areaAWS-native capabilityAdapter target statusImplementation direction
appendIceberg append through Glue Spark/DataFrame writerSUPPORTEDRuntime script generated. If the target table does not exist, the job bootstraps it from the prepared dataframe before future append runs and applies extensions.aws.iceberg.table_properties on that create path. Keep table format version and catalog config explicit.
overwriteIceberg create/replace or overwrite patternsSUPPORTED_WITH_WARNINGSRuntime script generated with create/replace today and applies extensions.aws.iceberg.table_properties; add partition/scope-aware overwrite before claiming full production parity.
upsertSpark SQL MERGE INTO against IcebergSUPPORTEDRuntime script generated with merge-key guards. If the target table does not exist, the job bootstraps it from the prepared dataframe and applies extensions.aws.iceberg.table_properties before future MERGE runs. Needs AWS integration test for concurrency and large upsert volumes.
hash_diff_upsertHash staging plus Iceberg mergeSUPPORTED_WITH_WARNINGSRuntime script generated: merges on merge_keys, computes row_hash from hash_keys or hash_strategy: all_columns_except, excludes user-declared and generated columns, prefilters unchanged rows before Iceberg MERGE, and records no-change runs as SKIPPED / no_hash_changes. If the target table does not exist, the job bootstraps it from the prepared dataframe and applies extensions.aws.iceberg.table_properties. Performance/concurrency warning stays until validated on Glue.
historicalIceberg merge/history patternREVIEW_REQUIREDFeasible, but effective dating, deletes, late-arriving policy and concurrent writers need explicit AWS tests.
snapshot_reconcile_soft_deleteFull snapshot reconciliation plus Iceberg mergeREVIEW_REQUIREDFeasible only when source completeness and target concurrency are proven.
required_columnsGlue Data Quality ColumnExists via EvaluateDataQualitySUPPORTEDEvaluated natively in-job; abort severity fails the run, results persisted to ctrl_ingestion_quality.
not_nullGlue Data Quality IsComplete via EvaluateDataQualitySUPPORTED_WITH_WARNINGSEvaluated natively; warn rules record quality evidence and continue; row-level quarantine rules write offending rows to ctrl_ingestion_quarantine, drop them before the target write and update run quarantine counts.
unique_keyGlue Data Quality IsUnique/IsPrimaryKey via EvaluateDataQualitySUPPORTEDEvaluated natively in-job; abort severity fails the run, results persisted to ctrl_ingestion_quality.
accepted_valuesGlue Data Quality ColumnValues ... in [...] via EvaluateDataQualitySUPPORTED_WITH_WARNINGSEvaluated natively; non-abort outcomes record quality evidence. Row-level quarantine writes offending rows to ctrl_ingestion_quarantine, drops them before the target write and updates run quarantine counts.
max_null_ratioGlue Data Quality Completeness threshold via EvaluateDataQualitySUPPORTED_WITH_WARNINGSEvaluated natively; warn outcomes recorded as quality evidence.
expression qualitySpark SQL DataFrame filter checksSUPPORTED_WITH_WARNINGSRuntime script generated outside DQDL because the expression dialect is Spark SQL. abort fails the run; warn records evidence; row-level quarantine writes offending rows and filters them before the target write.
select_columnsSpark DataFrame projectionSUPPORTEDRuntime script generated.
column_mappingSpark DataFrame renameSUPPORTEDRuntime script generated with collision checks.
filter_expressionSpark SQL/DataFrame filterSUPPORTED_WITH_WARNINGSRuntime script generated; expression dialect is Spark SQL, so complex cross-platform filters may need review.
shape.parse_jsonSpark from_json and schema structsSUPPORTED_WITH_WARNINGSRuntime script generated for concrete schemas (and resolved schema_ref), with optional cast_input: STRING and drop_source.
shape.arrays.explodeSpark explode functionsSUPPORTED_WITH_WARNINGSRuntime script generated for all modes (size/to_json/first/explode/explode_outer). explode/explode_outer are blocked in bronze unless allow_cardinality_change_on_bronze; sibling explodes under one parent need allow_cartesian.
shape.zip_arraysSpark arrays_zip plus transform field renamingSUPPORTED_WITH_WARNINGSRuntime script generated with temporary array columns so nested paths keep deterministic struct field names.
transform.castSpark DataFrame castsSUPPORTEDImplement in AWS renderer before broad transform support.
transform.standardizeSpark string functionsSUPPORTEDImplement trim/lower/upper/whitespace/null normalization.
transform.deriveSpark SQL expressionsSUPPORTED_WITH_WARNINGSFeasible; expression dialect must be Spark SQL.
transform.composite_keysSpark concat/coalesce functionsSUPPORTEDUses Spark concat_ws/coalesce with the core contract's delimiter semantics.
transform.deduplicateSpark window/groupingSUPPORTED_WITH_WARNINGSRuntime script generated: Window.partitionBy(keys).orderBy(order_by) + row_number() == 1. order_by is required by the core contract (deterministic); list-of-dicts and simple column-string clauses render, while unsafe free-form SQL strings become review-only.
shape.flattenSpark schema introspection + selectSUPPORTED_WITH_WARNINGSRenders a runtime _cf_flatten helper that expands struct fields to leaf columns with the contract's separator/include/exclude/max_depth (arrays kept intact). Glue native Relationalize is not used because it pivots arrays into separate frames with fixed naming.
shape.columnsSpark select projectionSUPPORTEDRuntime script generated: string shorthand maps a path to an alias; object form supports expression (Spark SQL), cast and alias.
incremental_filesGlue job bookmarks for selected S3 formats or evidence stateSUPPORTED_WITH_WARNINGSRenders an S3 create_dynamic_frame.from_options bookmark read with transformation_ctx for JSON/CSV/Parquet/ORC/Avro/XML. Portable CSV options are translated to Glue DynamicFrame options (header -> withHeader, delimiter -> separator) and Spark-only options such as inferSchema are not passed to Glue bookmark reads. enable_job_bookmark wires --job-bookmark-option. Unsupported formats stay review-only.
JDBC incrementalGlue job bookmarks for JDBC bookmark keysSUPPORTED_WITH_WARNINGSRuntime script generated when source.incremental.watermark_column is a simple column and the JDBC URL/connector maps to a Glue JDBC connection type; otherwise it falls back to the Spark JDBC reader plus core watermark predicates/evidence state.
state evidenceIceberg state tables in Glue CatalogSUPPORTED_WITH_WARNINGSGenerated Glue jobs upsert ctrl_ingestion_state after committed writes with run id, status, rows written, Iceberg snapshot id and batch watermark candidates when the declared watermark column exists in the prepared DataFrame. Streaming checkpoint progress is tracked in ctrl_ingestion_streams.
schema-change evidenceIceberg schema-change table in Glue CatalogSUPPORTED_WITH_WARNINGSGenerated Glue jobs compare target table schema before/after the write, append added-column and type-change rows to ctrl_ingestion_schema_changes, and copy the summary into run evidence. Complex schema policy decisions still require review.
kafka_bounded / eventhubs_boundedSpark bounded Kafka/Event Hubs read in GlueSUPPORTED_WITH_WARNINGSRuntime script generated; matching Spark connector jars must be supplied and offset/range semantics remain connector-owned.
kafka_available_now / eventhubs_available_nowGlue structured streaming with the availableNow triggerSUPPORTED_WITH_WARNINGSRenders a readStream + writeStream.foreachBatch job with trigger(availableNow=True) and the contract's checkpoint_location; preparation/quality/write run per micro-batch for append/merge modes. Per-batch evidence is written to ctrl_ingestion_streams and rolled into final run evidence. Validate offset/checkpoint progress before production.
row filtersLake Formation data filtersREVIEW_REQUIREDRenders a CreateDataCellsFilter scaffold (fail-closed false row expression) plus a SELECT grant on the filter; the row-filter function cannot be auto-translated to an LF FilterExpression, so a reviewer completes it. Planning stays REVIEW_REQUIRED.
column masksLake Formation column/cell filtersREVIEW_REQUIREDRenders a data-cells-filter scaffold that excludes the masked column (column-level security); LF has no value-masking function, so a transformed value must come from the ingestion job or a consumer view. Planning stays REVIEW_REQUIRED.
access grantsLake Formation grants and IAM policiesSUPPORTED_WITH_WARNINGSrender_aws_lake_formation_plan emits applyable GrantPermissions requests (.lakeformation.json), plus ctrl_ingestion_access evidence SQL. Write-side job-role permissions are separate from consumer grants.
annotationsAWS Glue Catalog table/column metadataSUPPORTED_WITH_WARNINGSRenders .annotations.json with planned glue:UpdateTable metadata changes and .annotations_evidence.sql for ctrl_ingestion_annotations. Application remains explicit because Glue table updates must preserve the full current TableInput.
operations metadataIceberg evidence table in Glue CatalogSUPPORTEDRenders normalized .operations.json and .operations.sql for ctrl_ingestion_operations, preserving owners, groups, criticality, SLA, alert flags, runbook and tags.
source metadata / lineageGlue JobRun, Spark schema metadata, optional OpenLineage exportSUPPORTED_WITH_WARNINGSGenerated Glue jobs persist source metadata, rows read, source column count and runtime source schema to ctrl_ingestion_metadata, then append OpenLineage-compatible events to ctrl_ingestion_lineage. Glue Catalog API enrichment remains adapter-owned future work.
cost signalsGlue DPUSeconds, worker type/count and job durationSUPPORTEDReconciliation maps Glue DPUSeconds into ctrl_ingestion_cost; *.cost.sql estimates USD only when an explicit DPU-hour rate is provided.
http_file (http_csv/json/text)Driver-side bounded fetch + Spark in-memory parseSUPPORTED_WITH_WARNINGSRuntime _cf_http_dataframe helper validates the resolved host (rejects private/IMDS) and refuses redirects; auth secrets resolve via Secrets Manager.
rest_apiShared core bounded REST clientSUPPORTED_WITH_WARNINGSThe Glue job calls contractforge_core.connectors.read_rest_api_records (pagination/auth/limits, implemented under contractforge_core.connectors.api.rest); requires contractforge-core on the job. The shared client validates API and OAuth token URLs, rejects unsupported schemes/private hosts by default and refuses redirects. Secrets resolve via Secrets Manager.
delta_shareSpark Delta Sharing readerSUPPORTED_WITH_WARNINGSspark.read.format('deltaSharing'); provide the delta-sharing-spark jar.
native passthroughGlue native connectors, custom connectors, AppFlow/DMS patternsREVIEW_REQUIREDRenders .native_passthrough.json with AWS-native recommendations and review-only API-shaped candidates for AppFlow CreateFlow, DMS CreateReplicationConfig and Glue CreateConnection. Do not implement SaaS algorithms in ContractForge.

  1. Preparation parity

    • Keep transform.cast, transform.standardize, transform.derive, transform.composite_keys covered by AWS Glue script tests.
    • Add transform.deduplicate only with deterministic order validation. Done: runtime generation requires keys and order_by, then renders a Spark window and row_number() == 1.
    • Add shape.parse_json for concrete schemas before array explosion. Done: runtime generation supports concrete schemas/resolved schema_ref, optional cast_input: STRING, arrays, zip arrays, columns and flattening with guardrails.

  2. AWS-native quality

    • Render DQDL artifacts for all portable quality rules with faithful DQDL equivalents. Done: render_aws_quality_dqdl emits a Rules = [...] ruleset (.quality.dqdl artifact) mapping required_columns/not_null/unique_key/accepted_values/row_count_minimum/max_null_ratio; expression rules are evaluated by Spark SQL runtime checks instead of DQDL.
    • Add a Glue Data Quality runtime path (EvaluateDataQuality). Done: the job evaluates the DQDL ruleset in-job, partitioned by severity — abort rules raise on failure, warn/quarantine rules continue.
    • Persist DQ results to ContractForge evidence tables. Done: one immutable row per rule is appended to ctrl_ingestion_quality (_cf_persist_quality_evidence) sharing the run's _cf_run_id.
    • Enable quarantine/warn runtime behavior. Done for mapped row-level rules: non-abort rules are recorded and never fail the run; not_null and accepted_values quarantine rules use rowLevelOutcomes to write offending rows to ctrl_ingestion_quarantine and filter them before the target write. Spark SQL expression quarantine rules use DataFrame filters for the same evidence behavior. Aggregate/schema quarantine rules remain recorded evidence only.

  3. Incremental ingestion

    • Render Glue bookmark configuration for eligible S3/JDBC sources. Done: S3 incremental files and eligible JDBC incremental sources render DynamicFrame bookmark reads and enable the job bookmark deployment flag.
    • Record selected incremental strategy in state/evidence.
    • Fallback to ContractForge state tables when bookmarks cannot preserve semantics.

  4. Governance

    • Generate Lake Formation data filter and grant artifacts. Done: render_aws_lake_formation_plan emits GrantPermissions requests and CreateDataCellsFilter scaffolds (.lakeformation.json).
    • Keep application behind explicit apply commands.
    • Record governance application evidence.

  5. Advanced write modes

    • Implement hash_diff_upsert with adapter-owned staging. Done: runtime generation computes a core-compatible row hash and merges only changed rows.
    • Prototype historical and snapshot_reconcile_soft_delete in integration tests before changing planning status.

  6. Native passthrough

    • Add Glue native connector artifacts for supported SaaS systems.
    • Add AppFlow/DMS review/apply artifacts where those services preserve the contract intent better than Glue code.

Known AWS Constraints To Preserve​

  • Lake Formation filters apply to read access. They are not the same thing as write-time row rejection or engine-side table functions.
  • Glue job bookmarks track source progress; they do not clean up or reconcile target data during rewind/reset.
  • JDBC bookmarks depend on suitable bookmark keys and do not support case-sensitive bookmark columns.
  • Glue Data Quality can evaluate rich rule sets, but ContractForge row-level quarantine is only enabled where Glue rowLevelOutcomes can identify offending rows without changing rule semantics.
  • Iceberg support varies by Glue version. The adapter must keep Glue version in runtime configuration and evidence.
  • AWS SDK dependencies stay outside the base adapter import path. Runtime helpers may use contractforge-aws[runtime] or caller-provided clients.

Official References​