Runtime validation matrix
ContractForge separates connector compatibility from runtime validation.
Compatibility means the contract shape and implementation path are supported. Validation means the path has been exercised against a real source, runtime or file layout. Production readiness still depends on the user's network, identity, cloud policy, provider limits and data volume.
Connector maturity levels
| Status | Meaning | Production guidance |
|---|---|---|
stable | Core connector path with broad automated coverage and low platform-specific variability. | Suitable for normal use after environment validation. |
validated | Tested against at least one real external service/runtime path. | Review documented prerequisites and repeat validation in the target workspace. |
experimental | Limited or adapter-specific behavior that should be validated in the target runtime. | Use for pilots or controlled onboarding; add real-source validation before production. |
planned | Intended connector or validation path not yet supported. | Track as roadmap only. |
validated is intentionally scoped. It does not certify every authentication mode, provider edition, network topology, workspace policy or data-volume profile for that connector. It means ContractForge has evidence for at least one representative real-runtime path.
Current maturity summary
| Connector family | Connectors | Maturity | Evidence summary |
|---|---|---|---|
| Spark catalog and files | table, sql, csv, json, parquet, orc, delta, text | stable | Covered by automated tests and real file-folder validation patterns. |
| Object/file formats | source.format=avro, xml, jsonl, ndjson | validated | Used through object-storage style connectors when the runtime provides the required Spark readers. |
| HTTP files | http_file, http_csv, http_json, http_text | validated | Used for public HTTP file ingestion where direct Spark https:// reads are not reliable. |
| Object storage | s3, azure_blob, adls, object_storage, blob | validated | Validated through governed external locations and selected direct credential paths where the runtime allows them. |
| Auto Loader | autoloader | validated | Validated with available-now ingestion, checkpoints, child runs and stream metric aggregation. |
| JDBC | jdbc, postgres, postgresql, mysql, sqlserver, oracle, redshift | validated | Validated with real PostgreSQL/Supabase and RDS/Aurora IAM scenarios; other JDBC engines share the same Spark JDBC contract and require driver/runtime validation. |
| Bounded REST | rest_api | validated | Validated with nested public API payloads, raw response preservation and transform.shape.parse_json. |
| Externalized sources | SaaS, ERP, CRM, drives, SFTP/FTP, Kafka/Event Hubs, Snowflake, BigQuery, MongoDB/search | externalized | Use native platform/vendor connectors or custom resolver packages, then expose data to ContractForge through official primitives. |
Runtime evidence
| Scenario | Runtime | Status | What was validated |
|---|---|---|---|
| Public CSV over HTTP | Databricks serverless | validated | Driver-side HTTP download, explicit CSV schema/options, overwrite target and control-table metadata. |
| Nested REST JSON | Databricks serverless/classic pattern | validated | Raw API payload preservation and transform.shape.parse_json for nested structs/arrays. |
| Azure Blob via External Location | Azure Databricks serverless | validated | Governed storage access without direct SAS/Hadoop credential injection in the contract. |
| Azure Blob via SAS/direct filesystem config | Azure Databricks classic cluster | validated | Direct storage credential behavior where runtime filesystem configuration is allowed. |
| AWS S3 via External Location | AWS Databricks serverless | validated | Unity Catalog-governed S3 access and object storage ingestion patterns. |
| AWS S3 direct credentials | Classic cluster | validated | S3A credential setup with access key/session token from secrets. |
| Auto Loader available-now | Databricks classic/serverless patterns | validated | Checkpoints, child runs and aggregate stream metrics in ctrl_ingestion_streams. |
| JDBC PostgreSQL/Supabase | Databricks | validated | JDBC reads, pushdown, partitioning, watermark and SCD/hash-diff patterns. |
| RDS/Aurora IAM JDBC | AWS Databricks serverless and classic diagnostics | validated | IAM token generation, SSL parameters, network diagnosis and Spark JDBC options. |
| Public USGS GeoJSON shape | Azure Databricks classic cluster | validated | REST/raw GeoJSON with schema_ref, array-of-struct explosion, coordinate extraction, SCD hash diff and gold marts. Latest evidence: execution run 516033488155314, validation run 1040635449894221; 1 bronze payload, 1,807 silver events, 31 daily gold rows and 4 magnitude-band rows. |
| Large known dataset TPCH | Azure Databricks classic cluster | validated | samples.tpch.lineitem source with 29,999,795 observed rows and a 250,000-row processed validation. Validated sql -> table -> sql, hash_diff_upsert, transform derivation/deduplication, gold aggregation and control-table evidence. Latest evidence: execution run 714754107506943, validation run 369877249106831. |
| Native connector handoff | Databricks | validated pattern | External systems can be landed or federated by Databricks/native tooling, then ingested by ContractForge as table, sql, files or object storage. |
Production adoption rule
Before marking a new connector path as production-ready in a project, record:
- Runtime type and Databricks workspace constraints.
- Connector version and external library version when applicable.
- Authentication mode and secret redaction behavior.
- Network path and provider-side allowlists/policies.
- Representative data volume, pagination and file-count limits.
- Control-table evidence: run status, row counts, errors, quality status and stream metrics when applicable.