Runtime compatibility.
ContractForge is adapter-aware, but storage access, connector dependencies and governance features still depend on where the code runs. The core validates semantics; adapters validate and execute platform-specific runtime behavior.
Runtime matrix
| Capability | Classic cluster | Serverless / Spark Connect | Local Spark |
|---|---|---|---|
| Delta batch writes | Supported | Supported | Supported with Delta dependencies |
| Python DeltaTable APIs | Usually supported | Limited | Depends on dependencies |
| SQL MERGE | Supported | Supported | Supported with Delta |
| Auto Loader | Databricks only | Databricks only | Not available |
| Unity Catalog annotations | Depends on permissions | Depends on permissions | Not available |
| Direct object-storage credentials | Common pattern | Prefer External Location or Volume | Environment-dependent |
| Federated sources | Supported when UC connection exists | Recommended for governed external systems | Not available |
| JDBC | Requires driver/network | Requires driver/network support | Requires driver/network |
Connector dependency model
Built-in Spark formats
CSV, JSON, Parquet, Delta, ORC and text usually work when Spark supports them.
Extra Spark packages
XML, Avro and JDBC drivers must be available in the runtime that executes the read. Snowflake, BigQuery, MongoDB and similar systems are externalized source patterns; use platform/vendor connectors, federation or custom resolver packages, then pass the landed result to ContractForge.
Cloud object stores
Serverless should use external locations or volumes; classic clusters can use Spark/Hadoop credentials when policy allows it.
Driver-side HTTP
REST and HTTP file depend on driver network egress, DNS, proxy rules and API limits. Microsoft Graph, SharePoint Server, SFTP and similar sources are externalized unless implemented by a project-specific custom resolver.
Spark and Delta package policy
| Environment | Status | Dependency owner | Decision |
|---|---|---|---|
| Databricks classic clusters | Supported | Databricks Runtime | Use the runtime-provided Spark and Delta stack. |
| Databricks serverless / Spark Connect | Supported with runtime limits | Databricks Runtime | Prefer governed storage, SQL-compatible operations and documented fallbacks. |
| Local core-only planning | Supported | contractforge-core | No Spark required. |
| Local Databricks adapter rendering | Supported | contractforge-databricks | Renders artifacts without executing Spark. |
| Local PySpark execution | Adapter/project-managed | User or adapter test environment | Requires explicit validation of Spark, Delta and connector dependencies together. |
Install contractforge-core plus the adapter wheel required by the target runtime. On Databricks, let Databricks Runtime provide Spark and Delta.
Serverless connector patterns
Use serverless for governed, repeatable ingestion where the access path is configured by the platform. Use classic clusters when you need to control low-level libraries, JVM options or filesystem credentials directly.
| Source | Preferred serverless pattern | When to use classic instead |
|---|---|---|
| S3 / Azure Blob / ADLS | Unity Catalog External Location or Volume; ContractForge reads the governed path. | Direct S3A/ABFS/SAS credentials, custom Hadoop libraries or nonstandard credential providers. |
| Specialized SaaS/files/streams | Use Databricks-native connectors, Lakeflow Connect, Lakehouse Federation, marketplace/vendor tools or custom resolver packages; ContractForge reads the landed/federated result. | Direct connector libraries, custom networking or vendor-specific runtime dependencies that the platform path cannot cover. |
| JDBC / RDS IAM | Supported when driver, route, SSL and IAM credentials are available to the runtime. | Private databases that require VPC/VNet-level customization, driver installation or long-running extraction tuning. |
| REST / HTTP file | Driver-side HTTP with explicit timeouts, retries and size/page limits. | Large payloads that should first be landed in object storage or API routes requiring custom proxies. |
Serverless failure model
Serverless failures often happen before the external provider evaluates the credential. Treat runtime reachability, provider authentication and contract syntax as separate layers.
| Symptom | Likely layer | Next check |
|---|---|---|
| Secret resolves but access is denied by the provider. | Provider authentication or authorization. | Validate app permissions, Snowflake network policy, IAM grants, database grants or Graph site permissions. |
| DNS, connect timeout, port blocked or endpoint unavailable. | Workspace network policy or routing. | Validate serverless network connectivity, PrivateLink/NCC, firewall allowlists, proxy requirements or use a governed connection. |
| Spark rejects Hadoop credential configuration. | Spark Connect/runtime policy. | Use Unity Catalog External Location or Volume instead of notebook-level filesystem configuration. |
| Connector class or JDBC driver is missing. | Runtime dependency. | Install the package on classic/job clusters, use a supported serverless library path or use federation. |
HTTP 403 or malformed response is returned by a driver connector. | Deterministic source response. | Fix credentials, request URL, permissions or response parsing. ContractForge does not retry deterministic failures. |
Compatibility rule
If a feature depends on a platform capability, document it in the connector page and expose a clear error. Do not silently fall back to a weaker behavior that may hide runtime misconfiguration.