Write modes
Merge strategies.
scd1_upsert supports different strategies for current-state tables. Choose the strategy that matches source completeness and partition safety.
Strategy comparison
| Strategy | Use when | Risk |
|---|---|---|
delta | Generic key-based merge. | Can scan more target data on large tables. |
delta_by_partition | The merge can safely restrict target partitions. | Requires correct merge_partition_column. |
replace_partitions | The source is complete for the partitions being replaced. | Data loss if source completeness is wrong. |
Examples
mode: scd1_upsert
merge_strategy: delta
merge_keys: [order_id]result = ingest(..., mode="scd1_upsert", merge_strategy="delta", merge_keys=["order_id"])mode: scd1_upsert
merge_strategy: delta_by_partition
merge_keys: [order_id]
merge_partition_column: order_dateresult = ingest(
...,
mode="scd1_upsert",
merge_strategy="delta_by_partition",
merge_keys=["order_id"],
merge_partition_column="order_date",
)mode: scd1_upsert
merge_strategy: replace_partitions
merge_partition_column: order_date
replace_partitions_source_complete: trueresult = ingest(
...,
mode="scd1_upsert",
merge_strategy="replace_partitions",
merge_partition_column="order_date",
replace_partitions_source_complete=True,
)Partition replacement requires evidence
Use replace_partitions only when the source is known to be complete for every partition in the batch. If that cannot be proven, use a merge strategy instead.