A 2 TB table changes by about 1% a day and a full reload misses the window. What do you implement?
Why A is the answer
An incremental load based on a watermark or change data
Load only what changed. Throwing capacity at a full reload solves the wrong problem.
In detail
- With a 1 percent change, load only what changed.
- Store the last loaded value, so a failed run can resume safely.
Use case
An illustrative scenario, not a real incident.
A nightly reload of a 2 TB orders table overruns its window. Loading only the roughly 20 GB that changed finishes well inside it.
See it in different ways
flowchart LR A["Read the last watermark"] --> B["Select newer rows"] B --> C["Merge into the target"] C --> D["Save the new watermark"]
sequenceDiagram participant P as Pipeline participant S as Source participant T as Delta table P->>P: Reads the last watermark P->>S: Rows newer than the watermark S-->>P: Changed rows P->>T: MERGE P->>P: Saves the new watermark
flowchart TD
Q{"How much changes each day?"}
Q -->|"A small share"| A0["Incremental load"]
Q -->|"Most of the table"| A1["Full reload"]Key point: Load only what changed.
Why the other options are wrong
- B. A copy of the table into a second lakehouse each night
- A second copy adds cost and no new data.
- C. A larger capacity with the same full reload each night
- More capacity keeps the same wasteful reload.
- D. A weekly full reload with a daily summary table
- A weekly reload leaves data out of date.