A batch contains duplicate order rows. Which PySpark call removes them by order ID?
Why C is the answer
dropDuplicates(["order_id"])
dropDuplicates keeps one row per key.
In detail
- dropDuplicates with a key keeps one row for each order ID.
- Say which columns define a duplicate.
Use case
An illustrative scenario, not a real incident.
A retry sends the same orders twice. dropDuplicates on order_id keeps one row per order before the load.
See it in different ways
sequenceDiagram participant P as Pipeline participant S as Source participant T as Delta table P->>P: Reads the last watermark P->>S: Rows newer than the watermark S-->>P: Changed rows P->>T: MERGE P->>P: Saves the new watermark
flowchart LR S0["Batch with repeats"] --> S1["dropDuplicates on order_id"] --> S2["One row per order"] --> S3["Load"]
Key point: dropDuplicates keeps one row per key.
Why the other options are wrong
- A. repartition(1)
- repartition changes partitions.
- B. printSchema()
- printSchema shows the columns.
- D. cache()
- cache() stores data in memory.