Spark jobs fail because one partition holds most of the data. What is this?
Why C is the answer
Data skew
Skew overloads one task. Repartitioning or salting can help.
In detail
- One partition holding most of the data is data skew.
- Repartitioning or salting spreads the keys.
Use case
An illustrative scenario, not a real incident.
One customer has most of the rows, so one Spark task runs out of memory. That is data skew.
See it in different ways
mindmap
root((Transform))
Glue
Serverless Spark
Job bookmarks
EMR
Custom tuning
Lambda
Small files
DataBrew
No code
Redshift
Stored proceduresmindmap
root(("Spark job"))
n1["Partition 1"]
n2["90% of rows"]
n3["Partition 2"]
n4["5% of rows"]
n5["Partition 3"]
n6["5% of rows"]Key point: Skew overloads one task.
Why the other options are wrong
- A. Overfitting
- That is a machine learning problem.
- B. Data leakage
- That is an ML problem as well.
- D. Schema drift
- That is a change in columns.