A Spark notebook fails with an out-of-memory error. What is a sensible first step?
Why D is the answer
Review the Spark logs and data skew
Logs show which stage failed, and skew is a common cause.
In detail
- Logs show where memory ran out.
- Skew is a common cause.
- Skew makes one task do most of the work.
Use case
An illustrative scenario, not a real incident.
A notebook runs out of memory. Logs show one join key holding most rows, so the fix is the skew, not a larger pool.
See it in different ways
flowchart TD
A["Monitor hub"] --> B{"Which item failed?"}
B -->|"Pipeline"| C["Activity error output"]
B -->|"Notebook"| D["Spark logs"]
B -->|"Capacity pressure"| E["Capacity Metrics app"]mindmap
root((Monitoring))
Activities
Monitor hub
Capacity
Capacity Metrics app
Warehouse
Query insights
Spark
Logs
Copy step
Run detailsflowchart TD
Q{"What do the logs show?"}
Q -->|"Uneven keys"| A0["Fix the skew"]
Q -->|"Even data, too much of it"| A1["Consider more memory"]Key point: Check logs and skew before you scale.
Why the other options are wrong
- A. Add a sensitivity label to the notebook
- A label has no effect on memory.
- B. Add a shortcut to the source table
- A shortcut has no effect on memory.
- C. Switch the notebook language to Scala
- A language change does not fix skew.