Spark jobs run nightly on Dataproc, and the cluster idles all day. What is better?
Why C is the answer
Job-based clusters
A cluster created for each job avoids paying for idle time.
In detail
- Job-based clusters run only for the job.
- Keep state in Cloud Storage.
Use case
An illustrative scenario, not a real incident.
A cluster sits idle for 20 hours a day. A job-based cluster starts for the work, and stops after it.
See it in different ways
flowchart TD
A{"Workload"} -->|"Scheduled jobs"| B["Job-based cluster: create, run, delete"]
A -->|"Always-on interactive use"| C["Persistent cluster"]sequenceDiagram participant C as Managed Service for Apache Airflow participant J as Dataproc job participant L as Cloud Logging participant O as Operator C->>J: Start the step J--xC: Fails C->>J: Retry J->>L: Logs C-->>O: Alert if retries run out
flowchart TD
Q{"When is it used?"}
Q -->|"A few hours a day"| A0["Job-based"]
Q -->|"All day"| A1["Persistent"]Key point: Use job-based clusters for scheduled work.
Why the other options are wrong
- A. A second always-on cluster
- That costs more.
- B. Longer logs
- Logs do not reduce idle time.
- D. A larger persistent cluster
- That costs more.