A daily pipeline has many dependent steps, with retries and alerting. What is the standard tool?
Why C is the answer
Managed Service for Apache Airflow with a DAG
Managed Service for Apache Airflow, formerly Cloud Composer, is built for dependent workflows.
In detail
- Managed Service for Apache Airflow runs DAGs with dependencies, retries and alerting.
- Google renamed Cloud Composer to this service, so older material uses the old name.
Use case
An illustrative scenario, not a real incident.
Forty dependent steps run each night, with retries and alerts, in a Managed Service for Apache Airflow DAG.
See it in different ways
flowchart LR A["Extract"] --> B["Transform"] A --> C["Validate"] B --> D["Load"] C --> D
sequenceDiagram participant C as Managed Service for Apache Airflow participant J as Dataproc job participant L as Cloud Logging participant O as Operator C->>J: Start the step J--xC: Fails C->>J: Retry J->>L: Logs C-->>O: Alert if retries run out
flowchart LR S0["Extract"] --> S1["Validate"] --> S2["Transform"] --> S3["Publish"]
Key point: Managed Airflow runs dependent workflows.
Why the other options are wrong
- A. Cloud Scheduler calling each step
- Scheduler does not manage dependencies.
- B. Cloud Run jobs with no dependencies
- No dependencies is the opposite of the need.
- D. A cron job on a Compute Engine VM
- Cron has no dependency handling.