You have existing Spark jobs and want to lift them to Google Cloud with minimal rewriting. What do you use?
Why B is the answer
Dataproc
Dataproc runs Spark and Hadoop as a managed service, so existing jobs move with little change.
In detail
- Dataproc runs existing Spark with minimal rewriting.
- Google now calls Dataproc Managed Service for Apache Spark.
- Choose Dataflow for new pipelines if you have no Spark to keep.
Use case
An illustrative scenario, not a real incident.
A firm has years of Spark jobs, and moves them to Dataproc with few code changes.
See it in different ways
flowchart TD
A{"What do you have?"} -->|"New batch or streaming pipeline"| B["Dataflow with Apache Beam"]
A -->|"Existing Spark or Hadoop"| C["Dataproc"]
A -->|"Visual, code-free integration"| D["Cloud Data Fusion"]
A -->|"SQL transformations in BigQuery"| E["Dataform"]mindmap
root((Processing))
Dataflow
Apache Beam
Dataproc
Existing Spark
Data Fusion
Visual, code free
Dataform
SQL in BigQuery
Managed Airflow
DAG orchestrationflowchart TD
Q{"What do you have?"}
Q -->|"Existing Spark jobs"| A0["Dataproc"]
Q -->|"New pipelines"| A1["Consider Dataflow"]Key point: Dataproc runs existing Spark.
Why the other options are wrong
- A. Firestore
- Firestore is a database.
- C. Cloud Functions
- Functions do not run Spark.
- D. Bigtable
- Bigtable is a database.