You must preprocess terabytes of streaming and batch data at scale. What fits?
Why A is the answer
Dataflow
Dataflow runs large batch and streaming pipelines, so it suits preprocessing at this scale.
In detail
- Dataflow processes batch and streaming data at scale.
- Dataflow runs Apache Beam pipelines.
Use case
An illustrative scenario, not a real incident.
A pipeline cleans terabytes of events, in batch and streaming, using Dataflow.
See it in different ways
flowchart TD
A{"Data size and shape"} -->|"Fits in memory"| B["In-memory Python"]
A -->|"Tabular, SQL-friendly"| C["BigQuery"]
A -->|"Large or streaming pipelines"| D["Dataflow"]
A -->|"Existing Spark code"| E["Apache Spark"]mindmap
root((ML data))
Processing
Dataflow
Features
Feature Store
Prototyping
Workbench
Colab Enterprise
Tracking
Experiments
ML Metadata lineageflowchart LR S0["Events and files"] --> S1["Dataflow"] --> S2["Clean features"]
Key point: Dataflow handles large pipelines.
Why the other options are wrong
- B. An in-memory pandas script on a single notebook
- Terabytes do not fit in pandas on one machine.
- C. A single notebook run each morning
- One notebook run does not scale.
- D. A Cloud Run service that holds all of the data in memory
- Memory is too small for that volume.