You need to process events in near real time, and some arrive late. Which pairing fits?
Why A is the answer
Pub/Sub and Dataflow with windowing
Pub/Sub ingests the stream, and Dataflow handles windows and late data.
In detail
- Pub/Sub carries events, and Dataflow handles late data with windows.
- Watermarks estimate how complete a window is.
Use case
An illustrative scenario, not a real incident.
A delivery app streams location events, and some arrive late over poor signal. Pub/Sub takes them in, and Dataflow windows them correctly.
See it in different ways
flowchart LR
A["Events"] --> B["Pub/Sub"]
B --> C["Dataflow windows"]
C --> D{"Late event?"}
D -->|"Within allowed lateness"| E["Update the result"]
D -->|"Too late"| F["Drop or send apart"]
E --> G["BigQuery"]sequenceDiagram participant P as Producers participant S as Pub/Sub participant D as Dataflow participant B as BigQuery P->>S: Events S->>D: Delivered with event time D->>D: Windows and watermarks D->>B: Aggregates
flowchart LR S0["Events"] --> S1["Pub/Sub"] --> S2["Dataflow windows"] --> S3["BigQuery"]
Key point: Pub/Sub ingests, and Dataflow handles time.
Why the other options are wrong
- B. Cloud Storage and a nightly batch job
- A nightly batch is not near real time.
- C. Cloud SQL triggers
- Triggers are not a streaming pipeline.
- D. Transfer Appliance and BigQuery
- An appliance moves bulk data offline.