Bad records keep reaching downstream tables, and you want rules checked in the pipeline. What fits?
Why B is the answer
AWS Glue Data Quality
Glue Data Quality defines and evaluates rules inside your pipelines.
In detail
- Glue Data Quality checks rules inside the pipeline.
- Decide what happens to failing records.
Use case
An illustrative scenario, not a real incident.
Orders with negative totals keep reaching reports. A Glue Data Quality rule stops them inside the pipeline.
See it in different ways
stateDiagram-v2 [*] --> Load Load --> Validate: rules and row counts Validate --> Publish: passes Validate --> Quarantine: fails Quarantine --> Load: fix, then rerun safely Publish --> [*]
flowchart TD
Q{"Do records pass?"}
Q -->|"Yes"| A0["Continue"]
Q -->|"No"| A1["Quarantine or stop"]Key point: Check quality rules in the pipeline.
Why the other options are wrong
- A. AWS Glue DataBrew recipes run by hand
- Manual recipes are not automated checks.
- C. A bigger EMR cluster
- A bigger cluster does not check data.
- D. AWS Budgets
- Budgets tracks spending.