A Glue job reprocesses all files each night, although few are new. What avoids this?
Why C is the answer
Job bookmarks
Job bookmarks track what has already been processed.
In detail
- Job bookmarks track what has been processed.
- Cost falls with the work.
Use case
An illustrative scenario, not a real incident.
A Glue job reads 10,000 files nightly, although only 50 are new. Job bookmarks skip what it has already processed.
See it in different ways
mindmap
root((Transform))
Glue
Serverless Spark
Job bookmarks
EMR
Custom tuning
Lambda
Small files
DataBrew
No code
Redshift
Stored proceduresflowchart TD
Q{"Has the job already processed it?"}
Q -->|"Yes"| A0["Skip"]
Q -->|"No"| A1["Process"]Key point: Bookmarks track processed data.
Why the other options are wrong
- A. A longer timeout for the job
- A timeout does not skip files.
- B. A second crawler on the same bucket
- A crawler does not skip files.
- D. A larger number of DPUs for each run
- More DPUs repeat the same work faster.