New folders appear in S3, and the catalog must discover them. What do you use?
Why A is the answer
A Glue crawler
Crawlers infer schemas and update the catalog.
In detail
- A crawler discovers new data and updates the catalog.
- Schedule the crawler to match new data.
Use case
An illustrative scenario, not a real incident.
Each day creates a new S3 folder, and a Glue crawler adds it to the catalog as a partition.
See it in different ways
flowchart LR A["Data in S3"] --> B["Glue crawler"] B --> C["Glue Data Catalog"] C --> D["Athena"] C --> E["Redshift Spectrum"]
mindmap
root((Stores))
Redshift
Complex analytics
DynamoDB
Fast key lookups
S3
Data lake
Athena
SQL on S3flowchart LR S0["New folder in S3"] --> S1["Crawler runs"] --> S2["Catalog updated"] --> S3["Athena sees it"]
Key point: Crawlers discover schemas and partitions.
Why the other options are wrong
- B. An S3 Inventory report
- That report lists objects, and does not update tables.
- C. An Athena CTAS query
- CTAS creates a table from a query.
- D. A Lake Formation grant
- A grant controls access.