Why partition a data lake by date?
Why B is the answer
So that queries scan only the needed data
Partition pruning cuts both cost and time.
In detail
- Partitions let queries skip data they do not need.
- Partition on columns that queries filter by.
Use case
An illustrative scenario, not a real incident.
A team asks about one week, and Athena reads only those date folders, rather than ten years.
See it in different ways
flowchart LR A["Raw CSV"] --> B["Convert to Parquet"] B --> C["Partition by date"] C --> D["Queries scan less data"]
stateDiagram-v2 [*] --> RawCsv RawCsv --> Parquet: columnar, compressed Parquet --> Partitioned: by date Partitioned --> Compacted: fewer small files Compacted --> [*]
flowchart TD
Q{"Does the filter use the partition?"}
Q -->|"Yes"| A0["Few files scanned"]
Q -->|"No"| A1["Everything scanned"]Key point: Partition pruning cuts cost and time.
Why the other options are wrong
- A. So that the catalog needs no table schema
- A schema is still needed.
- C. So that files are encrypted with a new key
- It has nothing to do with keys.
- D. So that S3 stores each object twice
- It does not store objects twice.