Spark on Glue reads many tiny files slowly. What should you do?
Why C is the answer
Compact them into larger files
Too many small files add overhead.
In detail
- Compacting reduces the number of files to open.
- Schedule compaction as a regular job.
Use case
An illustrative scenario, not a real incident.
Spark spends more time opening files than reading them. Compaction turns thousands of tiny files into a few large ones.
See it in different ways
flowchart LR A["Raw CSV"] --> B["Convert to Parquet"] B --> C["Partition by date"] C --> D["Queries scan less data"]
stateDiagram-v2 [*] --> RawCsv RawCsv --> Parquet: columnar, compressed Parquet --> Partitioned: by date Partitioned --> Compacted: fewer small files Compacted --> [*]
stateDiagram-v2 state "Thousands of tiny files" as s0 state "A few large files" as s1 state "Faster reads" as s2 [*] --> s0 s0 --> s1: compact s1 --> s2 s2 --> [*]
Key point: Compact small files.
Why the other options are wrong
- A. Store them in S3 Glacier
- Glacier makes them harder to read.
- B. Add more crawlers on the same path
- Crawlers read metadata, and do not speed up jobs.
- D. Split them into smaller files
- Smaller files make it worse.