Why remove near-duplicate documents before training?
Why D is the answer
They bias the model, and can leak into the test split
Duplicates over-represent content, and across splits they inflate scores.
In detail
- Duplicates over-represent content.
- Across splits, they inflate scores.
- Remove near-duplicates before splitting.
Use case
An illustrative scenario, not a real incident.
The same text appears many times, and in both splits, so scores look better than they are.
See it in different ways
flowchart LR A["Raw text"] --> B["Clean and deduplicate"] B --> C["Split without leakage"] C --> D["Tokenize"] D --> E["Truncate and pad"]
mindmap
root((Data preparation))
Cleaning
Remove near duplicates
Splitting
Avoid leakage
Tokenizing
Same tokenizer everywhere
Paddingflowchart LR S0["Find near-duplicates"] --> S1["Remove them"] --> S2["Split the data"]
Key point: Duplicates bias training and inflate scores.
Why the other options are wrong
- A. They make the tokenizer forget how to split words
- Tokenizers do not forget.
- B. They cause the GPU to run at a lower clock speed
- Clock speed is unrelated.
- C. They stop the model from loading any checkpoints
- Checkpoints are unrelated.