Why must you use the same tokenizer for training and inference?
Why A is the answer
Token IDs must map to the embeddings the model learned
Different mappings feed the model the wrong tokens.
In detail
- Token IDs must map to the embeddings the model learned.
- Save the tokenizer with the model.
Use case
An illustrative scenario, not a real incident.
A different tokenizer at inference gives wrong IDs, and output turns to nonsense.
See it in different ways
flowchart LR A["Raw text"] --> B["Clean and deduplicate"] B --> C["Split without leakage"] C --> D["Tokenize"] D --> E["Truncate and pad"]
mindmap
root((Data preparation))
Cleaning
Remove near duplicates
Splitting
Avoid leakage
Tokenizing
Same tokenizer everywhere
PaddingsequenceDiagram participant P0 as Tokenizer participant P1 as Model P0->>P1: Token IDs P1-->>P0: Embeddings match only if IDs do
Key point: Use the same tokenizer.
Why the other options are wrong
- B. The tokenizer is what decides the model's learning rate
- It has no effect on learning rate.
- C. A different tokenizer always raises the GPU temperature
- It has no effect on temperature.
- D. Inference does not run if the file name has changed
- File names do not matter here.