A model is too large for your GPUs, and a teacher model already performs well. Which technique trains a smaller model to imitate it?
Why A is the answer
Knowledge distillation
Distillation trains a student on the teacher's outputs.
In detail
- Distillation trains a student on a teacher's outputs.
- The student may lose some quality.
Use case
An illustrative scenario, not a real incident.
A strong model is too large to deploy, so a smaller student learns from its outputs.
See it in different ways
flowchart TD
A{"Which goal?"} -->|"Fewer bits per weight"| B["Quantization"]
A -->|"Remove low-value weights"| C["Pruning"]
A -->|"Teach a smaller model"| D["Knowledge distillation"]stateDiagram-v2 [*] --> Goal Goal --> Quantization: smaller weights Goal --> Pruning: fewer weights Goal --> Distillation: a smaller student model Goal --> Speculative: fewer main model steps
flowchart LR S0["Teacher model"] --> S1["Teacher outputs"] --> S2["Student trains"] --> S3["Smaller model"]
Key point: Distillation trains a student on a teacher.
Why the other options are wrong
- B. Layer normalization, applied across the teacher
- Normalization does not produce a smaller model.
- C. Beam search, used with a larger beam width
- Beam search is about decoding.
- D. Tokenizer retraining with a larger vocabulary
- A tokenizer does not shrink a model.