Validation loss worsens early when fine-tuning on a small dataset. What helps?
Why B is the answer
Early stopping and a lower learning rate
Small datasets overfit quickly.
In detail
- Small datasets overfit quickly.
- Early stopping and a lower rate help.
- Keep a validation set.
Use case
An illustrative scenario, not a real incident.
A small dataset overfits within an epoch. Early stopping and a lower learning rate help.
See it in different ways
stateDiagram-v2 [*] --> Need Need --> Retrieval: facts that change Need --> FineTuning: stable style or format FineTuning --> Lora: limited GPU budget FineTuning --> FullTuning: budget allows it Lora --> EarlyStopping: small dataset
flowchart LR S0["Lower the learning rate"] --> S1["Watch validation loss"] --> S2["Stop early"]
Key point: Small datasets overfit quickly.
Why the other options are wrong
- A. A larger sampling temperature during training
- Temperature is for generation.
- C. More epochs and a higher learning rate
- That makes it worse.
- D. Removing the validation set from the run
- You would lose sight of the problem.