Loss swings wildly and never settles. Which first change is sensible?
Why C is the answer
Lower the learning rate
Steps that are too large overshoot the minimum.
In detail
- Steps that are too large overshoot the minimum.
- Change one setting at a time.
Use case
An illustrative scenario, not a real incident.
Loss jumps around, because each step overshoots. Lowering the learning rate settles it.
See it in different ways
flowchart TD
A{"Training loss falling?"} -->|No| B["Underfitting or a learning-rate problem"]
A -->|Yes| C{"Validation loss falling?"}
C -->|Yes| D["Keep training"]
C -->|"No, it rises"| E["Overfitting: regularize or stop early"]stateDiagram-v2 [*] --> Watch Watch --> Healthy: both losses fall Watch --> Overfitting: validation loss rises Watch --> Unstable: loss swings wildly Overfitting --> Regularize Unstable --> LowerLearningRate Regularize --> Watch LowerLearningRate --> Watch
flowchart LR S0["Loss swings"] --> S1["Lower learning rate"] --> S2["Watch loss"]
Key point: Large steps overshoot.
Why the other options are wrong
- A. Raise the learning rate by a factor of ten
- That makes it worse.
- B. Stop shuffling the training data
- Shuffling usually helps.
- D. Remove the validation set from the training run
- Validation is how you see problems.