Why do many LLMs use residual connections?
Why D is the answer
They keep gradients flowing through deep networks
Skip paths make very deep stacks trainable.
In detail
- Skip paths keep gradients flowing through deep stacks.
- Without them, gradients can vanish.
Use case
An illustrative scenario, not a real incident.
A very deep model trains well, because skip paths keep gradients flowing.
See it in different ways
flowchart LR A["Input embeddings"] --> B["Query, key, value"] B --> C["Several attention heads"] C --> D["Concatenate"] D --> E["Linear layer"]
mindmap
root((Transformer block))
Attention
Query, key, value
Several heads
Stability
Residual connections
Normalization
Feed-forward
Per tokenflowchart LR S0["Input"] --> S1["Layer"] --> S2["Add the input back"] --> S3["Output"]
Key point: Residual connections help deep networks.
Why the other options are wrong
- A. They remove the need for positional information
- Position is still needed.
- B. They reduce the tokenizer's vocabulary at runtime
- Vocabulary is separate.
- C. They replace the attention layers completely
- Attention stays.