In the original Transformer, what does masked self-attention in the decoder prevent?
Why D is the answer
Attention to tokens that come later
During training the decoder must not see future tokens, so they are masked.
In detail
- During training, the decoder must not see tokens that come later.
- Otherwise it could copy the answer.
Use case
An illustrative scenario, not a real incident.
During training, a translation decoder must predict each word without seeing the next one, so later tokens are masked.
See it in different ways
flowchart LR A["Tokens"] --> B["Embeddings plus position"] B --> C["Self-attention"] C --> D["Feed-forward layers"] D --> E["Next-token probabilities"]
sequenceDiagram participant T as Token participant O as Other tokens participant R as Result T->>O: How related are we? O-->>T: Scores T->>R: Weighted mix of the others
mindmap
root(("Masked self-attention"))
n1["Visible"]
n2["Token 1"]
n3["Token 2"]
n4["Masked"]
n5["Token 4"]
n6["Token 5"]Key point: Masking hides future tokens.
Why the other options are wrong
- A. Weight updates during the inference phase
- Masking is not about weight updates.
- B. Sharing of embeddings between the encoder and the decoder
- Sharing embeddings is a separate choice.
- C. Padding tokens from reaching the loss function
- Padding is handled by a separate mask.