Why do transformers need positional information?
Why B is the answer
Because attention alone does not capture token order
Attention treats tokens as a set, so order has to be added through positional encoding.
In detail
- Attention treats tokens as a set, so order has to be added.
- Attention alone sees tokens as a set.
Use case
An illustrative scenario, not a real incident.
"Dog bites man" and "man bites dog" use the same words. Positional information tells them apart.
See it in different ways
flowchart LR A["Tokens"] --> B["Embeddings plus position"] B --> C["Self-attention"] C --> D["Feed-forward layers"] D --> E["Next-token probabilities"]
sequenceDiagram participant T as Token participant O as Other tokens participant R as Result T->>O: How related are we? O-->>T: Scores T->>R: Weighted mix of the others
flowchart LR S0["Token embeddings"] --> S1["Add positional information"] --> S2["Attention layers"]
Key point: Positions supply word order.
Why the other options are wrong
- A. To reduce the vocabulary size before embeddings are computed
- It has no effect on vocabulary size.
- C. To speed up the GPU clock during matrix multiplication
- It has no effect on clock speed.
- D. To remove stop words from the input sequence
- It does not remove words.