What does self-attention let a transformer do?
Why D is the answer
Weigh how much each token relates to every other token
Attention computes relevance between tokens, which is how context gets captured across a sequence.
In detail
- Self-attention weighs how much each token relates to the others.
- Each token gets a weighted view of the others.
Use case
An illustrative scenario, not a real incident.
In "the trophy did not fit in the case because it was too big", attention helps link "it" to "trophy".
See it in different ways
flowchart LR A["Tokens"] --> B["Embeddings plus position"] B --> C["Self-attention"] C --> D["Feed-forward layers"] D --> E["Next-token probabilities"]
sequenceDiagram participant T as Token participant O as Other tokens participant R as Result T->>O: How related are we? O-->>T: Scores T->>R: Weighted mix of the others
mindmap
root(("Token it"))
n1["Strong link"]
n2["trophy"]
n3["Weak link"]
n4["case"]
n5["because"]Key point: Attention weighs how tokens relate.
Why the other options are wrong
- A. Store the training data inside the model for later lookup
- A model does not store training data for lookup.
- B. Run without any GPU memory by computing tokens in sequence
- Transformers need GPU memory.
- C. Skip tokenization entirely by reading raw characters
- Tokenization still happens.