In scaled dot-product attention, what determines how much a token contributes?
Why C is the answer
The match between query and key
Query and key scores become weights on the values.
In detail
- Query and key scores become weights on the values.
- The scores become weights.
Use case
An illustrative scenario, not a real incident.
A token's query matches another's key, so that token's value gets a large weight.
See it in different ways
flowchart LR A["Input embeddings"] --> B["Query, key, value"] B --> C["Several attention heads"] C --> D["Concatenate"] D --> E["Linear layer"]
mindmap
root((Transformer block))
Attention
Query, key, value
Several heads
Stability
Residual connections
Normalization
Feed-forward
Per tokenflowchart LR S0["Compare query and key"] --> S1["Scale and normalize"] --> S2["Weight the values"]
Key point: Query and key decide the weight.
Why the other options are wrong
- A. The length of the output so far
- Output length is not the measure.
- B. The token's byte size in UTF-8
- Byte size is irrelevant.
- D. The token's frequency in the corpus
- Corpus frequency is not used here.