Which advantage does a transformer have over a plain RNN on long sequences?
Why A is the answer
It attends directly across distant tokens, in parallel
RNNs pass state step by step, so distant context fades and work is sequential.
In detail
- Attention reaches distant tokens directly, and tokens can be processed in parallel.
- RNNs pass state one step at a time.
Use case
An illustrative scenario, not a real incident.
In a long contract, a late clause depends on a term from page one. Attention can reach it directly.
See it in different ways
flowchart LR A["Tokens"] --> B["Embeddings plus position"] B --> C["Self-attention"] C --> D["Feed-forward layers"] D --> E["Next-token probabilities"]
sequenceDiagram participant T as Token participant O as Other tokens participant R as Result T->>O: How related are we? O-->>T: Scores T->>R: Weighted mix of the others
flowchart TD
Q{"Which architecture?"}
Q -->|"Plain RNN"| A0["Step by step"]
Q -->|"Transformer"| A1["Direct attention, in parallel"]Key point: Transformers attend directly across distance.
Why the other options are wrong
- B. It needs no training data, since attention is fixed
- Transformers need training data.
- C. It stores a separate model for each sentence length
- There is one model for all lengths.
- D. It processes tokens strictly one at a time
- That describes an RNN.