Which sampling setting keeps only the smallest token set covering a set probability?
Why B is the answer
Top-p
Top-p, or nucleus sampling, cuts on cumulative probability. Top-k cuts on count.
In detail
- Top-p cuts on cumulative probability.
- Top-k cuts on count instead.
Use case
An illustrative scenario, not a real incident.
A top-p of 0.9 keeps the smallest set of tokens that covers 90% of the probability.
See it in different ways
sequenceDiagram participant B as Batch participant M as Model participant C as KV cache B->>M: Newest token only C-->>M: Past keys and values M->>C: Adds new keys and values M-->>B: Next token
flowchart TD
Q{"What is the limit?"}
Q -->|"Cumulative probability"| A0["Top-p"]
Q -->|"A token count"| A1["Top-k"]Key point: Top-p cuts on cumulative probability.
Why the other options are wrong
- A. Maximum output tokens
- That limits length.
- C. Top-k
- Top-k cuts on count.
- D. Repetition penalty
- That discourages repeats.