What is a token in a language model?
Why C is the answer
A small unit of text that the model processes
Text is split into tokens, which can be words or parts of words. Usage and limits are measured in tokens.
In detail
- A token is a small unit of text that the model processes.
- Limits and pricing are often stated in tokens.
Use case
An illustrative scenario, not a real incident.
A team estimates cost by counting tokens, not words, because a token is the small unit of text that a model processes.
See it in different ways
flowchart LR A["Prompt"] --> B["Split into tokens"] B --> C["Model predicts the next token"] C --> D["Add it to the text"] D --> C D --> E["Response"]
stateDiagram-v2 [*] --> Tokens: prompt becomes tokens Tokens --> Predict Predict --> Append: pick the next token Append --> Predict: not finished Append --> [*]: stop or token limit
flowchart LR S0["Sentence"] --> S1["Tokens"] --> S2["Model"]
Key point: A token is a small unit of text.
Why the other options are wrong
- A. One complete document in the training data
- A document holds many tokens.
- B. A password used to sign in to the model
- Here, token does not mean a credential.
- D. A full sentence that has been checked for grammar
- A token is usually smaller than a word or sentence.