Before a model can train on text, what must the text become?
Why A is the answer
Token IDs
Tokenization maps text to the IDs that the model embeds.
In detail
- Tokenization maps text to IDs.
- Models read numbers, not letters.
Use case
An illustrative scenario, not a real incident.
A tokenizer maps text to IDs, and the model embeds those IDs.
See it in different ways
flowchart LR A["Collect and tokenize"] --> B["Pretraining"] B --> C["Fine-tuning"] C --> D["RLHF"] D --> E["Evaluation"]
stateDiagram-v2 [*] --> Tokenize Tokenize --> Pretraining Pretraining --> FineTuning FineTuning --> RLHF RLHF --> Evaluation Evaluation --> [*]
flowchart LR S0["Text"] --> S1["Tokens"] --> S2["Token IDs"] --> S3["Embeddings"]
Key point: Text becomes token IDs.
Why the other options are wrong
- B. Images of each document page
- Images are not the input format.
- C. Rows of sorted word counts
- Sorted counts lose the sequence.
- D. Encrypted strings
- Encryption is not part of training.