How does inference typically differ from training?
Why B is the answer
Inference serves requests with a focus on latency
Training learns weights over long runs. Inference applies them to requests, usually under latency targets.
In detail
- Inference serves live requests, so latency is the focus.
- Each has different hardware priorities.
Use case
An illustrative scenario, not a real incident.
Training a model takes weeks. Serving it means answering each user within moments.
See it in different ways
flowchart TD A["Training"] --> B["Large clusters, fast networks, long jobs"] C["Inference"] --> D["Low latency, many requests, scaling with demand"]
mindmap
root((AI))
Machine learning
Deep learning
Generative AI
Drivers
GPUs
Data
Algorithms
Stages
Training
Inferencemindmap
root(("AI workloads"))
n1["Training"]
n2["Long runs"]
n3["Learn weights"]
n4["Inference"]
n5["Requests"]
n6["Latency targets"]Key point: Inference focuses on latency.
Why the other options are wrong
- A. Inference always needs more GPUs than training does
- Inference usually needs fewer GPUs than large-scale training.
- C. Inference updates weights continuously from new data
- Serving does not update weights.
- D. Inference needs no memory beyond the input batch
- Inference still needs model memory.