A new inference server is about to go live. What should its test confirm?
Why B is the answer
It meets latency and throughput targets under realistic load
One quiet request says little about production behavior.
In detail
- A test should use realistic load against targets.
- Use real prompt lengths.
Use case
An illustrative scenario, not a real incident.
A server handles one request well, but fails at 100. A realistic load test finds that before launch.
See it in different ways
flowchart LR A["Users"] --> B["API"] B --> C["Batched GPU inference"] C --> D["Streamed tokens"] B --> E["Latency and error monitoring"]
mindmap
root((LLM service))
Hardware
GPU memory
Throughput
Batching
Experience
Streaming
Conversation history
Packaging
NIMflowchart TD
Q{"Does it meet targets under load?"}
Q -->|"Yes"| A0["Launch"]
Q -->|"No"| A1["Fix and test again"]Key point: Test under realistic load.
Why the other options are wrong
- A. It uses the same chassis as the previous server
- A chassis says nothing.
- C. It runs one prompt correctly on a quiet afternoon
- One prompt says little.
- D. It has the most recent operating system release
- An OS version says little.