After deployment, p99 latency climbs slowly while GPU utilization stays flat. What do you examine first?
Why A is the answer
Request queue depth and batch behavior
Rising tail latency with steady utilization suggests time spent queueing, not computing.
In detail
- Rising tail latency with flat utilization suggests queuing.
- Average latency can hide this.
Use case
An illustrative scenario, not a real incident.
Tail latency grows, but utilization is flat. Requests are waiting in a queue, so the team reviews queue depth and batching.
See it in different ways
flowchart LR
A["Slow training"] --> B["Nsight Systems timeline"]
B --> C{"Where is the time?"}
C -->|"Data loading"| D["Fix the input pipeline"]
C -->|"GPU kernels"| E["Inspect with Nsight Compute"]sequenceDiagram participant U as Users participant K as Kubernetes participant T as Triton or NIM participant G as GPUs U->>K: Requests K->>K: Scales replicas on load K->>T: Routes the request T->>G: Batched inference
flowchart TD
Q{"Is GPU utilization rising?"}
Q -->|"Yes"| A0["Compute is the limit"]
Q -->|"No"| A1["Check queue and batching"]Key point: Flat utilization with rising latency means queueing.
Why the other options are wrong
- B. The model's license and usage terms
- A license has no effect on latency.
- C. The tokenizer's vocabulary size
- Vocabulary does not change over time.
- D. The fine-tuning parameters of the model
- Tuning does not change after deployment.