Traffic varies a lot during the day. How should an inference service respond?
Why A is the answer
Autoscale replicas from a load metric
Autoscaling follows demand. A fixed peak size wastes GPUs.
In detail
- Autoscaling follows demand.
- GPU instances can be slow to start.
Use case
An illustrative scenario, not a real incident.
Demand peaks at noon, so replicas scale on a load metric, and fall at night.
See it in different ways
flowchart LR A["Model"] --> B["Serving container"] B --> C["Kubernetes"] C --> D["Autoscaling replicas"] D --> E["Monitoring"]
sequenceDiagram participant U as Users participant K as Kubernetes participant T as Triton or NIM participant G as GPUs U->>K: Requests K->>K: Scales replicas on load K->>T: Routes the request T->>G: Batched inference
stateDiagram-v2 state "Few replicas" as s0 state "Many replicas" as s1 [*] --> s0 s0 --> s1: load rises s1 --> s0: load falls
Key point: Autoscale from a load metric.
Why the other options are wrong
- B. Run at peak size at all times, with no monitoring
- A fixed peak wastes GPUs.
- C. Reject all requests above the average load
- Rejecting users is a failure.
- D. Restart the service each hour
- Restarts do not add capacity.