The agent service scales, but requests queue at the model. What is the bottleneck?
Why D is the answer
GPU inference capacity
Scale the inference tier separately from the agent tier.
In detail
- Requests queue at the model, so inference capacity is the limit.
- Scale the two tiers separately.
Use case
An illustrative scenario, not a real incident.
Agent replicas are fine, but requests wait at the model, so GPU inference needs scaling.
See it in different ways
flowchart LR A["Requests"] --> B["Load balancer"] B --> C["Agent replicas"] C --> D["External state store"] C --> E["GPU inference service"]
sequenceDiagram participant U as Users participant L as Rate and budget limits participant A as Agent replicas participant S as External state participant M as GPU inference U->>L: Requests L->>A: Within limits A->>S: Reads and saves state A->>M: Model calls, in parallel if independent
flowchart TD
Q{"Where do requests wait?"}
Q -->|"At the model"| A0["Scale inference"]
Q -->|"At the agent tier"| A1["Scale agent replicas"]Key point: Scale inference separately.
Why the other options are wrong
- A. The size of the system prompt file
- Prompt file size is not the bottleneck.
- B. The number of tools listed in the docs
- Documentation is unrelated.
- C. The load balancer's idle timeout
- An idle timeout does not limit the model.