A multi-agent service must handle growing load. Where should you look first when scaling?
Why C is the answer
Stateless components, external state, and queueing for long tasks
Keeping services stateless, with external state, lets you add replicas freely.
In detail
- Stateless components, external state and queues let services scale.
- Stateless services scale more easily.
Use case
An illustrative scenario, not a real incident.
A team moves session state to an external store, and queues long tasks, so it can add replicas freely.
See it in different ways
flowchart LR A["Requests"] --> B["Load balancer"] B --> C["Agent replicas"] C --> D["External state store"] C --> E["GPU inference service"]
sequenceDiagram participant U as Users participant L as Rate and budget limits participant A as Agent replicas participant S as External state participant M as GPU inference U->>L: Requests L->>A: Within limits A->>S: Reads and saves state A->>M: Model calls, in parallel if independent
mindmap
root(("Multi-agent service"))
n1["Stateless"]
n2["Agent replicas"]
n3["External"]
n4["State store"]
n5["Task queue"]Key point: Stateless services scale.
Why the other options are wrong
- A. Adding more context to each prompt to reduce repeated tool calls
- More context makes each request slower.
- B. Removing logging and tracing to lower the overhead per request
- You need logging to run at scale.
- D. Storing session state in each process's memory for faster access
- State in a replica prevents scaling.