Agent replicas must scale out behind a load balancer. Where should conversation state live?
Why C is the answer
In an external store
Any replica can then serve the next request.
In detail
- External state lets any replica serve the next request.
- Sticky sessions are harder to scale.
Use case
An illustrative scenario, not a real incident.
Any replica can continue a conversation, because state is in an external store.
See it in different ways
flowchart LR A["Requests"] --> B["Load balancer"] B --> C["Agent replicas"] C --> D["External state store"] C --> E["GPU inference service"]
sequenceDiagram participant U as Users participant L as Rate and budget limits participant A as Agent replicas participant S as External state participant M as GPU inference U->>L: Requests L->>A: Within limits A->>S: Reads and saves state A->>M: Model calls, in parallel if independent
sequenceDiagram participant P0 as Load balancer participant P1 as Replica participant P2 as State store P0->>P1: Request P1->>P2: Load state P1->>P2: Save state
Key point: External state lets any replica serve.
Why the other options are wrong
- A. In the GPU driver's configuration
- Drivers do not hold state.
- B. In the model's pretrained weights
- Weights do not hold state.
- D. In each replica's local memory
- Local memory ties a user to one replica.