Requests are throttled in one Region at peak, while volume is unpredictable. What should you evaluate first?
Why D is the answer
Cross-Region inference to spread demand
Cross-Region inference spreads traffic across Regions to handle bursts, without committing to fixed capacity.
In detail
- Cross-Region inference spreads demand during peaks.
- Check that data rules allow those Regions.
Use case
An illustrative scenario, not a real incident.
A shopping assistant hits limits during a sale. Cross-Region inference spreads traffic, with no capacity commitment.
See it in different ways
stateDiagram-v2 [*] --> Closed Closed --> Open: too many failures Open --> HalfOpen: after a wait HalfOpen --> Closed: test call works HalfOpen --> Open: test call fails
mindmap
root((Inference))
On demand
Variable traffic
Provisioned Throughput
Steady and high
Batch
Offline and cheaper
Cross-Region
Spreads peaks
Streaming
Show text as it arrivesflowchart TD
Q{"What is the demand like?"}
Q -->|"Unpredictable bursts"| A0["Cross-Region inference"]
Q -->|"Steady and high"| A1["Provisioned Throughput"]Key point: Cross-Region inference handles bursts.
Why the other options are wrong
- A. Moving to a smaller embedding size
- Embeddings do not affect this throttling.
- B. Disabling Guardrails
- Disabling safety is never the fix.
- C. Lowering max tokens to zero
- Zero tokens means no answer.