Traffic to an online model rises sharply each morning. What do you scale?
Why C is the answer
The serving backend, based on throughput
Scaling the serving backend on throughput keeps predictions fast as traffic rises.
In detail
- Scale the serving backend on throughput.
- Allow for time to scale.
Use case
An illustrative scenario, not a real incident.
Traffic rises each morning, and the serving backend scales on throughput.
See it in different ways
flowchart TD
A{"Is a caller waiting?"} -->|No| B["Batch inference"]
A -->|Yes| C{"How much control?"}
C -->|"Managed"| D["Agent Platform endpoint"]
C -->|"Own containers"| E["Cloud Run or GKE"]mindmap
root((Serving))
Online
Autoscaled backend
Batch
Overnight tables
Private endpoint
Internal only
Edge
No connection
Custom container
Any frameworkflowchart LR S0["Traffic rises"] --> S1["Throughput measured"] --> S2["Backend scales"]
Key point: Scale serving on throughput.
Why the other options are wrong
- A. The size of the test set
- A test set is about evaluation.
- B. The Feature Store retention period only
- Retention has no effect on serving capacity.
- D. The number of training epochs
- Epochs are about training.