You want to compare a new model against the current one with real traffic and limited risk. What do you do?
Why B is the answer
Send a small share of traffic in a canary
A canary exposes the new version to a small share of real traffic, with an easy rollback.
In detail
- A canary sends a small share of real traffic to the new model.
- Decide the success metric first.
Use case
An illustrative scenario, not a real incident.
A new ranking model gets 5% of traffic, and a rollback is one step away.
See it in different ways
flowchart LR
A["Traffic"] -->|"95%"| B["Current model"]
A -->|"5%"| C["New model"]
C --> D{"Metrics better?"}
D -->|Yes| E["Increase traffic"]
D -->|No| F["Roll back"]sequenceDiagram participant T as Traffic participant C as Current model participant N as New model participant M as Metrics T->>C: Most requests T->>N: Small share C->>M: Outcomes N->>M: Outcomes M-->>T: Shift or roll back
stateDiagram-v2 state "Canary at 5%" as s0 state "Full traffic" as s1 state "Rolled back" as s2 [*] --> s0 s0 --> s1: metrics hold s0 --> s2: metrics fall s1 --> [*] s2 --> [*]
Key point: Canaries test with limited risk.
Why the other options are wrong
- A. Add more replicas and move all traffic at once
- Moving everything at once is the risk you want to avoid.
- C. Replace the current model and watch the dashboards
- Replacing it risks all users.
- D. Test only on the held-out training data
- Held-out data is not real traffic.