A new model version must reach production with low risk and quick recovery. What do you do?
Why C is the answer
Roll out progressively to a share of traffic with a rollback path
A progressive rollout limits the blast radius, and keeping the old version ready makes rollback immediate.
In detail
- A share of traffic limits the impact.
- A rollback path gives quick recovery.
- Decide the metrics that trigger a rollback.
Use case
An illustrative scenario, not a real incident.
A forecast model goes to 10% of traffic first, with a rollback ready. A problem would reach few users.
See it in different ways
flowchart LR
A["Blue: current"] -->|"90%"| C["Traffic"]
B["Green: new"] -->|"10%"| C
C --> D{"Healthy?"}
D -->|Yes| E["Increase green"]
D -->|No| F["Back to blue"]sequenceDiagram
participant E as Evaluation
participant N as New deployment
participant O as Old deployment
participant T as Traffic
E->>N: Passes checks
T->>N: Small share
T->>O: The rest
alt Metrics hold
T->>N: Increase the share
else Metrics fall
T->>O: Roll back
endstateDiagram-v2 state "10% of traffic" as s0 state "50% of traffic" as s1 state "100% of traffic" as s2 state "Rolled back" as s3 [*] --> s0 s0 --> s1: metrics hold s1 --> s2: metrics hold s0 --> s3: metrics fall s2 --> [*] s3 --> [*]
Key point: Roll out progressively, with a way back.
Why the other options are wrong
- A. Replace the old deployment in one step during a quiet period
- One step puts all users at risk.
- B. Ask a small group of users to report any problems they see
- Asking users is not a control.
- D. Deploy on a Friday when traffic is lower and the team can watch it
- A quiet day is not a rollout strategy.