Model quality slowly degrades in production with no change to code. What should you set up?
Why A is the answer
Amazon SageMaker Model Monitor with drift alerts
Model Monitor compares live data with a baseline and alerts on drift, which is the usual cause here.
In detail
- With no code change, slow decline suggests drift.
- Model Monitor detects it.
- AWS closed SageMaker Model Monitor to new customers. Existing customers can keep using it.
- The code has not changed, but the world has.
Use case
An illustrative scenario, not a real incident.
A pricing model weakens as customer behavior shifts. Model Monitor compares live data to a baseline and raises an alert.
See it in different ways
sequenceDiagram participant E as Endpoint participant S as Captured data in S3 participant M as Model Monitor participant C as CloudWatch E->>S: Capture requests and responses M->>S: Compare with the baseline M->>C: Publish violations C-->>E: Alarm starts retraining
sequenceDiagram participant E as Endpoint participant S as S3 participant M as Model Monitor participant C as CloudWatch E->>S: Data capture M->>S: Compares with the baseline M->>C: Publishes metrics C-->>M: Alarm on drift
flowchart LR S0["Live traffic"] --> S1["Compare with baseline"] --> S2["Drift found"] --> S3["Alert"]
Key point: Monitor live data against a baseline.
Why the other options are wrong
- B. A bigger instance type with more memory for the endpoint
- More memory does not fix drift.
- C. A new VPC with stricter routing for inference traffic
- Network changes do not fix drift.
- D. A larger S3 bucket to retain more inference logs
- Storing logs does not detect drift.