How does evaluation differ from monitoring?
Why A is the answer
Evaluation is before release, and monitoring is on live traffic
Evaluation tests a version before release on known data. Monitoring watches real traffic after release.
In detail
- Evaluation tests a version before release.
- Monitoring watches live traffic after release.
- Failures from monitoring feed the next evaluation.
- Each failure found in monitoring becomes a new test case.
Use case
An illustrative scenario, not a real incident.
Before launch, a team scores its agent on a test set. After launch, it watches live traffic for problems the test set missed.
See it in different ways
flowchart LR A["Evaluate on a test set"] --> B["Deploy"] B --> C["Monitor live traffic"] C --> D["Add failures to the test set"] D --> A
sequenceDiagram participant U as Caller participant G as AI Gateway participant E as Endpoint participant T as Inference tables U->>G: Request G->>G: Apply rate limits G->>E: Forward E-->>U: Response E->>T: Record the request and response
flowchart TD
Q{"When is it happening?"}
Q -->|"Before release"| A0["Evaluation on a test set"]
Q -->|"After release"| A1["Monitoring of live traffic"]Key point: Evaluate before release, and monitor after it.
Why the other options are wrong
- B. They are two names for exactly the same activity
- They happen at different stages, for different reasons.
- C. Evaluation only happens after a user complains
- Evaluation should come before users are affected.
- D. Monitoring only happens during development
- Monitoring needs live traffic.