After launch, you want to catch quality problems from real traffic. What do you set up?
Why C is the answer
Monitoring and logging of requests, with evaluation on samples
Log inference traffic and evaluate samples of it, so quality issues show up before users report them.
In detail
- Log inference traffic, then evaluate samples of it.
- Quality problems then show up before users report them.
- Feed failing samples back into your evaluation set.
Use case
An illustrative scenario, not a real incident.
A chatbot launches. Requests are logged, and a sample is scored each day, so a fall in quality after a policy change shows up before customers complain.
See it in different ways
flowchart LR A["Requests"] --> B["AI Gateway"] B --> C["Rate limits"] B --> D["Inference tables"] B --> E["Usage tables"] D --> F["Agent Monitoring"]
sequenceDiagram participant U as Caller participant G as AI Gateway participant E as Endpoint participant T as Inference tables U->>G: Request G->>G: Apply rate limits G->>E: Forward E-->>U: Response E->>T: Record the request and response
stateDiagram-v2 state "Live traffic" as s0 state "Logged requests" as s1 state "Evaluate samples" as s2 state "Fix and release" as s3 [*] --> s0 s0 --> s1 s1 --> s2 s2 --> s3: problem found s3 --> s0
Key point: Log live traffic, and evaluate samples of it.
Why the other options are wrong
- A. A second vector index kept as a warm standby
- A standby index adds resilience, not visibility.
- B. A once-a-year review of the prompts and the model
- A yearly review finds problems far too late.
- D. A bigger cluster to handle the extra traffic
- More capacity does not reveal quality problems.