Index freshness and relevance drop while the model is unchanged. What should monitoring cover?
Why A is the answer
Data ingestion quality and search index health
Ingestion quality and index health decide what retrieval can return, and RAG quality depends on that.
In detail
- If the model is unchanged, look at the data around it.
- Monitor ingestion quality and index health.
- Monitor index freshness alongside the model.
Use case
An illustrative scenario, not a real incident.
Answers worsen with no model change. Monitoring shows an ingestion job failing, so the index is out of date.
See it in different ways
sequenceDiagram participant A as Agent or app participant O as OpenTelemetry participant M as Application Insights A->>O: Spans for each step O->>M: Exports the trace M-->>A: Debug and monitor
flowchart TD
Q{"Did the model change?"}
Q -->|"No"| A0["Check ingestion and the index"]
Q -->|"Yes"| A1["Compare versions"]Key point: If the model did not change, check the data.
Why the other options are wrong
- B. Only the number of users who sign in each day
- Sign-ins do not show index health.
- C. Only the size of the system prompt
- The prompt size has not changed.
- D. The temperature setting of the deployed model
- Temperature has not changed, and it does not affect the index.