A data collection job failed overnight, and nobody noticed for days. What was missing?
Why A is the answer
Monitoring and alerts on the job
Data collection is a software process like any other, so it needs monitoring and alerts.
In detail
- Monitoring and alerts catch failed jobs quickly.
- Alert on failure, and on missing data.
Use case
An illustrative scenario, not a real incident.
A nightly collection job breaks silently. An alert would have told someone that morning.
See it in different ways
mindmap
root((LLM service))
Hardware
GPU memory
Throughput
Batching
Experience
Streaming
Conversation history
Packaging
NIMsequenceDiagram participant P0 as Collection job participant P1 as Monitoring participant P2 as Engineer P0->>P1: Run failed P1->>P2: Alert
Key point: Monitor data collection jobs.
Why the other options are wrong
- B. A larger batch size for the data loader
- Batch size does not report failures.
- C. A newer version of the embedding model
- A model version does not report failures.
- D. A longer system prompt for the chatbot
- A prompt does not report failures.