Training keeps failing partway through a long job. What is a useful first step?
Why D is the answer
Read the job logs for the failing step
Job logs show which step failed and why, so they come before any other change.
In detail
- The logs show which step failed, and why.
- Read logs before you change resources.
Use case
An illustrative scenario, not a real incident.
A long job dies at the same point each time, and the logs show a bad record there.
See it in different ways
sequenceDiagram participant B as Cloud Build participant P as Agent Platform Pipelines participant R as Model Registry participant E as Endpoint B->>P: Runs after a change P->>P: Validate data and train P->>R: Register if it passes R->>E: Deploy E-->>P: Retrain on a threshold
flowchart LR S0["Job fails"] --> S1["Open logs"] --> S2["Find the step"] --> S3["Fix it"]
Key point: Logs show why training fails.
Why the other options are wrong
- A. Add more layers to the model
- More layers are a guess.
- B. Change the region of the bucket
- Region is a guess.
- C. Ask the team to rerun it until it passes
- Reruns waste compute.