A long conversation starts failing with a 400 invalid_request_error saying the prompt is too long. What is the right response?
Why D is the answer
Reduce the input, such as by trimming or summarizing history
A 400 will fail the same way every time, so backoff is wasted. The input has to shrink. max_tokens controls the reply, and raising it does not help an oversized prompt.
In detail
- A 400 fails the same way every time, because the request itself is invalid.
- The input has to shrink, by trimming or summarizing history.
- Count tokens before you send long conversations.
Use case
An illustrative scenario, not a real incident.
A tutoring chat grows until each request returns a 400. Backoff only repeats the failure. Summarizing older turns brings the input back under the limit.
See it in different ways
flowchart TD
A{"Which status?"} -->|"429, 500, 529"| B["Back off with jitter and retry"]
A -->|"400, 401, 403, 404, 413"| C["Fix the request or credentials"]
B --> D{"Still failing?"}
D -->|Yes| E["Contact support with the request ID"]stateDiagram-v2 [*] --> Failed Failed --> Backoff: 429 or 529 Backoff --> Failed: still failing Backoff --> Done: success Failed --> FixRequest: 400 Failed --> FixKey: 401 FixRequest --> [*] FixKey --> [*] Done --> [*]
flowchart LR S0["Conversation grows"] --> S1["Check input size"] --> S2["Summarize older turns"] --> S3["Send the smaller request"]
Key point: A 400 needs a changed request.
Why the other options are wrong
- A. Raise max_tokens so that the model has room for the whole conversation
- max_tokens controls the reply, so raising it does not help an oversized prompt.
- B. Switch to the Message Batches API
- Batches accept the same limits, so the request would still be too long.
- C. Retry with exponential backoff
- Backoff suits temporary failures, and this one is permanent until the input changes.