A very long conversation gets a 400 saying the prompt is too long. What is the right design?
Why D is the answer
Trim or summarize history before it reaches the limit
Oversized input returns a 400 on every model, and retrying sends the same input. Manage history before you reach the limit.
In detail
- Oversized input returns a 400, because the request itself is too long.
- Retrying sends the same input and fails the same way.
- Manage history before the limit by trimming or summarizing it.
- Track input size as the conversation grows, and act before the limit.
Use case
An illustrative scenario, not a real incident.
A support chat runs for hours and then fails with a 400 saying the prompt is too long. Retries fail the same way. Summarizing older turns before the limit keeps the conversation going.
See it in different ways
flowchart TD
A["Request"] --> B{"Input over the window?"}
B -->|Yes| C["400: prompt is too long"]
B -->|No| D["Generate"]
D --> E{"Window fills?"}
E -->|Yes| F["stop_reason: model_context_window_exceeded"]
E -->|No| G["Normal stop"]stateDiagram-v2 [*] --> CheckInput CheckInput --> Rejected400: input too long CheckInput --> Generating: input fits Generating --> MaxTokens: reply limit reached Generating --> WindowFull: model_context_window_exceeded Generating --> Finished: end_turn
flowchart TD
Q{"Is the conversation near the limit?"}
Q -->|"No"| A0["Continue"]
Q -->|"Yes"| A1["Trim or summarize history"]
Q -->|"Already over"| A2["A 400 until the input shrinks"]Key point: A prompt that is too long will not pass on a retry.
Why the other options are wrong
- A. Move the conversation to another API key
- The context limit belongs to the model, not to an API key.
- B. Retry the same request with backoff until it succeeds
- Backoff helps temporary failures. This one is permanent until the input changes.
- C. Raise max_tokens so that the larger conversation will fit
- A larger max_tokens asks for more output and does not make the input fit.