Every night you summarize 40,000 documents and nobody reads the output until morning. Which approach cuts the cost most without changing the model?
Why D is the answer
Submit the work through the Message Batches API
Batch usage is charged at 50 percent of standard prices, and a nightly job can tolerate the wait of up to 24 hours. Streaming and extra workers change nothing on the bill.
In detail
- Nobody needs the output until morning, so the job can wait.
- Batch usage costs 50 percent of standard prices.
- It can take up to 24 hours, which suits a nightly run.
- Build the digest once the batch has ended, not at a fixed time.
Use case
An illustrative scenario, not a real incident.
A publisher summarizes 40,000 articles overnight for a morning digest. Moving the job to the Batches API cuts the bill in half, with no change to the model or the prompt.
See it in different ways
flowchart LR A["Requests with custom_id"] --> B["Batch runs, up to 24 hours"] B --> C[".jsonl results in any order"] C --> D["Join on custom_id"]
sequenceDiagram participant J as Your job participant B as Batches API J->>B: Requests, each with a custom_id B-->>J: Results, in any order J->>J: Match each result on custom_id J->>B: Resubmit errored and expired ones
stateDiagram-v2 state "Nightly job" as s0 state "Batch processing" as s1 state "Batch ended" as s2 state "Morning digest" as s3 [*] --> s0 s0 --> s1: submit s1 --> s2 s2 --> s3: match by custom_id s3 --> [*]
Key point: Work nobody is waiting for belongs in batch.
Why the other options are wrong
- A. Lower max_tokens on each request
- This shortens summaries, and it does not change the rate you pay.
- B. Run the requests from more workers
- More workers finish sooner at the same price.
- C. Stream each response to a queue and process it later, when traffic is lower
- Streaming changes how text arrives, not what it costs.