Learn · Prompt engineering & structured output · CCAR-F task 4.5
Batch processing, and when not to
The Message Batches API halves the cost of work nobody waits for. Match results by custom_id, plan for 24 hours, and keep blocking checks real-time.
Key ideas
- The Message Batches API costs 50% less for input and output on asynchronous work.
- Up to 100,000 requests or 256 MB per batch; results arrive in any order, so match on custom_id.
- Most batches finish within an hour, but plan for the 24-hour limit. Results are kept for 29 days.
- Each batch request is one Messages call: no multi-turn tool loop inside a batch.
- Use real-time calls for anything someone waits on; batch for overnight, back-fill and eval jobs.
- Batches stack with prompt caching; a 1-hour cache TTL suits a shared system prompt.
Worked example: Planning around an SLA
Requirement: every document processed within 30 hours. A batch can take up to 24, so collect and submit every 4 hours: worst case 4 + 24 = 28 hours. Submitting once a day would risk 24 + 24 = 48.
Exam traps
- Putting a blocking pre-merge check or live chat on the batch API to save money.
- Assuming results come back in submission order.
- Planning around the typical one-hour finish instead of the 24-hour limit.
- Resubmitting the whole batch when only a few requests failed.
Official docs
Practise CCAR-F questions · Next lesson: Independent and multi-pass review