You have a job that runs 5,000 requests a night, takes two hours, and nobody looks at the output until morning. And you’re paying realtime prices for it — because somewhere along the way, “the API” became “the synchronous API,” and the batch endpoint with the 50% discount never made it into the architecture.
That’s the most common cost leak in 2026 LLM budgets: delay-tolerant work paying realtime rates. Every major provider now discounts asynchronous workloads by roughly 50% — the LLM batch API market standardized on half price — OpenAI’s batch API with its ~24-hour completion window, Anthropic’s Message Batches with a ~1-hour window, Gemini’s batch tier, and DeepSeek folding the same idea into its peak/off-peak pricing. The money is sitting on the table, and this guide is about picking it up: what the batch APIs are, the provider-by-provider matrix, a workflow you can copy, the batch-vs-realtime decision rule, and the combination math that stacks batch savings with caching and routing.
What the Batch API Is
Takeaway: batch = submit now, collect later, pay half — every major provider standardized on the same shape.
The mechanism is consistent everywhere: you submit a file of requests (JSONL), the provider queues them, processes them when capacity allows, and you collect results when the batch completes. No streaming, no interactive latency — the discount is the compensation for flexibility.
The 2026 provider matrix:
| Provider | Discount | Completion window | Notes |
|---|---|---|---|
| OpenAI | ~50% | ~24 hours | the reference implementation |
| Anthropic | ~50% | ~1 hour | fastest window of the four |
| Gemini | ~50% | flexible tier | batch maps to the Flex inference tier |
| DeepSeek | off-peak pricing | peak/off-peak windows | the same idea via the clock, effective August 2026 |
The cross-provider comparison tracks the details; the pattern is identical — if you can wait, you pay half.
Why Batch Processing Saves Real Money
Takeaway: 50% off the largest line item in your bill is a pricing decision, not an optimization.
The math is deliberately boring. Say your monthly bill is $1,000 and 60% of it is delay-tolerant work (evals, indexing, enrichment, nightly generation). Moving that $600 to batch: $300 saved per month, $3,600 a year, zero quality change. The output tokens are identical; the only difference is when they arrive.
Two caveats keep the math honest:
- The discount applies at the provider level, not the platform level. A unified endpoint bills what the underlying provider bills — batch rates pass through as batch rates. Our cost-optimization playbook covers the full lever stack; batch is the lever with the least engineering cost.
- Batch is not free. It’s half price. The other half still benefits from caching and routing — which is the stacking math in section five below.
How to Build a Batch Workflow
Takeaway: the workflow is four pieces — submission, idempotency, collection, recovery — and the recovery piece is the one everyone skips.
The skeleton, provider-agnostic in shape — the request format follows the chat completions endpoint docs:
import json
from openai import OpenAI
client = OpenAI() # point at your provider or unified endpoint
# 1. Build the JSONL request file
with open("batch.jsonl", "w") as f:
for i, task in enumerate(tasks):
f.write(json.dumps({
"custom_id": f"task-{i}", # idempotency key — never omit
"method": "POST",
"url": "/v1/chat/completions",
"body": {"model": "gpt-4o-mini", "messages": task["messages"]},
}) + "\n")
# 2. Submit once — the custom_id makes retries safe
batch = client.batches.create(input_file=upload(batch_path), endpoint="/v1/chat/completions")
# 3. Poll or webhook until complete, then map results back by custom_id
# 4. Recover: failed rows get re-queued into the NEXT batch, not re-run inlineFour rules that keep this production-grade:
custom_idis your contract. Idempotency is what makes retries safe; without it, a network blip during submission duplicates work and doubles the bill.- Collect by ID, not by order. Batch results arrive in arbitrary order; map
custom_idback to your records or you’ll write the join wrong twice. - Webhooks beat polling at scale. A completion callback replaces the “check every 5 minutes” loop; the error codes reference tells you which failures are retryable.
- Recovery is a queue, not a hack. Failed rows go back into the next batch cycle. The teams that re-run failures inline are the teams that discover batch’s rate limits the hard way.
How to Choose: Batch vs Realtime
Takeaway: the decision is one question — can this work wait? — with two explicit exceptions where the answer is always no.
The decision rule, stated bluntly:
- Can the work wait an hour? Batch it.
- Can it wait a day? Batch it at the provider with the cheapest window fit.
- Is a user waiting? Never batch.
- Is it an agent loop? Never batch.
The second exception deserves emphasis: agent loops look like batch candidates on token volume alone, but their requests are causally chained — each call depends on the last — which makes them interactive by construction. The voice-agent and support-chatbot architectures in this series are interactive for the same reason. Batch is for parallelizable work with a deadline, and that’s a smaller set than most teams think — which makes the 50% all the more valuable where it does apply.
How to Stack Savings: Batch × Caching × Routing
Takeaway: batch, caching, and routing multiply — a batch job that reuses cached prefixes on a budget model costs a fraction of the naive version.
The three levers don’t overlap, which is exactly why they stack:
- Batch — 50% off the base rate for delay-tolerant work.
- Prompt caching — cached input prefixes at ~0.1×, and batch jobs are ideal caching workloads: the same templates run thousands of times, so byte-stable prefixes hit almost every time. The caching guide has the mechanics; the batch synergy is the multiplier.
- Routing — the model tier for the batch is a routing decision: evals on a frontier model because you’re measuring the frontier; enrichment on a budget model because nobody reads it. The production optimization docs cover the routing mechanics.
A worked stack: a nightly enrichment job at 10M tokens. Base rate on a mid-tier model: $30. Batch: $15. Same job with stable prefixes hitting cache at 0.1× on the input side (say 80% of tokens are cached): roughly $4-6. Same job, same output quality, one routing config and one stable prompt template later. The quickstart gets the pipeline wired up in minutes; the multiplier is the part that compounds.
Common Mistakes That Inflate Your Bill
Takeaway: four ways to undo the discount — each one silently converts 50% back into 100%.
- Evaluating on failed rows. Run your eval on the batch’s completed rows only; failed rows are a data-quality problem, and including them produces a fake score you’ll optimize against. The CI-style eval discipline from this series’ testing guide shows the pattern.
- Letting results expire. Batch results have retention windows; a job that finishes while you’re on vacation and expires before collection is a full-price job with no output. Wire collection to the completion event, not to your calendar.
- Ignoring batch-specific rate limits. Batch quotas are separate from realtime quotas — teams that assume the same limits discover their own ceiling mid-batch.
- Skipping idempotency. No
custom_id, no safe retry, no recovery path — the most expensive four characters you can omit.
FAQ
How much do batch APIs actually save?
About 50% on OpenAI, Anthropic, and Gemini batch tiers, with DeepSeek’s peak/off-peak scheme as the clock-based equivalent. The savings are per-token, so they scale with your delay-tolerant volume — the largest line item in most bills.
How long does a batch take to complete?
OpenAI’s window is ~24 hours, Anthropic’s ~1 hour, Gemini’s is tied to its Flex tier, and DeepSeek’s off-peak windows are defined by the clock. Pick the provider whose window fits your deadline; the discount is the same.
Can I run evals in batch mode?
Yes — evals are the canonical batch workload. Run them on completed rows only, keep the prompt template byte-stable for cache hits, and gate CI on the completed-batch score.
Does batch work with prompt caching?
Exceptionally well — batch jobs repeat the same templates thousands of times, which is the ideal caching profile. Stable prefixes + batch = the discount stacking in this guide.
Which workloads should never be batched?
Anything a user waits on, and anything causally chained — agent loops and interactive voice/chat are realtime by construction. The batch decision is “can it wait,” not “is it big.”
Do unified endpoints pass through batch discounts?
Yes — batch rates are provider rates, and the endpoint bills what the provider bills, with the discount intact. The endpoint’s job is one key and one dashboard; the 50% is the provider’s, and it flows through.
Summary
The 2026 LLM batch API landscape is remarkably standardized: roughly 50% off on every major provider, with completion windows as the only real differentiator. The play is mechanical — find the delay-tolerant share of your workload, move it to batch, keep the prefixes stable, route the model tier by task — and the combination of batch, caching, and routing routinely lands 60-80% below the naive bill. The discount is sitting on the table; the question is only whether your architecture picks it up.
Take the delay-tolerant 60% of your bill and halve it. Get your TokSpan API key — $5 free to spend on your first batch — and let per-job costs show you the savings.