You loaded the entire 800-page contract into the model. It answered — and the bill for that one question was more than your entire API spend last month. The model had “handled” the context window. Your budget hadn’t.
Long-context is the most hyped capability in the LLM stack, and the hype has produced a debate — “is RAG dead?” — that is almost entirely wrong-shaped. The 1M-token window is real: Gemini ships a 2M-class window, Claude ships 1M, Moonshot’s Kimi K3 launched with 1M in July 2026, DeepSeek’s V4 Pro carries 1M context with 384K output. The physics, though, didn’t change: tokens cost money, attention degrades with distance, and “can I stuff it” was never the same question as “should I.”
This guide covers the part the debate skips: the cost math (full context versus caching versus RAG), the workflow patterns that make 1M-token windows usable (map-reduce, compaction, sliding windows, layered context), and the decision framework for long-context-versus-RAG that ends the argument with numbers instead of vibes.
What Long-Context APIs Deliver in 2026
Takeaway: the 1M window is now standard across the frontier — and the differentiators are cost, cache behavior, and attention quality, not the window size.
The 2026 lineup: Gemini’s 2M-class windows, Claude’s 1M GA, Kimi K3’s 1M (the open flagship that reset expectations on both context and price), DeepSeek V4 Pro at 1M context, and the GPT-class families at 400K-and-up. The model catalog tracks current limits per provider through a unified endpoint.
Three things the window size doesn’t tell you:
- Cost per token at full window. A 1M-token input at frontier rates is real money per question — the exact number is provider- and tier-specific, which is the point of section three.
- Cache pricing. Most providers discount cached input tokens to roughly a tenth of the base rate — the cache convention documented across the industry — which changes the economics of repeated long-context workloads completely.
- Attention quality at depth. “The window is 1M” means the model accepts 1M; it does not mean it uses 1M equally. Retrieval-style degradation at depth is real, and it’s the engineering reason the workflow patterns below exist.
Why “1M Tokens” Is Not “1M Useful Tokens”
Takeaway: window size is a ceiling, not a quality promise — and the cost curve is linear while the attention curve isn’t.
Three realities behind the marketing:
- Cost scales linearly, painfully. Every token in the window is billed, whether it contributed or not. A 1M-token question at a dollar-scale rate is a dollar-scale question — repeatable, until it isn’t.
- Attention degrades with distance. Evidence across model families consistently shows models using the beginning and end of long contexts better than the middle — “lost in the middle” is the well-documented pattern. Stuffing is not understanding.
- Cache changes the shape. With cache pricing at ~0.1×, the second full-window question is cheap — which rewards stable, repeated long contexts (the same document queried many times) and punishes one-off full-context loads.
The design consequence: long-context works best as a reusable asset — a stable corpus you query repeatedly — and worst as a per-question gamble.
The Cost Math: Full Context vs Cache vs RAG
Takeaway: for repeated queries over the same corpus, cache-then-RAG usually wins; for one-off whole-document questions, full context is honest; for everything else, run the numbers.
The comparison, on a concrete shape — a 500K-token corpus, ten questions:
| Approach | Cost driver | Typical relative cost |
|---|---|---|
| Full context, every question | 500K tokens × 10 | 10× full-input price |
| Cache + full context | ~0.1× on cached input | ~1× full-input price |
| RAG (top-k retrieval) | 5K tokens × 10 + index | fraction of one full input |
The structural conclusions: RAG wins on cost by orders of magnitude for repeated queries; cache makes full-context competitive when the same corpus is queried often; full context wins outright only for one-off deep-read questions. Quality follows a different curve — retrieval can miss, full context can bury — which is exactly why the decision framework exists.
The numbers are provider- and corpus-shaped; the caching guide in this series has the cache mechanics, and the worked example below is the shape to fill with your own rates.
How to Build 1M-Token Workflows
Takeaway: four patterns — map-reduce, compaction, sliding windows, layered context — and the design rule is “never pay for tokens you don’t need.”
- Map-reduce. Split the corpus, process chunks, merge results. The workhorse for “answer over the whole document”: each chunk is small, parallelism is natural, and the final synthesis reads only the summaries. Cost scales with chunks, not with the full window.
- Context compaction. Compress the middle — summarize, extract key facts, drop redundancy — before the next turn. Compaction is how agents hold multi-turn context without quadratic costs; it’s the difference between “the agent remembers” and “the agent carries the whole transcript.”
- Sliding windows. Keep the recent N tokens, summarize what falls out. For conversations and streams, the window follows the action — the pattern that keeps interactive costs bounded.
- Layered context. A fast path and a full path: summaries for routine turns, on-demand retrieval of the full passage when the question needs it — custom routing makes the fast/full decision mechanical. Layering is the production pattern that makes “1M context” a capability instead of a bill.
The common thread: every pattern is a way to pay for the tokens that matter. The chat completions endpoint handles the mechanics; the patterns decide the economics.
How to Choose: Long Context vs RAG vs Hybrid
Takeaway: the question isn’t “is RAG dead” — it’s “what is my query pattern” — and the decision framework is three questions, not a religion.
- Is the corpus static and repeatedly queried? Cache-plus-RAG — index once, retrieve per query, cache the stable prefixes.
- Is the question a one-off deep read over a large document? Full context — the honest use case where retrieval would miss and the cost is bounded by the question’s rarity.
- Is it a long-running agent conversation? Layered context — summaries plus retrieval plus a bounded window, never full history.
The relationship with the RAG guide in this series is complementary, not competitive: that guide owns retrieval internals — chunking, hybrid search, reranking — and this guide owns the context-window economics and the workflow patterns. Both apply to most production systems, and the hybrid is the norm, not the compromise.
Common Mistakes
Takeaway: four traps — three cost-shaped, one quality-shaped — and each one is the “1M window” hype in action.
- Full context by default. The window exists, so everything gets stuffed — the linear-cost trap with no cache design and no retrieval. The budget tables in this guide exist because this mistake is the default.
- No compaction in agent loops. Every turn appends the full history; by turn twenty the agent is paying for everything it ever said. Compaction is not optional for long-running agents.
- Cache keys that drift. The cache multiplier only applies when the prefix is byte-stable — dynamic headers, timestamps, or reordered sections zero the hit rate, turning the ~0.1× into 1×. The caching guide has the mechanics; the discipline is yours.
- Benchmarking with the middle of the window. “It handled 1M tokens” tested on the start and end of a context — the lost-in-the-middle pattern survives, and the eval set that doesn’t sample the middle will tell you the wrong story.
FAQ
Can 1M-token context replace RAG?
For one-off deep reads over a large document, yes — that’s the honest use case. For repeated queries over a corpus, no: the cost math (10× full-input versus retrieval-sized) makes RAG the answer, with caching as the bridge. The three-question framework decides, not the marketing.
How much does a 1M-token query actually cost?
Full input at frontier rates, per query — the exact figure is provider- and tier-specific. With cache hits at ~0.1×, repeated queries over the same corpus collapse the cost. The table in this guide is the shape; your provider page is the number.
What is context compaction?
Compressing the conversation or corpus between turns — summarizing, extracting key facts, dropping redundancy — so the agent carries meaning instead of full history. It’s the pattern that keeps long-running agents affordable, and it’s non-negotiable past a few turns.
Is cache pricing really ~0.1×?
The industry convention is a cache-hit multiplier around a tenth of the base input price, documented consistently across providers. It only applies on byte-stable prefixes — which is why cache-key discipline is the hidden cost lever in every long-context deployment.
Which providers offer 1M-token windows?
Gemini (2M-class), Claude (1M), Kimi K3 (1M, open flagship), and DeepSeek V4 Pro (1M context) among the current generation — with the model catalog linked above as the current-availability reference through one endpoint.
When should I use full context instead of retrieval?
When the question is a one-off deep read where retrieval would miss — contract review, single-document analysis, whole-corpus reasoning — and the query is rare enough that the cost is an event, not a pattern. For everything repeated, cache or retrieve.
Summary
Long-context LLM APIs changed the cost math, not the physics: 1M windows are standard across the frontier, and the differentiators are now cache behavior, attention quality at depth, and the workflow patterns that keep the bill sane. Map-reduce for whole-corpus questions, compaction for agents, sliding windows for streams, layered context for production — and a three-question framework that replaces the “is RAG dead” debate with numbers. The window is a capability; the patterns make it a budget.
The decision framework is a spreadsheet problem. Get your TokSpan API key — $5 in free credits to test the math (quickstart) — and price all three approaches on your own corpus.