A 429 Too Many Requests is the most useful response your LLM provider sends. More useful than the JSON body. More actionable than the error code. Because buried in its headers is a real-time capacity gauge —x-ratelimit-remaining, retry-after —that most production code ignores. Engineers treat rate limits as a wall to crash into. They’re a gauge to read.
Here’s what crashing into the wall looks like. Your app runs fine at 2 PM. Traffic picks up at 3 PM. By 3:15, every request returns 429. Your retry logic fires —fixed one-second intervals —and creates a thundering herd. Every retry hits the same rate-limit window. Ten minutes of cascading failures. Users see errors. Your on-call gets paged. The fix wasn’t “retry harder.” It was never retry harder.
The gap between those two outcomes —crashing into the wall versus reading the gauge —is three layers of code. This article covers all three. The reactive layer: exponential backoff with jitter. The proactive layer: header-aware throttling that reads your remaining budget and slows down before you hit the wall. The predictive layer: pre-call suspension that checkpoints agent state before the next call would exhaust your quota. Working Python code for each layer. Production-tested patterns.
For the foundational concepts of API authentication and key management that underpin rate-limit handling, see our API key security guide.
Why Rate Limits Exist —and How They Actually Work
Rate limits aren’t punishment. They’re infrastructure protection. Every API request consumes GPU memory and compute. An un-throttled client can saturate a provider’s inference cluster in seconds. Rate limits ensure fair allocation across all users.
The three limits you need to care about:
- RPM (Requests Per Minute): How many API calls you can make. Pay-as-you-go tiers: typically 500–3,000 RPM. Free tiers: 10–50 RPM. Enterprise: custom.
- TPM (Tokens Per Minute): Total tokens across all requests —input + output. A single 100K-token prompt consumes as much quota as 500 normal requests. TPM limits protect against this.
- Concurrent Requests: How many requests can be in-flight simultaneously. Exceed this and new requests queue or reject. This limit is often undocumented and learned through painful experience.
Provider-specific tiers put real numbers on these limits. OpenAI’s Tier 5 (the highest pay-as-you-go level) grants 10,000 RPM and 30,000,000 TPM for GPT-4.x models —but Tier 1 starts at just 500 RPM and 200,000 TPM. Anthropic’s Claude API offers 1,000 RPM at their standard tier with 80,000 TPM for Claude Opus and 400,000 TPM for Claude Sonnet, reflecting different inference costs per model.
Google’s Gemini API provides 1,500 RPM on pay-as-you-go with a 2,000,000 TPM ceiling. Each provider also enforces per-model overrides —Claude Opus 4’s TPM limit is tighter than Claude Sonnet 4’s because larger models consume proportionally more compute. Moving from Tier 1 to Tier 5 at OpenAI requires both increased spend history ($250+ monthly) and a demonstrated track record of non-abusive usage over 30+ days.
You don’t get high limits by asking nicely —you earn them through sustained production traffic patterns that prove you won’t flood the cluster. Knowing your exact tier and its limits is step one of building any of the three layers that follow.
How to read your current limits. Every API response includes rate-limit headers —and almost nobody reads them. OpenAI’s rate limit documentation explains the header format and tier structure:
x-ratelimit-remaining-requests: 487
x-ratelimit-remaining-tokens: 823000
x-ratelimit-reset-requests: 12sThese are on 200 responses, not just 429s. They tell you exactly how much budget remains before you’re throttled. Surface them as a gauge in your monitoring dashboard. Alert when remaining drops below 20%. The difference between “we hit a rate limit” and “we saw it coming and routed around it” is reading these headers.
Rate Limit Horror Stories: Two Incidents You Don’t Want to Repeat
A European e-commerce platform launched a Black Friday AI shopping assistant powered by GPT-4.5. Their QA tested at 50 concurrent users —production hit 2,300 in the first hour. Fixed-interval retries turned 429s into a 47-minute outage.
Revenue loss: $180,000 in tracked cart abandonment during the window. The root cause wasn’t traffic volume —it was retry logic that amplified the spike instead of absorbing it.
A SaaS analytics company migrated between OpenAI API versions without reading the rate-limit changelog. The new version halved their RPM from 3,000 to 1,500 at their tier. Their existing throttling code assumed the old limit.
Production ran fine for six days —until a monthly reporting cycle tripled their request volume. Every report job hit 429s simultaneously. Detection took 22 minutes because their monitoring only tracked 5xx errors, not 429s.
The fix was three lines: update the RPM constant. The lesson was permanent: every API version migration is a rate-limit migration.
Layer 1: Client-Side Throttling
The simplest layer. A token bucket or semaphore that prevents your application from ever exceeding the provider’s stated limits.
import asyncio
import time
class RateLimiter:
"""Token bucket rate limiter for LLM API calls."""
def __init__(self, max_rpm: int):
self.max_rpm = max_rpm
self.tokens = max_rpm
self.last_refill = time.monotonic()
self.semaphore = asyncio.Semaphore(max_rpm // 6) # Concurrency cap
async def acquire(self):
"""Wait until a request can be sent without exceeding RPM."""
# Refill tokens based on elapsed time
now = time.monotonic()
elapsed = now - self.last_refill
refill = elapsed * (self.max_rpm / 60)
self.tokens = min(self.max_rpm, self.tokens + refill)
self.last_refill = now
if self.tokens < 1:
wait_time = (1 - self.tokens) / (self.max_rpm / 60)
await asyncio.sleep(wait_time)
self.tokens = 1
self.tokens -= 1
limiter = RateLimiter(max_rpm=500)
async def rate_limited_api_call(model: str, messages: list):
await limiter.acquire()
# Make the API callImportant ordering: Always acquire the RPM token before the concurrency semaphore. Reversing the order causes head-of-line blocking —concurrency slots fill up with requests that can’t send, starving requests that could.
This layer prevents the most common self-inflicted rate-limit wound: exceeding your tier’s stated limit because you didn’t track how fast you were calling. For most low-to-moderate traffic applications, this is sufficient.
Layer 2: Header-Aware Backoff
Layer 1 prevents you from exceeding known limits. Layer 2 handles what happens when the provider’s actual capacity fluctuates —which it does, constantly, based on overall cluster load.
import random
from openai import OpenAI
client = OpenAI(
base_url="https://api.tokspan.com/v1",
api_key="ts-your-key-here"
)
def chat_with_backoff(messages, model="claude-opus-4-8", max_retries=4):
"""Exponential backoff with jitter + header awareness."""
for attempt in range(max_retries):
try:
response = client.chat.completions.create(
model=model,
messages=messages,
timeout=30
)
# Read remaining budget from headers —even on success
remaining = response.headers.get("x-ratelimit-remaining-requests")
if remaining and int(remaining) < 50:
print(f"Rate limit low: {remaining} requests remaining. Slow down.")
return response
except Exception as e:
if "429" in str(e) or "rate_limit" in str(e).lower():
if attempt == max_retries - 1:
raise # Out of retries
# Exponential backoff: 1s —2s —4s —8s
wait = (2 ** attempt) + random.uniform(0, 1)
print(f"Rate limited. Retrying in {wait:.1f}s (attempt {attempt + 1}/{max_retries})")
import time
time.sleep(wait)
else:
raise # Not a rate limit error —don't retryThe three rules of backoff:
- Never fixed-interval retry. It creates thundering herds. Every client retrying at t+1s hits the same rate-limit window.
- Always add jitter.
+ random.uniform(0, 1)spreads retries across the window. This alone prevents most cascading failures. - Never retry 401, 403, or 400. Retrying a bad API key or a malformed request won’t fix it. Only retry 429 and 5xx.
What NOT to do. This pattern —seen in production code surprisingly often —is the rate-limit equivalent of screaming louder at someone who doesn’t speak your language:
# DO NOT DO THIS
while True:
try:
response = client.chat.completions.create(...)
break
except:
time.sleep(1) # Fixed interval, no jitter, infinite retryThis creates a thundering herd and guarantees you’ll stay rate-limited. Every retry arrives at exactly the same point in the rate-limit window. The provider’s infrastructure sees a spike of identical requests, throttles them all, and your app enters a death spiral.
Layer 3: Predictive Suspension
Layers 1 and 2 are reactive —they respond after hitting (or approaching) a limit. Layer 3 is predictive —it reads the budget before the call and decides: continue, wait briefly, or checkpoint and suspend.
def predict_rate_limit(response_headers: dict) -> str:
"""Three-valued decision based on remaining budget."""
remaining_req = int(response_headers.get("x-ratelimit-remaining-requests", 1000))
remaining_tok = int(response_headers.get("x-ratelimit-remaining-tokens", 1000000))
if remaining_req > 100 and remaining_tok > 200000:
return "continue" # Plenty of budget
elif remaining_req > 20:
return "wait" # Budget running low —short pause
else:
return "checkpoint" # Budget nearly exhausted —suspend
# Usage in an agent loop:
for step in agent_steps:
response = call_llm(current_state)
decision = predict_rate_limit(response.headers)
if decision == "continue":
process(response)
elif decision == "wait":
time.sleep(5) # Short pause, let budget recover
process(response)
else: # checkpoint
save_agent_state(current_state) # Save progress
time.sleep(60) # Wait for rate-limit window reset
resume_agent_from_checkpoint() # Resume without losing workThe 2026 state of the art: agentpause. A Python library that reads rate-limit headers on every response, predicts exhaustion before the next call, and checkpoints agent state before suspending. Measured results: 0% crash rate vs. 100% reactive baseline. Zero 429 errors. 80% less token waste from failed retries. If you’re running high-throughput production agents, agentpause or equivalent predictive logic is no longer optional —it’s the difference between “our agents are reliable” and “our agents randomly crash when traffic spikes.”
Multi-Provider Routing: The Best Rate-Limit Solution
Every strategy above assumes you’re talking to one provider. The most effective rate-limit strategy is talking to several.
The core insight: Instead of waiting for one provider’s rate limit to reset, send the request to a different provider. Your effective rate limit becomes the sum of all providers you can route to.
A round-robin router with per-provider token buckets and automatic cooldown:
PROVIDERS = {
"openai": {"rpm": 2000, "cooldown_until": 0},
"anthropic": {"rpm": 1500, "cooldown_until": 0},
"google": {"rpm": 1000, "cooldown_until": 0},
}
def route_request(messages):
now = time.time()
available = [
p for p, cfg in PROVIDERS.items()
if now > cfg["cooldown_until"]
]
if not available:
raise RuntimeError("All providers in cooldown")
# Round-robin among available providers
provider = available[hash(str(messages)) % len(available)]
try:
return call_provider(provider, messages)
except RateLimitError:
PROVIDERS[provider]["cooldown_until"] = now + 30 # Cooldown for 30s
return route_request(messages) # Retry with a different providerAggregation platforms handle this at the infrastructure level —your single endpoint routes to all providers, with automatic cooldown, failover, and rate-limit monitoring. You set your desired throughput. The platform manages the per-provider rate limits. For the full routing implementation and latency-aware failover, see our production multi-model setup guide and custom routing docs.
FAQ
What’s the most common rate-limit mistake?
Fixed-interval retry. One-second delays across all clients create a thundering herd that guarantees more 429s. Always use exponential backoff with random jitter. The jitter alone —adding random.uniform(0, 1) to your wait time —prevents most cascading failures.
How do I know what my rate limits are?
Check your provider dashboard for your tier’s stated limits. Then read x-ratelimit-remaining-* headers on every API response —they tell you your actual remaining budget in real time. Monitor them. You don’t know your limits until you hit them —and that’s how most teams operate.
Does using multiple providers really solve rate limits?
Yes —effectively. A 500 RPM limit per provider becomes 2,000 RPM across four providers with a round-robin router. Aggregation platforms with multi-provider routing make this transparent: one endpoint, automatic provider-level rate-limit management. Pair multi-provider routing with cost optimization strategies to increase throughput without doubling your bill —route cheaper models for non-critical requests and reserve expensive ones for tasks that need them.
What’s the simplest fix I can implement today?
Replace your fixed-interval retry with exponential backoff + jitter. Five lines of code. Prevents the thundering herd that turns a single 429 into a cascading outage. The Layer 2 code in this article is copy-paste ready.
Can aggregation platforms handle rate limits for me?
Yes. Multi-provider routing, automatic cooldown for throttled providers, and unified rate-limit dashboards are standard features. You configure your desired throughput. The platform handles per-provider quota management, header monitoring, and automatic failover. One endpoint. No 429s.
The three layers here —throttle, back off, predict —turn rate limits from a reliability threat into a solved engineering problem. But as multi-provider routing becomes table stakes and providers compete on throughput guarantees, the question shifts: will “rate limit exceeded” join “disk full” and “out of memory” as errors that modern infrastructure simply makes obsolete? For now, the code in this article keeps you running. The longer arc points somewhere more interesting.
Layer 1 and Layer 2 you can implement in an afternoon with the code above. Layer 3 —predictive suspension —takes more investment. Aggregation platforms bundle all three into the request path: multi-provider routing absorbs provider-level throttling, header monitoring feeds a shared rate-limit dashboard, and automatic cooldown keeps healthy providers in rotation when one is degraded. No architecture eliminates 429s entirely, but spreading traffic across providers and reading the headers before you hit the wall turns them from a weekly incident into a rare edge case.