At 100 million tokens per day, cloud APIs burn $180,000/month while self-hosting costs roughly $22,000 —an 88% gap that makes the decision look obvious. At 10 million, the fully loaded self-hosting TCO —reserved GPUs idling through overnight troughs, 0.3-0.5 FTE of MLOps overhead, the 2-6 week delay behind every model launch —routinely exceeds cloud API. Most teams land between these extremes, and getting the calculation wrong costs two to three times what they should be paying. Below is a complete TCO framework. Plug in your own numbers to find your break-even point —not someone else’s rule of thumb.
Why “Cloud Is Expensive” and “Self-Hosting Is Free” Are Both Wrong
The Cloud Cost Trap
At 100 million tokens per day with GPT-4o pricing, your monthly API bill hits roughly $18,750. That number looks alarming —but it assumes zero optimization. In practice, most teams running at that scale are overpaying by 30-50% because they haven’t implemented tiered model routing, prompt caching, or batch API usage. Optimizing your cloud spend —covered exhaustively in our 12-way cost optimization guide —typically costs less than a month of GPU reservations and delivers savings within the same billing cycle. Exhaust cloud optimization before you consider self-hosting.
The Self-Hosting Hidden Cost Trap
Two reserved H100 instances run about $4,200 per month. Compared to that $18,750 cloud bill, it looks like a 78% savings. But hardware is about 30-40% of the real cost. Add 0.5-1.0 FTE MLOps engineer ($6,000-12,000/month), GPU utilization risk (70% idle time during traffic troughs = effectively paying for compute you don’t use), model update maintenance (2-6 weeks per upgrade cycle), and the opportunity cost of not having immediate access to new frontier models. At below 50 million tokens per day, the fully loaded self-hosting TCO often exceeds cloud API costs —sometimes substantially.
The Fundamental Insight
Cloud APIs charge a per-token markup. Self-hosting carries fixed infrastructure and engineering overhead. Which is cheaper depends entirely on where your usage falls on the cost curve. Cloud wins at low-to-moderate volume and spiky workloads. Self-hosting wins at high, steady volume. Hybrid wins for almost everyone in between.
The Financial Model: What Actually Goes Into TCO
Compute Costs
| GPU | Hourly (Reserved) | Tokens/sec (8B Model) | Best For |
|---|---|---|---|
| H100 | ~$2.10 | 2,500+ | 70B+ models, high throughput (pricing cross-referenced with Artificial Analysis) |
| A100 | ~$1.20 | 1,500+ | 8-70B models, moderate throughput |
| RTX 4090 | ~$0.10 (self-owned equivalent) | 100+ | 8B models, low-volume serving |
Quantization —4-bit GPTQ or AWQ —cuts VRAM requirements 60-75% with a 1-3 point quality trade-off. For a Qwen3-8B, that’s the difference between needing an A100 and running comfortably on an RTX 4090. Test the quality impact on your specific workload before committing to a quantization level.
Engineering Labor
This is where most TCO models collapse. Self-hosting isn’t “set up once and it runs.” It’s a continuous production pipeline: GPU procurement and driver compatibility management, framework selection and version upgrades (vLLM, SGLang, or TGI), auto-scaling and queue management, monitoring and on-call rotation, model upgrade cycles with A/B testing, security patches and supply chain risk mitigation. The March 2026 LiteLLM malware incident wasn’t a one-off —it’s the new normal for self-managed infrastructure. Budget 0.5-1.0 FTE for any self-hosted deployment handling production traffic.
Utilization Risk
Your traffic drops to 10% of peak between 1 AM and 5 AM. Those H100s are still running, still drawing power, still costing you $2.10/hour each. Reserved instances guarantee availability but demand accurate capacity forecasting. Spot instances save 60-80% but can be reclaimed with 30 seconds’ notice —unacceptable for production workloads. Cloud APIs charge zero for idle time. Self-hosting charges full price 24/7.
Model Freshness Opportunity Cost
A new flagship model drops. Cloud API users change a model string and deploy in minutes. Self-hosted users begin a two-to-six-week evaluation and migration cycle: download weights, benchmark against current model, re-optimize prompts, A/B test, production rollout. For products where competitive differentiation depends on model quality, this delay has real business cost —potentially exceeding the infrastructure savings.
Plug your own numbers into the model above. To ground the analysis in real pricing, here’s how the TCO breaks down at three representative scales —covering the range where most teams operate today.
Break-Even Analysis at Three Scales
Small: 10M Tokens/Day (~300M/Month)
Cloud API: Blended rate $2/M tokens across tiered routing —$600/day, ~$18K/month.
Self-hosted: 1×H100 reserved ($1,512/month) + 0.3 FTE MLOps ($3,600/month) + infra overhead ($500/month) —~$5,612/month.
On paper, self-hosting saves 69%. But this assumes >85% GPU utilization and a 0.3 FTE who actually exists and isn’t also your lead backend engineer. At 50% utilization (spiky traffic), self-hosted cost rises to ~$7,500/month —savings shrink to 58%. At 30% utilization (common for early-stage products), savings drop below 40% —and a single GPU driver incident erases months of savings in engineering time.
Verdict: Cloud, with cost optimization. Implement tiered routing and prompt caching first. Reassess at 50M tokens/day.
Medium: 50M Tokens/Day (~1.5B/Month)
Cloud API: ~$90K/month blended rate.
Self-hosted: 2×H100 ($3,024) + 0.5 FTE MLOps ($6,000) + dedicated infra ($1,000) —~$10,024/month.
Self-hosting is now clearly cheaper —60-70% savings —even accounting for 40% utilization risk and full engineering overhead. This is the crossover zone where self-hosting earns its operational complexity.
Verdict: Self-hosting, or hybrid. Start planning GPU procurement at 30M tokens/day so you’re operational by the time cloud costs hit $75K/month.
Large: 100M+ Tokens/Day (~3B/Month)
Cloud API: $180K+/month.
Self-hosted: 4-8×H100 cluster ($6-12K) + 1.0 FTE MLOps ($10K) + dedicated infra ($2K) —$18-24K/month.
Self-hosting advantage widens to 80%+. At this scale, you likely already have a dedicated infrastructure team —TCO calculations should include depreciation, data center space, network bandwidth, and redundancy.
Verdict: Self-hosting is the clear economic winner. But at this scale, you’re not deciding between self-hosting and cloud —you’re running a hybrid architecture where self-hosted handles baseline and cloud provides overflow and frontier model access.
The Hybrid Row
70% of routine traffic —self-hosted or cheap API (DeepSeek V3.2 at $0.27/M input). 30% of complex traffic —cloud frontier models (Claude Sonnet 4, GPT-5.5). Total hybrid at 50M tokens/day: ~$32K/month vs. pure cloud $90K/month —64% savings while retaining access to the best models available.
Picking the right model for each tier requires current pricing data. Our 2026 LLM API pricing comparison ranks every major model by cost and capability per task type —so your routing decisions reflect this month’s prices, not last quarter’s.
The Hybrid Architecture: What “Best of Both” Actually Looks Like
This section gives you the architectural overview and cost rationale for each hybrid pattern. For complete Python implementations with routing code, infrastructure glue, and the PII classifier architecture, see S10: Hybrid LLM Architecture —Implementation Guide.
Pattern 1: Tiered Model Routing
All traffic flows through a unified API gateway. A lightweight classifier determines complexity. Simple tasks (classification, extraction, simple Q&A) route to cheap models —self-hosted Qwen3-8B or DeepSeek V3.2 via API. Medium tasks route to mid-tier cloud models. Complex tasks —multi-step reasoning, agentic coding, sensitive customer interactions —route to frontier cloud models.
The key infrastructure: one base URL, one API key, routing logic at the gateway layer. Application code doesn’t know or care where a request ultimately executes. See our multi-model architecture guide for the implementation.
Pattern 2: Self-Hosted Baseline + Cloud Overflow
Self-hosted inference handles baseline load —the predictable, steady-state traffic you’d otherwise be paying cloud markup on. When demand spikes beyond capacity, overflow routes automatically to cloud API endpoints. The self-hosted stack maintains high utilization. Cloud provides elasticity without over-provisioning. The critical infrastructure piece: queue depth monitoring at the gateway with an auto-scale trigger that spins up cloud routing when self-hosted latency exceeds threshold.
Pattern 3: Local Sensitive + Cloud General
PII, PHI, or other regulated data —self-hosted inference. Data never leaves your network. Compliance checklist stays green. General queries, public data, non-sensitive workloads —cloud API for access to the latest frontier models. Same application. Same codebase. Same API format. Only the routing rule differs —PII classifier tags the request, gateway routes accordingly.
Counterarguments & Honest Limitations
”Open-weight models are closing the gap —soon there’ll be no reason to pay for cloud APIs.”
On broad benchmarks, yes: DeepSeek V3.2 trails GPT-4o by under 5 points on MMLU. On specific high-stakes tasks —complex multi-step agentic coding (SWE-bench verified), frontier-level reasoning (GPQA Diamond), multimodal video understanding —proprietary models still lead by 8-15 points. If your product’s competitive differentiation depends on these frontier capabilities, pure self-hosting isn’t viable yet. Also: multimodal (video, audio) capabilities remain significantly weaker in open-weight models. For a side-by-side view of capability gaps by task category —not just aggregate benchmark scores —see the TokSpan models directory with current pricing and capability tags for every supported model.
”vLLM and Ollama make self-hosting trivial.”
They make running a model trivial —from days of configuration to minutes. Running a model is not running a production service. Auto-scaling across multiple GPUs, graceful degradation during model updates, multi-model serving with fair queuing, observability integration, A/B testing infrastructure —vLLM and Ollama don’t solve these. You still need 0.3-0.5 FTE for production readiness, even with the best tooling.
”My team needs data sovereignty —self-hosting is our only option.”
If your compliance requirements mandate that data never leaves your network, self-hosted inference for sensitive workloads is non-negotiable. But that doesn’t mean 100% self-hosting. Hybrid Pattern 3 —sensitive data local, everything else cloud —preserves frontier model access while satisfying the most stringent compliance requirements.
FAQ
At what volume does self-hosting become cheaper?
The crossover sits around 50M tokens per day (1.5B/month) for most scenarios. Between 10-50M, the answer depends on utilization rates, engineering capacity, and workload characteristics. Below 10M, cloud API is almost always more economical. The most practical advice: run on cloud, instrument everything, collect three months of real usage data, then model your specific break-even. Don’t self-host based on projected volume —self-host based on measured volume.
Can I just use cheaper models instead of self-hosting?
DeepSeek V3.2 at $0.27/M input tokens approaches the per-token marginal cost of self-hosting. For many scenarios, tiered routing with cheap cloud models achieves most of the cost reduction of self-hosting with none of the operational overhead. Exhaust routing optimization —using the right model for each complexity tier —before considering self-hosting. It delivers 50-70% cost reduction with zero infrastructure changes. Our regularly updated ranking of the cheapest LLM API providers tracks which models offer sub-$0.15 per million input token pricing each month —a quick reference when assigning models to your lower tiers.
How often do self-hosted models need upgrading?
Major open-weight model families (Llama, Qwen, DeepSeek, Gemma) release significant new versions every 4-6 months. Each upgrade requires 2-6 weeks for evaluation, prompt re-optimization, A/B testing, and production rollout. This isn’t an occasional disruption —it’s a recurring engineering commitment that should be factored into team capacity planning.
Is hybrid architecture simpler, or am I just adding complexity?
Hybrid introduces routing logic and multi-environment management —real additional complexity. But if you use a unified API platform as the gateway, most routing complexity is absorbed at the infrastructure layer. Your application code sees one endpoint. Gateway-emitted spans (see our observability guide) show all endpoints —self-hosted and cloud —in one dashboard. The operational gain of hybrid is that you’re not forced into an all-or-nothing decision. Start cloud-only. Add self-hosted for specific tiers when the data supports it.
What about using TokSpan or OpenRouter versus going direct?
API aggregation platforms provide wholesale pricing 15-35% below direct retail API rates —the first and easiest cost lever for cloud API users. They also provide tiered routing, centralized observability, and unified billing across providers through a single endpoint. If you’re considering self-hosting to reduce costs, try aggregation-platform pricing first. The savings may be enough to keep you on cloud —with zero infrastructure changes. The tiered routing logic that drives these savings is covered in the custom routing documentation —rule-based routing, semantic routing, and automatic model fallback from a single API endpoint.
“Self-host vs. cloud” is a math problem, not a philosophy. The math says: cloud wins below 10M tokens/day, hybrid wins from 10-100M, and self-hosting’s advantage becomes decisive above 100M —but even then, hybrid with cloud overflow for frontier model access is the practical winner for most teams.
Start with cloud. Instrument everything. Collect real data. Model your break-even. The GPU reservation can wait. The cost optimization —tiered routing, prompt caching, batch API usage —cannot.
Want to see how TokSpan’s pricing compares to direct provider rates and self-hosting costs at your scale? Explore TokSpan pricing —unified API access across 200+ models with wholesale rates 15-35% below direct retail.