Compliance says customer data stays inside the VPC. Product says the new features need frontier reasoning. Both demands are absolute — and every architecture discussion deadlocks before it begins.
“Go local” breaks the product. “Go cloud” breaks compliance. Pick one and you lose.
Hybrid architecture resolves the contradiction: sensitive data runs self-hosted, complex reasoning reaches frontier cloud APIs, routine queries hit the cheapest model either way.
Here are three production patterns with complete routing logic — and the anti-patterns that sink teams who skip the unified gateway.
Pattern 1: Tiered Model Routing
All traffic flows through a unified API gateway. A lightweight classifier determines complexity. The routing logic:
if task_complexity == "simple":
route → self-hosted Qwen3-8B or cheap cloud API (DeepSeek V3.2 at $0.27/M)
elif task_complexity == "medium":
route → mid-tier cloud API (GPT-4o at $2.50/M, Claude Sonnet 4 at $3/M)
else:
route → frontier cloud API (Claude Opus 4 at $15/M, GPT-5.5)The classifier itself is a cheap model call — one extra API request that costs $0.0001 and adds 50ms of latency. It pays for itself within the first 100 requests by preventing a single simple query from reaching a frontier model.
Cost impact: 70% of traffic routes to cheap tier at $0.15-0.27/M tokens. 25% routes to mid-tier at $2.50-3/M. 5% routes to frontier at $10-15/M. Blended cost is 50-70% lower than routing everything through a single premium model. These per-token figures shift frequently — check current pricing across all major providers before locking in your tier thresholds.
TokSpan role: all tiers accessible through one base URL and API key. The routing logic lives in your application. The unified endpoint means you manage one integration, not three. For reducing the input token cost of your tiered routing classifier and system prompts, see our prompt caching guide. Anthropic’s prompt caching documentation provides the authoritative reference on cache lifetime, pricing tiers, and minimum token thresholds — useful when selecting which prompts to cache at each tier of your hybrid architecture.
Pattern 2: Self-Hosted Baseline + Cloud Overflow
Self-hosted inference handles steady-state traffic — the predictable baseline load that would otherwise generate consistent cloud API costs. When demand spikes beyond self-hosted capacity, overflow routes automatically to cloud APIs.
class HybridRouter:
def __init__(self, local_client, cloud_client, max_local_queue=50):
self.local = local_client
self.cloud = cloud_client
self.max_queue = max_local_queue
async def route(self, request):
queue_depth = await self.local.get_queue_depth()
if queue_depth < self.max_queue:
return await self.local.generate(request)
else:
return await self.cloud.generate(request)The self-hosted stack maintains high utilization — avoiding the idle GPU waste that kills self-hosting economics. Cloud provides elasticity — absorbing spikes without requiring you to provision for peak capacity 24/7.
Key infrastructure: queue depth monitoring at the gateway. Auto-scale trigger: when self-hosted latency exceeds threshold, route overflow to cloud. When queue depth drops below target, route back to self-hosted. This feedback loop keeps utilization high and cloud costs contained.
Pattern 3: Local Sensitive + Cloud General
PII, PHI, and regulated data → self-hosted inference. Data never leaves your network — keeping API traffic within your security perimeter. Compliance stays clean. General queries, public data, non-sensitive workloads → cloud API for frontier model access.
class PIIRouter:
def __init__(self, pii_classifier, local_client, cloud_client):
self.classifier = pii_classifier
self.local = local_client
self.cloud = cloud_client
async def route(self, user_input: str):
contains_pii = await self.classifier.detect(user_input)
if contains_pii:
return await self.local.generate(user_input)
else:
return await self.cloud.generate(user_input)The PII classifier runs before any LLM call — regex patterns for structured PII (credit cards, SSNs), NER model for unstructured PII (names, addresses in free text). Classification takes <10ms. The routing decision is zero-latency from the user’s perspective.
Same application. Same codebase. Same API format. Only the routing rule differs — classify, route, generate. The application layer doesn’t need to know whether a request ultimately executed locally or in the cloud.
The Infrastructure Glue: Unified API Gateway
All three patterns depend on a gateway that presents one interface to application code while routing to diverse backends. The gateway handles:
- Unified authentication. One API key. The gateway translates to provider-specific credentials.
- Request normalization. OpenAI-compatible format in. Provider-specific format out to whatever backend is targeted.
- Response normalization. Whatever the backend returns, the application receives a consistent structure.
- Observability. Every request — local or cloud — emits the same span format to the same trace backend. One dashboard. All endpoints.
A unified API platform absorbs this complexity at the infrastructure layer. Your application code sees one endpoint. The platform handles routing, normalization, and observability. You configure routing rules. You don’t build routing infrastructure.
For hardening this setup for production traffic, review the production optimization guidelines.
Hybrid Architecture Anti-Patterns
These architecture decisions look reasonable on a whiteboard. They fail in production. Learn from the teams that already made these mistakes.
The Router That Thinks Too Hard. A developer tools company built a complexity classifier that called GPT-4o to decide whether a query needed GPT-4o. The classifier prompt was 400 tokens of evaluation criteria. The classification step added 800ms of latency and cost $0.002 per request — to decide whether to spend $0.003 or $0.015 on the actual response. Net effect: they added cost and latency to every request to save cost on 30% of them. The math only works if your classification step costs less than 10% of the savings it enables. Use a regex + embedding-similarity classifier that runs in <10ms, not an LLM chain that costs more than the cheap-tier response it’s trying to route to.
The Dual Stack That Became Two Stacks. A healthcare startup built Pattern 3 (PII to local, general to cloud). They deployed two separate API gateways — one for the self-hosted vLLM instance, one for the cloud provider. The application team had to integrate both, handle two authentication schemes, parse two response formats, and maintain two sets of error-handling logic. The “hybrid” architecture was really two parallel stacks with nothing shared. Observability was split across two dashboards. When a request failed, debugging meant correlating logs from two systems with different timestamp formats. The fix — a unified API gateway that normalizes both backends behind one endpoint — took three weeks to retrofit and should have been built first. One gateway. One interface. Multiple backends. If you’re building the gateway after the backends, you’re building in the wrong order.
The “Self-Hosted Is Cheap” Delusion. A mid-stage startup budgeted $60/month for self-hosting Qwen3-8B on a single GPU instance. Their spreadsheet assumed 90% GPU utilization. Real utilization: 35%. Traffic is bursty — 3 hours of peak load, 21 hours of near-idle. The GPU sat idle 65% of the time but still cost $60/month. On a per-request basis, their self-hosted model cost 2.1× more than routing the same queries to DeepSeek V3.2 at $0.27/M tokens. The self-hosting math only works with sustained high utilization — 70%+ GPU usage across a 24-hour window. Before provisioning hardware, instrument your production traffic for two weeks. Measure request arrival patterns. Calculate expected GPU utilization with your actual traffic shape, not an idealized constant load. If utilization comes in under 50%, cloud API is cheaper — even at list prices.
Further reading. The architecture patterns here complement our integration patterns guide for production deployments.
FAQ
Is hybrid architecture’s extra complexity worth it?
If your alternative is “all cloud” with a monthly bill crossing $50K — yes, the complexity pays for itself. If your alternative is “all self-hosted” with a compliance team telling you PII in cloud APIs is a violation — yes, hybrid is the only compliant path to frontier model access. If your monthly bill is $2K and you have no compliance requirements — hybrid is overengineering. The tipping point is around 30M tokens/day or one hard compliance requirement. For lowering your cloud bill before committing to a hybrid migration, our cost optimization strategies cover caching, batching, and model tiering techniques.
How do I decide which traffic goes self-hosted vs. cloud?
Three criteria, in order: (1) Does the data contain PII or regulated information? → Self-hosted. (2) Is the task simple enough for an 8B model? → Self-hosted or cheap cloud API. (3) Does the task require frontier model reasoning? → Cloud API. Instrument your traffic for a week. Classify every request by these criteria. The distribution tells you what your hybrid split should be.
How do I handle fallback between self-hosted and cloud?
Self-hosted model fails → fallback to cloud API. Cloud API rate limited → fallback to self-hosted. Both fail → degraded mode (cached responses or graceful error). The fallback chain is the same architecture described in our multi-model routing guide — self-hosted endpoints are just another handler in the chain.
How do I prevent routing logic from becoming the new single point of failure?
The routing layer becomes a single point of failure the moment it’s the only component that knows which backend serves which traffic. Mitigations: (1) health checks on every backend — if self-hosted is down, route everything to cloud automatically; (2) the router itself should be stateless and horizontally scalable — two instances behind a load balancer; (3) keep routing rules simple — PII check and complexity classification, nothing that requires its own LLM call chain to decide. The router’s job is to decide fast, not to decide perfectly. If the routing decision takes longer than the LLM call it’s routing, you’ve built the wrong abstraction.
What’s the latency penalty for the routing layer?
A well-designed routing layer adds 5-15ms — the time to run a regex-based PII check or an embedding-based complexity classifier. This is noise compared to LLM inference latency (500ms-5s). The teams that see 100ms+ routing latency made one of two mistakes: (1) they used an LLM call inside the routing decision, or (2) they added a synchronous network hop to an external classification service. Keep the routing logic in-process, deterministic where possible, and network-call-free. If your router talks to an external service, you’ve already lost the latency budget.
Can I use the same observability platform for both self-hosted and cloud?
Not just can — you must. Split observability across two platforms (e.g., Grafana for self-hosted, vendor dashboard for cloud) is the fastest way to lose visibility into hybrid behavior. Every request — regardless of which backend executed it — must emit the same span format with the same attributes: model name, token count, latency, cost, and a backend tag (self-hosted or cloud). This is what makes the unified API gateway pattern critical: it normalizes observability at the egress point, before spans diverge into backend-specific formats. One dashboard. One query language. All backends. If you find yourself opening two tabs to debug one request, your observability architecture is wrong.
Hybrid architecture isn’t a compromise between self-hosting and cloud. It’s the architecture that gives you the strengths of both without the weaknesses of either. Sensitive data stays local. Routine traffic stays cheap. Complex reasoning reaches the frontier. One system. One API. All the capability. None of the compromise.
Self-hosted models and cloud APIs, one control plane. Deploy your hybrid stack on TokSpan — intelligent routing between local and cloud, unified observability across both.