800 milliseconds. That’s the distance between “a phone call” and “a support ticket” — and it’s a budget to engineer, not a benchmark to admire. Here’s the budget, the build-vs-buy math, and the compliance floor, before you pick a single vendor.
Building AI voice agents is the fastest-growing corner of the agent space, and the worst-documented one. Almost every guide online is a vendor tutorial — Vapi, Retell, LiveKit, Pipecat each explain how to use their platform — and none of them answer the questions that decide production outcomes: what your end-to-end latency budget actually is, what each minute costs across stacks, and whether building beats buying.
This guide is the neutral layer: the component architecture (STT → LLM → TTS) with vendor choice left to you, the latency budget math that turns “feels slow” into a spreadsheet, a cost-per-minute model you can apply to any stack, and the compliance checklist that keeps the whole thing legal. The TTS and STT comparisons in this series cover vendor selection; this one covers the architecture that holds them together.
What a Voice Agent Actually Is
Takeaway: a voice agent is a chat agent plus a voice I/O layer — and the voice layer, not the LLM, is where production difficulty lives.
Strip the marketing away and the architecture is three components in a loop:
Audio in → STT (speech to text) → LLM (decide & respond) → TTS (text to speech) → Audio out
↑ |
└────────────── interruption / barge-in handling ←────────────┘The LLM in the middle is the same model your chat product uses — that part you already know. What makes voice agents hard is everything around it: recognizing speech in real time, deciding when to respond (turn-taking), stopping the response when the caller interrupts, and keeping all of it inside a latency budget that a text interface never had to meet. A chat user tolerates a two-second wait; a caller interprets it as a broken system.
The other misconception to kill early: a voice agent is not a chatbot with a microphone bolted on. It’s a realtime system with audio as its interface, and it needs the discipline of one — which is also why the support chatbot architecture you already know still applies for everything above the voice layer.
Why Production Voice AI Is Different
Takeaway: three constraints separate voice from every other agent workload — latency, interruption, and cost-per-minute.
- Latency is the product. Human conversation tolerates roughly 800ms end-to-end before it feels wrong, and under 500ms feels natural — the industry rule of thumb is documented well in Twig’s latency budget analysis. A text API that returns in 2 seconds is fine; a voice agent that responds in 2 seconds is broken. Every component contributes to that budget, so every component must be measured.
- Interruption is a feature. Barge-in — the caller talking over the agent — must stop the TTS mid-sentence, restart STT listening, and re-plan the LLM response. Platforms that handle it natively are worth real money; implementations that ignore it produce agents that talk over their customers, which is the fastest way to a refund request.
- Cost is per minute, not per token. A voice conversation burns STT seconds, LLM tokens, and TTS characters simultaneously, for every minute of every call. The per-minute cost model below is the one number your finance team will ask for first.
How to Choose: Build vs Buy
Takeaway: buy the platform when voice is a feature; build the stack when voice is the product — and the 2026 market made both options dramatically cheaper.
The platform tier — Vapi, Retell, LiveKit Agents, Pipecat, and the new entrants — has matured into a real option: managed telephony, turn-taking, barge-in, and evaluation tooling out of the box, priced per minute. The 2026 price war made it genuinely cheap to start: xAI entered with a voice agent builder at $0.05 per minute, undercutting the established platforms, and ElevenLabs’ agent product starts around $0.08 per minute. If your company needs a phone line that talks to customers, and voice is a support channel rather than the product itself, buy — the platform markup buys you the interrupt handling and telephony integration you’d otherwise build for months.
Build the component stack when voice is the differentiator: you need custom turn-taking, specific STT/TTS vendor combinations, on-premise data boundaries, or per-component cost control. The build path is three integrations instead of one platform — more work, more control, and a per-minute cost that can land well below platform pricing at volume. The honest middle path: build the components, buy the plumbing — assemble STT/LLM/TTS yourself behind a single endpoint, and let a thin orchestration layer (or none at all) handle the loop.
How to Build: The Component Stack
Takeaway: the stack is three replaceable components, not one vendor — choose each on its own merits and keep them swappable.
- STT — streaming recognition with interruption awareness. The current generation (AssemblyAI’s Universal-3-Pro, Deepgram’s Nova-3, and the alternatives) is covered in the STT comparison in this series: realtime tier for the agent, and watch the time-to-first-token, not just WER, because in a conversation, WER you can’t hear matters less than latency you can feel.
- LLM — the same model layer your chat product uses, with tool calling for real actions (booking, lookup, payment) and structured output for slot-filling. Nothing voice-specific here; the function-calling patterns you already use apply unchanged.
- TTS — streaming output with the latency profile that fits your budget. The TTS comparison in this series covers the quality-versus-latency trade; for agents, time-to-first-audio is the selection criterion, with naturalness second.
The architectural rule: each component stays behind your unified endpoint — a single key, a shared billing relationship, and one dashboard spanning audio and chat. The unified layer is plumbing consolidation, not vendor replacement. STT stays provider-native, TTS stays provider-native, and the LLM routes by task — cheap model for slot-filling, frontier model for the nuanced answers. That’s the custom routing pattern applied to a conversation.
A note on the realtime-API question: the all-in-one realtime endpoints are a legitimate option, but they are not a requirement. A component-style pipeline — streaming STT, chat LLM, streaming TTS — is a proven production architecture with better vendor flexibility, which matters the moment one vendor reprices. The audio API docs cover both directions of voice on the same endpoint.
How to Hit the Latency Budget
Takeaway: the 800ms budget is a stack problem, not a model problem — budget every component and measure in production, not in the demo.
The end-to-end budget decomposes into roughly: VAD/turn detection (~100-200ms) → STT time-to-text (~200-300ms) → LLM time-to-first-token (~200-400ms) → TTS time-to-first-audio (~100-200ms) → playback. Add network hops and the sum is exactly why “each component is fast” doesn’t mean “the agent is fast.”
Three levers, in order of impact:
- Pipeline in parallel, not serial. Start TTS synthesis on the LLM’s first chunk while the rest of the response streams; start STT listening during TTS playback (that’s what enables barge-in). Serial implementations pay every component’s latency in full; parallel implementations pay only the longest path.
- Stream everything. Streaming STT, streaming LLM responses, streaming TTS — non-streaming components are budget-killers in a voice loop.
- Tier the model by turn type. Slot-filling and confirmations run on the cheap/fast tier; complex reasoning runs on the frontier tier. The latency difference between tiers is often bigger than the cost difference, and both favor tiering.
Measure with real call recordings, in production regions, at P50 and P95 — the median hides the calls that actually hurt. The latency budget is the first thing to test, not the last.
How Much It Costs: Cost per Minute
Takeaway: a typical production agent lands in the $0.02-0.10 per minute range depending on stack and tiers — with platform pricing and component pricing converging as the 2026 price war plays out.
The per-minute model, applicable to any stack:
Cost per minute =
(STT seconds × per-second rate)
+ (LLM tokens × per-token rate, input + output)
+ (TTS characters × per-character rate)A worked example with a 60-second call: ~40 seconds of STT audio, ~300-600 LLM tokens, ~100-150 TTS characters. At typical 2026 rates, the components sum to roughly $0.02-0.06 per minute for a well-tiered stack — and platform products like Inworld’s worked cost model and the $0.05-0.08 per-minute entries from xAI and ElevenLabs confirm the range. The spread comes from three places: which LLM tier handles the conversation, whether STT/TTS run on realtime or async tiers, and whether you’re paying platform markup or component rates.
The scaling trap: a support agent handling 10,000 call-minutes a month at $0.05/min is a $500 line item — small. A sales agent running 100,000 minutes at an untiered stack (frontier LLM for every turn, premium TTS for every sentence) can cost 3-5× more per minute. Tiering is not optimization; it’s the difference between a feature and a line item your CFO audits.
Common Mistakes to Avoid
Takeaway: four architecture-driven failures sink voice agents in production — none of them is a code bug.
- No latency budget before launch. The demo runs on a quiet desk with a fast connection; production runs on phone lines at P95. If the 800ms rule isn’t enforced in your test suite, it isn’t enforced anywhere.
- Ignoring barge-in and turn-taking. Agents that talk over callers, or pause awkwardly waiting for silence, get hung up on. Barge-in handling is a feature to buy or build deliberately — it’s never an accident.
- Single-vendor binding. One platform for STT+LLM+TTS feels convenient until a repricing or a model removal — and the rate limits of any vendor can bite a high-volume agent. Keep components swappable behind one endpoint.
- No cost attribution. If you can’t answer “what did this agent cost per minute, per call type” at the end of the month, you’ll discover the answer on the invoice — which is the worst time to learn it.
Compliance: Consent & Disclosure
Takeaway: disclosure and consent are now regulated, not optional — the FCC’s 2026 AI voice-call rules and the TCPA/CCPA/GDPR framework apply to your agent from call one.
The regulatory landscape for AI voice calls hardened in 2026: the US FCC’s AI voice-call rules, layered on TCPA, plus CCPA and GDPR for recording and data — a combination covered in depth by the call-recording compliance guides and the GDPR compliance framework. The practical baseline for every deployment:
- Disclose at call start — the caller must know they’re talking to an AI, not a human, before the conversation goes anywhere.
- Consent for recording — where recording is permitted at all, capture explicit consent and store it with the call data.
- Minimize retention — keep audio and transcripts only as long as the business need justifies, and honor deletion requests through the same process as your other personal data.
- Document the pipeline — the data-processing terms of every component vendor (STT, LLM, TTS) are part of your compliance record, not an afterthought.
The pattern for the whole checklist: treat every voice call as personal-data processing by default. That assumption costs nothing to adopt and everything to skip.
FAQ
What’s the difference between a voice agent and a chatbot?
The LLM core is identical — the difference is the voice I/O layer: realtime STT/TTS, turn-taking, barge-in, and a latency budget in hundreds of milliseconds instead of seconds. The chatbot architecture guide linked above covers everything above the voice layer.
Should I build or buy a voice agent?
Buy the platform (Vapi, Retell, LiveKit, Pipecat, and the new $0.05-0.08/min entrants) when voice is a support channel and your differentiator is elsewhere. Build the component stack when voice is the product — custom turn-taking, vendor combinations, or data boundaries make the build path the right one.
What latency budget is acceptable?
End-to-end under 800ms is the acceptability line for human conversation; under 500ms feels natural. Budget every component — VAD, STT, LLM TTFB, TTS first audio — and enforce the budget at P95 in production regions, not at the demo desk.
How much does a voice agent cost per minute?
A well-tiered component stack typically lands at $0.02-0.06 per minute; platform products enter around $0.05-0.08. The spread is driven by LLM tiering, realtime-vs-async audio pricing, and platform markup — the per-minute model in this guide applies to any stack.
Do I need a realtime API to build a voice agent?
No. A component-style pipeline — streaming STT, chat LLM, streaming TTS — is a proven production architecture with better vendor flexibility, and it keeps each component swappable when prices or models move. Realtime endpoints are an option, not a requirement.
What do I need for call recording and disclosure compliance?
Disclosure at call start, explicit consent for recording where required, minimal retention, and documented vendor data-processing terms — the FCC’s 2026 AI voice-call rules plus TCPA/CCPA/GDPR are the framework, and the GDPR guide linked above is the baseline checklist.
Summary
When you build AI voice agents in 2026, the architecture is three swappable components — STT, LLM, TTS — held together by a latency budget that rules at 800ms, a per-minute cost model that rules in $0.02-0.10, and a compliance layer that rules from call one. Buy the platform when voice is a feature, build the components when it’s the product, keep every component behind a unified endpoint, and treat interruption handling and disclosure as features — because both are.
Build the smallest voice loop that talks back. Get your TokSpan API key — your first $5 in credits comes free — and watch per-component latency land on one dashboard from call one.