How to Build an AI Voice Agent: Technical Architecture

By Amar Kumar, founder of stackcone · September 2026

This brief is the technical blueprint for building a production AI voice agent — the kind that answers a real phone line, holds a natural conversation, calls your tools mid-call, and hands off to a person cleanly. It covers the full pipeline: telephony, streaming speech-to-text, the LLM agent loop, tool calling, streaming text-to-speech, the latency budget that makes or breaks the experience, and how to deploy and monitor it.

Who this is for: engineers and technical founders scoping a voice agent build — whether on a managed platform (Vapi, Retell, LiveKit) or a custom pipeline — who want the real architecture, not a marketing page.

How a voice agent actually works

A phone call is an audio stream in both directions. A voice agent turns the caller's speech into text, decides what to say or do, and turns the reply back into speech — fast enough that it feels like a conversation, not a transaction with a machine.

Four capabilities have to work together in real time:

• Hear: transcribe speech to text as the caller talks (streaming, not wait-for-silence)
• Think: an LLM agent decides what to say and whether to call a tool
• Act: call your systems — calendar, CRM, order lookup — mid-conversation
• Speak: turn the reply into natural audio and stream it back, interruptible

None of this is exotic technology on its own. The engineering problem is latency and turn-taking: every stage adds delay, callers notice a pause over roughly a second, and the agent has to know when the caller is done talking versus just pausing to think.

Reference architecture

flowchart TB classDef tel fill:#ede9fe,stroke:#7c3aed,color:#5b21b6 classDef speech fill:#dbeafe,stroke:#2563eb,color:#1e3a8a classDef agent fill:#ccfbf1,stroke:#0d9488,color:#115e59 classDef tool fill:#f1f5f9,stroke:#64748b,color:#334155 classDef mon fill:#fef3c7,stroke:#d97706,color:#92400e CALLER["Caller\\nPSTN / mobile"]:::tel SIP["Telephony\\nTwilio / Telnyx / SIP trunk"]:::tel CALLER <-->|"audio stream"| SIP WS["Media WebSocket\\nbidirectional audio"]:::speech SIP <--> WS VAD["Voice activity detection\\n+ endpointing"]:::speech STT["Streaming STT\\nDeepgram / Whisper / Azure"]:::speech WS --> VAD --> STT ORCH["Voice orchestrator\\nturn-taking + state"]:::agent STT --> ORCH LLM["LLM agent loop\\nfast model, short context"]:::agent ORCH <--> LLM TOOLS["Tool calls\\ncalendar, CRM, order lookup, SMS"]:::tool LLM -->|"function call"| TOOLS TOOLS -->|"result"| LLM TTS["Streaming TTS\\nElevenLabs / Cartesia / OpenAI"]:::speech ORCH --> TTS TTS --> WS OBS["Call recording, transcript,\\nlatency + tool traces"]:::mon ORCH -.-> OBS LLM -.-> OBS

Reference architecture — every stage streams; the orchestrator owns turn-taking, barge-in, and when to call a tool vs keep talking

sequenceDiagram autonumber participant C as Caller participant T as Telephony (Twilio) participant S as Streaming STT participant A as Agent (LLM) participant F as Tool / API participant V as Streaming TTS C->>T: speaks ("book me Tuesday at 3") T->>S: audio chunks (continuous) S-->>A: partial transcript (interim) S->>A: final transcript (endpoint detected) A->>A: decide: reply or call tool A->>F: check_availability(tuesday, 3pm) F->>A: slot open A->>V: response tokens (streamed) V-->>T: audio chunks (streamed) T-->>C: hears reply, ~700ms after finishing speaking Note over A,F: If caller speaks again mid-reply,
TTS stops immediately (barge-in)

One conversation turn — transcription, tool call, and streamed reply, target under ~800ms end to end

Call lifecycle

The loop (listen → decide → act → speak) repeats each turn until the call resolves or hands off to a human

The five pipeline stages

1 — Telephony

• Number: Twilio, Telnyx, or SIP trunk into your existing PBX
• Media streaming: Twilio Media Streams (or equivalent) opens a WebSocket carrying raw audio chunks both ways — this is what lets you process audio in real time instead of waiting for a recorded file
• Inbound and outbound: inbound answers calls to a number; outbound dials from your system (reminders, follow-ups) — outbound calling to consumers has stricter consent rules (TCPA in the US, PECR in the UK/EU)

2 — Streaming speech-to-text (STT)

• Providers: Deepgram, AssemblyAI, Azure Speech, or Whisper-based streaming services
• Partial vs final transcripts: partials update as the caller talks (for the agent to start "thinking early"); a final transcript fires on endpoint detection (the system's best guess that the caller finished a thought)
• Endpointing tuning: too aggressive and the agent interrupts; too relaxed and replies feel slow. This is one of the highest-leverage tuning knobs in the whole system

3 — Agent loop (the LLM)

• Model choice: a fast, low-latency model (e.g. a smaller Claude, GPT, or Gemini variant) for turn-taking; escalate to a stronger model only for complex reasoning steps
• Prompt: persona, business info, conversation rules, and available tools — kept short, since every token adds latency
• State: conversation history, caller identity (from caller ID or earlier verification), and call metadata
• Streaming output: the LLM's tokens stream directly into TTS as they're generated, rather than waiting for the full reply

4 — Tool calling

• Typed functions: check_availability, book_appointment, lookup_order, send_sms, transfer_call — same pattern as any AI agent's tools
• Filler phrases: if a tool call takes more than ~500ms, the agent says something ("let me check that for you") so the line doesn't go silent
• Timeouts: every tool call needs a hard timeout and a fallback ("I'm having trouble pulling that up — let me have someone call you back")

5 — Streaming text-to-speech (TTS)

• Providers: ElevenLabs, Cartesia, PlayHT, or OpenAI's voice models
• Streaming: audio starts playing from the first generated chunk, not after the full sentence is synthesized
• Voice selection: match tone to the business; disclose that it's an AI voice per your jurisdiction's rules

Latency budget

People notice a pause above roughly 1 second and start talking over the agent or assuming the call dropped. A well-tuned agent keeps the full turn — from the caller finishing a sentence to the agent's audio starting — under about 700–900ms. That budget is split roughly like this:

The biggest win is almost always streaming every stage rather than waiting for each one to fully finish before starting the next — transcribe while the caller talks, generate LLM tokens while streaming them to TTS, and start playing audio from the first synthesized chunk.

Tool calling mid-call

This is what separates a voice agent from an IVR menu: it can actually do something. The pattern is the same structured tool-calling used by any AI agent — the LLM returns a typed function call instead of text, your backend executes it against a real system, and the result feeds back into the conversation.

Interruptions and barge-in

Real conversations have interruptions. A voice agent that can't be talked over feels robotic and frustrates callers who already know what they want.

• Continuous listening: voice activity detection runs on the input stream even while the agent's own audio is playing
• Immediate stop: the moment the caller starts speaking, TTS playback is cut and the unspoken remainder is discarded
• Context repair: the agent should acknowledge the interruption naturally ("sorry, go ahead") rather than restarting its previous sentence
• False positives: background noise or a cough can trigger false barge-in; noise suppression and a short confirmation window reduce this

Managed voice platforms (Vapi, Retell, LiveKit) handle this out of the box. A custom pipeline needs an explicit interrupt signal wired from the input VAD into the TTS output stream — this is one of the more fiddly parts of building from scratch.

Managed platform vs custom build

For most businesses, Vapi or Retell gets a production-grade voice agent live in 3-5 weeks. Build custom only when you have a specific reason — self-hosting for data residency, extreme call volume, or telephony features those platforms don't expose. The work that actually differentiates a good voice agent — the script, the tools, the escalation rules, the testing — is the same either way.

Building one? stackcone builds production voice agents on Vapi, Retell and LiveKit — telephony setup, tool integrations, latency tuning and call testing included. Talk to stackcone →

Recommended stack

Implementation plan

Testing and monitoring

For the accuracy and regression-testing side of the agent itself (not just the audio pipeline), see AI agent testing and evaluation strategies.

Cost & effort estimate

FAQ

What is the fastest way to build an AI voice agent?

Use a managed voice platform (Vapi, Retell, or Bland) that bundles telephony, streaming STT/TTS, and tool calling behind one API. Configure a prompt, connect tools, and you have a working agent in days. Build the pipeline yourself only when you need self-hosting, unusual latency budgets, or telephony features those platforms don't expose.

What causes latency in a voice agent and how do you fix it?

The main sources are STT finalization, LLM time-to-first-token, tool calls, and TTS generation. Fix it by streaming every stage, using a fast model for turn-taking, pre-fetching data where possible, and keeping tool calls under 300ms with cached or async fallbacks.

How do you handle interruptions (barge-in)?

Run voice activity detection continuously, even while the agent speaks. When the caller starts talking, stop TTS playback immediately and feed the new input to the agent. Most voice platforms handle this natively; a custom build needs an explicit interrupt signal.

Can a voice agent call functions or APIs mid-call?

Yes — the same structured tool-calling used by any AI agent. The LLM returns a typed function call, your backend executes it, and the result feeds back into the conversation. Keep calls fast and give the agent a filler phrase for anything slower than ~500ms.

Who builds custom AI voice agents?

stackcone builds production AI voice agents on Vapi, Retell and LiveKit for businesses, including telephony setup, tool integrations, latency tuning, and call testing. See our AI voice agents for business guide for the buyer-side view, or contact us to scope a build.

Glossary