Building Voice Agents on Claude
A "voice agent" is a system you can talk to out loud that talks back — a phone support line, a hands-free assistant, a kiosk. Claude can be the brains of…
Claude guide · as of July 10, 2026 · 5 minutes read · details change — confirm current specs on claude.ai
A "voice agent" is a system you can talk to out loud that talks back — a phone support line, a hands-free assistant, a kiosk. Claude can be the brains of one. But let's be blunt up front: Claude does not hear or speak. It reads text and writes text. If you want a voice agent, you supply the ears and the mouth yourself and wire Claude in the middle.
The honest starting point
Claude ships no consumer voice mode. There is no always-on "tap and talk" feature like ChatGPT's Advanced Voice or Gemini Live. If your goal is simply to have a spoken conversation with an AI on your phone, Claude is the wrong product today — reach for a competitor. See /claude/voice for that full picture.
This guide is about the other thing: building your own voice experience, where Claude is one component in a pipeline you control through the Anthropic API.
How a voice agent actually works
Under the hood, every voice agent is the same relay race:
microphone → speech-to-text → Claude → text-to-speech → speaker
Claude only ever sees the transcript and only ever returns text. It never touches the raw audio. That means you assemble three separate services, and only the middle one is Anthropic's.
| Stage | What it does | You bring | Honest note |
|---|---|---|---|
| Speech-to-text (STT/ASR) | Turns spoken audio into text | A transcription provider | Accents, noise, and crosstalk all degrade this — errors flow downstream into Claude |
| Claude (the API) | Reads the transcript, decides what to say or do | An Anthropic API key | Text in, text out — no audio understanding |
| Text-to-speech (TTS) | Turns Claude's reply into spoken audio | A voice/synthesis provider | Voice quality and latency vary widely |
Because Claude is text-only here, anything the microphone captured beyond words — tone, emotion, a sigh, background sound — is lost unless your STT layer explicitly surfaces it as text.
Choosing your Claude model
Voice is unforgiving about delay, so model choice is really a speed-versus-smarts tradeoff.
| Model | Best for | Tradeoff |
|---|---|---|
| Haiku 4.5 | Fast turn-taking, simple routing, FAQ-style replies | Less depth on hard reasoning |
| Sonnet 5 | Balanced everyday agents | Slightly slower than Haiku |
| Opus 4.8 (flagship) | Complex multi-step reasoning, tool use | Slowest — noticeable in live conversation |
For most real-time voice agents, start with Haiku 4.5 and only escalate to a bigger model for the turns that genuinely need it. Confirm the current lineup at claude.ai / anthropic.com — model tiers move (as of July 2026).
Latency is the real engineering problem
The hard part isn't wiring the pieces together — it's making the whole loop feel like a conversation instead of a walkie-talkie. Every hop adds delay, and they stack:
- Stream everything. Feed audio to STT as the person speaks; stream Claude's tokens out as they generate; start TTS on the first sentence rather than waiting for the full reply.
- Handle barge-in. Real people interrupt. Your agent needs to stop talking and start listening mid-sentence — that's your plumbing, not Claude's.
- Keep prompts tight. A shorter system prompt and a trimmed conversation history mean faster first tokens. Consider prompt caching for the stable parts.
- Let Claude act. For "book it," "check my order," "transfer me," define tools (function calling) and, where it fits, connect real systems via MCP.
None of this is magic. A responsive voice agent is a genuine engineering project, not a checkbox.
When this is the wrong tool
Be honest with yourself before you build:
- You just want to talk to an AI hands-free. Don't build anything — use ChatGPT or Gemini's consumer voice mode. Claude has no equivalent.
- You need it fully offline or maximally private. A cloud pipeline sends audio and transcripts to third parties. A local model with on-device STT/TTS may fit better, at the cost of quality.
- You only need occasional dictation. Your phone's built-in voice typing already does that for free.
- You can't tolerate a knowledge cutoff or a wrong answer. Claude can be confidently wrong, and in a voice channel there's no text on screen to double-check. Add web search, retrieval, or a human handoff for anything high-stakes.
A minimal build checklist
- Pick an STT provider and test it on your real audio (accents, phone lines, noise)
- Get an Anthropic API key and choose a starting model (Haiku 4.5 is a sane default)
- Pick a TTS voice and confirm its streaming latency is acceptable
- Stream every stage; measure end-to-end round-trip time
- Implement barge-in / interruption handling
- Define tools for real actions; add a human-handoff path
- Decide what audio/transcripts you store and disclose it to users
Popular STT and TTS vendors change often and pricing shifts — check the current market rather than trusting any name you saw in an old tutorial (as of July 2026).
The short version: Claude makes an excellent reasoning core for a voice agent, and a poor voice product on its own. Bring your own ears and mouth, obsess over latency, and be clear-eyed about the cases where a competitor or a local setup simply serves the user better.
Related
- /claude/voice — why Claude has no consumer voice mode, and what to use instead
- /claude/api — building on Claude through the Anthropic API
- /claude/build — a plain-English overview of making things with Claude
