About this eBook

Voice is the hardest agent surface to get right. Text agents can pause, retry and dump a wall of tokens on the user. A voice agent has roughly 800 milliseconds before the caller thinks the line went dead, and no scroll-back to hide a bad answer in.

This book walks the whole build: designing a conversation that survives interruption, wiring server-side tools so the agent can actually look up an order or write to a CRM, connecting a phone number, and putting numbers on quality so you know whether last week's prompt change helped or hurt.

Every chapter ends with a working artefact. By the last page you have a deployed inbound-and-outbound voice agent, a regression suite of recorded test calls, and a cost model that tells you what each conversation actually costs.

What you will be able to do

• Ship a low-latency speech-to-speech agent and understand every millisecond in the latency budget
• Write server tools the agent can call mid-sentence without an awkward silence
• Ground answers in a knowledge base so the agent stops inventing policies
• Connect the agent to a real phone number over Twilio or native SIP trunking
• Handle interruption, barge-in, silence and the caller who says nothing for nine seconds
• Build a recorded-call regression suite and score changes before they reach production
• Model per-minute cost and pick voices, models and turn settings that fit the budget

Table of contents

1. Why voice agents fail — latency, turn-taking and the eight-hundred millisecond rule
2. The ElevenLabs Agents platform: models, voices, and the conversation loop
3. Your first agent — system prompt, first message, and guardrails that hold
4. Voice design: picking, cloning and tuning a voice people will stay on the line for
5. Server tools and client tools — letting the agent do things, not just say things
6. Knowledge bases and RAG for voice: short answers, cited sources, no rambling
7. Telephony: Twilio inbound, outbound campaigns, and native SIP trunking
8. Interruption, barge-in, backchannels and graceful silence handling
9. Human handoff — when to transfer, how to pass context, what to log
10. Evaluation: recorded test calls, LLM-as-judge rubrics, and regression gates
11. Observability, transcripts, PII redaction and call retention policy
12. Cost engineering and scaling to a thousand concurrent conversations

Who it is for

• Developers who can read JavaScript or Python and want a shipping voice agent, not a demo
• Agency owners packaging voice receptionists and outbound qualification for clients
• Support leads evaluating whether voice deflection is worth the operational cost

Questions