✦ Diljit's Corner ✦

Voice agents are the easiest AI product to demo and one of the hardest to ship. In a demo, a founder speaks clearly into a good microphone, asks the happy-path question, and the agent sounds like magic. In production, the caller is on a speakerphone in a car, a dog is barking, they interrupt the agent halfway through a sentence, and they wanted something your prompt never anticipated. The gap between those two situations is where voice AI products die.

Hamel Husain made an argument about LLM products in general that I think applies with double force to voice. The teams that ship are rarely the ones with the cleverest prompts or the newest model. They're the ones that built an evaluation system early, because success in this space mostly means iterating fast, and you can't iterate fast on something you can't measure. Most teams skip straight to tweaking prompts anyway, which in his words is like "attempting to fly an airplane without instruments."

This post is my attempt to assemble a field guide for voice agents specifically: what to measure, how to test before launch, how to judge conversations at scale, and how to observe what's actually happening in production. I've drawn on the best writing I could find on the subject — from Hamel Husain, Coval, Langfuse, Hamming, and ElevenLabs, plus a longer tail of engineering blogs from Daily/Pipecat, AssemblyAI, Braintrust, Cekura, Roark, LiveKit, and others linked throughout.

Why Voice Is Inherently Harder: Cascading Systems

A text chatbot is one model call. A production voice agent is a pipeline: audio comes in over telephony or WebRTC, a speech-to-text (STT) model transcribes it, an LLM decides what to say and which tools to call, a text-to-speech (TTS) model speaks the answer, and the audio goes back out — all while a turn-detection layer decides whether the caller has finished speaking or is interrupting.

The cascade is what makes evaluation hard. An error at any stage propagates downstream. If the STT hears "I want to handle" instead of "I want to cancel," the LLM will respond fluently and confidently to the wrong request. Hamming's engineers use exactly this example to show the limit of transcription metrics: the words were almost right, the intent was completely wrong. On top of that, everything happens under a real-time clock. By the time a slow response finally arrives, three seconds in, the caller has usually started talking again, and now two failure modes are stacked on top of each other.

One more thing before we get to metrics: transcripts lie. Several practitioners (Roark, Bland, Cekura) independently estimate that around 40% of production voice failures never show up in the text transcript. Talk-over. A cough that triggers a false interruption. The agent pausing awkwardly mid-sentence. An "uh huh" backchannel treated as a full turn. If your evaluation only ever looks at text, you're blind to all of it. Keep the audio.

Start with Eval Fundamentals, Not Voice Exotica

Before the voice-specific machinery, it's worth internalizing Hamel's three-level structure, because everything the voice vendors sell is an elaboration of it:

• Level 1 — unit tests and assertions. Cheap, deterministic checks that run on every change: the agent never leaks a raw UUID, always offers human escalation, never repeats a card number back. In voice-land these become "must-always / must-never" rules — Coval's guide has good vertical-specific examples (HIPAA disclosures in healthcare, mini-Miranda in collections). These are binary, they block deploys, and there should eventually be hundreds of them, harvested from real failures.
• Level 2 — human review and LLM-as-judge. Scoring whole conversations against rubrics, with humans calibrating the judge (more on this below).
• Level 3 — A/B testing. Once you have real traffic, measure whether changes actually move business outcomes.

The other fundamental that survives the jump to voice untouched: look at your data. Hamel's rule is to remove all friction from reading traces, and his stopping heuristic is to keep reading until you stop learning new things. For voice this means listening to calls, not just skimming transcripts — Coval quotes a healthcare engineer on why: "With the audio there's so much — I've got noise, I've got barking dogs, I've got kids, I've got accents." No dashboard replaces this.

The Metrics That Actually Matter

The literature converges on roughly five metric families. Here's the condensed version, with the consensus targets:

Three of these need unpacking, because the headline numbers hide most of the story.

Word Error Rate is necessary but increasingly distrusted

WER (substitutions plus deletions plus insertions, divided by total words) is the standard transcription metric, and the usual enterprise bar is under 5%. But 2026 has been the year the industry turned on it. AssemblyAI published a piece bluntly titled "Word Error Rate Is Broken", and the argument lands: WER penalizes "gonna" → "going to" exactly as much as a drug name transcribed into gibberish, and flawed ground-truth transcripts can make genuinely better models score worse.

Two replacements are gaining traction. Semantic WER has an LLM judge whether a transcription difference would actually change the agent's response; the Pipecat team's open-source STT benchmark is built on this idea. Critical-entity recall asks a simpler question: did the phone number, the account code, the medication name survive? Braintrust ran an STT evaluation across six providers and found them nearly tied on general accuracy but wildly divergent on structured tokens, and a cheap LLM post-correction pass improved entity recall by 12–14% for every single provider. So if your agent handles account numbers, measure entity recall separately, or you'll ship a "95% accurate" agent that fails every verification step.

Latency: the gap between marketing and production

Humans respond to each other in about 200–300ms, and every vendor advertises sub-300ms components. Yet Hamming's data across thousands of production agents shows a median voice-to-voice response of 1.4 to 1.7 seconds. That's five times the human baseline, with P95 above 4 seconds and P99 reaching 8–15 seconds, which is where callers simply hang up. The practical target most sources agree on is 800ms, budgeted roughly as STT 100–400ms, LLM 300–1,000ms, TTS 100–500ms, plus network and turn detection.

Notice the LLM's share: typically around 70% of total latency. This explains a production reality that Daily's voice LLM benchmark documents well — the smartest reasoning models are simply too slow for voice, so production agents run fast-tier models and make up the intelligence gap with better context engineering. Whatever stack you run, two habits follow from all this. Instrument every stage boundary so you know where the time goes. And alert on tail percentiles, never averages; as ElevenLabs puts it, a P99 problem at 10,000 daily calls is 500 real people having a bad conversation.

Containment is a vanity metric

Containment (calls that never reach a human) is the number executives ask for, and both Coval and ElevenLabs warn against optimizing it. An agent that traps a frustrated caller in a loop "contains" the call and fails the customer; a high containment rate paired with a low first-call-resolution rate is the signature of circular conversations. The better north star is resolution: was the caller's actual problem solved, and did they not call back within 24 hours? Watch escalation appropriateness in both directions — escalating too eagerly wastes the agent, escalating too late (or never) burns trust and, in regulated domains, creates liability.

Testing Before Production: Simulation Is the Unit Test of Voice

You can't manually QA your way to a reliable voice agent; Roark's team estimates traditional QA reviews maybe 1–2% of calls. The workhorse technique is simulation: an LLM-driven synthetic caller with a persona and a goal phones your agent, ideally over the real telephony path rather than a text shortcut, and the conversation gets scored automatically. Every serious platform in this space (Coval, Hamming, Cekura, Roark, plus the built-in offerings from Retell and Vapi) is some variation on this idea. One practitioner in a builder thread I read described replays well: it's regression testing for conversations, and it's the only way to catch an update that quietly made the agent dumber on a query it used to handle perfectly.

The design of your scenario suite is where the real engineering lives. Distilling the questions Coval's Brooke Hopkins poses, a good suite has to answer three things:

A few practical notes on building the suite:

• Grade personas by difficulty. Coval's ladder runs from easy (clear speech, single intent) through medium (slight accent, filler words) and hard (heavy noise, emotional caller, mid-call topic pivots) to adversarial. Ship gates run the easy tier; the full ladder runs weekly.
• Beware the people-pleaser problem. LLM-simulated callers are unnaturally cooperative — they don't stammer, ramble, or get confused unless you force them to. If all your synthetic callers are polite and articulate, your pass rates are fiction. Inject disfluencies, impatience, and confusion deliberately.
• Mind the combinatorics. Cekura's math: 5 languages × 3 emotional states × 3 speaking speeds × 4 interruption behaviors × 5 background-noise profiles is already ~900 variations for a single scenario. You can't run everything always. Use graduated testing: a small blocking suite per deploy (minutes), an extended suite nightly, the full matrix weekly.
• Regressions hide in tiny changes. Coval documents a team that changed its greeting from "How can I help you?" to "What can I help you with?" and silently dropped booking completion by 12% for two weeks. This is why the per-deploy suite must run on every prompt change, not just code changes.

LLM-as-Judge: Useful, but Only After Calibration

Automated conversation scoring means an LLM judge reads (or increasingly, listens to) each conversation and scores it against rubrics: did the agent complete the task, follow the workflow, stay in scope, handle the interruption gracefully? Before the mechanics, though, an uncomfortable truth that comes up constantly when practitioners compare notes: many of these qualities are subjective, and the disagreement starts inside your own company. What a product manager hears as a pleasantly "natural" response, a support lead hears as way too informal for the brand. Defining the baseline is a genuine battle, and no tool resolves it for you — the rubric conversation with your stakeholders is the work.

Once you have rubrics people actually agree on, two rules make automated judging trustworthy rather than theater:

• One metric, one dimension. A rubric like "knowledge-base utilization" that bundles accuracy and completeness produces scores nobody can act on. Unbundle everything; prefer binary checks over 1–10 scales wherever possible.
• Calibrate against humans, iteratively. Run the judge on 50–100 real calls, have humans score a subset with the identical rubric, compare, fix the rubric, repeat. Coval's bar is >85% agreement on binary metrics before you trust the judge unsupervised. Hamel adds an important statistical caveat: on imbalanced data (most calls are fine), raw agreement flatters the judge — measure precision and recall on the failure class instead.

And the calibration never fully ends. Periodically re-sample judged calls for human review, because your product, callers, and failure modes drift. The judge's written critiques have a bonus use Hamel points out: they're high-quality curated data if you ever fine-tune.

Observability: One Trace Per Conversation, Spans Per Stage

Testing tells you how the agent behaves on conversations you thought of. Observability tells you what's happening on the ones you didn't. The distinction Cekura draws is the right one: monitoring asks "is the service up?"; voice observability asks "why did this conversation fail?" — and answering that requires connecting telephony, STT, LLM, tools, and TTS in a single trace.

The implementation pattern is now fairly standard (Langfuse, LiveKit, Arize Phoenix, and Hamming all describe versions of it, generally on OpenTelemetry):

• One trace per conversation, one span group per turn, with child spans for each pipeline stage: audio in → STT → LLM (with every tool call) → TTS → audio out. Each span carries its latency, so any slow turn decomposes instantly into which stage was slow.
• Attach metadata that makes debugging possible: prompt version, model and parameters, STT confidence scores, endpointing timings, agent version, conversation ID. When a bad call surfaces, you want to know exactly which configuration produced it.
• Keep the raw audio as a trace attachment. Braintrust and LiveKit both emphasize replaying exactly what the agent heard — the transcript version of a failure and the audio version are often different stories. LiveKit's observability stack pairs the trace view with a scrubbable audio player for precisely this reason.
• Track cost per conversation alongside quality. Voice margins are made of STT minutes, LLM tokens, and TTS characters; a prompt change that improves quality 2% and doubles token spend is a decision, and you want it to be a visible one.

Coval frames the data you need as four pillars — conversation content (transcript and audio), context signals (is this the caller's third attempt today?), outcome data (resolution, callbacks), and performance metrics (per-stage latency). However you organize it, the operational pattern on top is consistent across sources: run your full eval suite on a sample of production calls (10–20%), run cheap anomaly detection on 100% (latency spikes, early hang-ups, silence gaps, repeated phrases), and route flagged calls to human review. Sensible starting alerts: P95 voice-to-voice latency above 1 second, task success below 80%, intent accuracy below 90%, any compliance failure at all.

Then close the loop. Every genuine production failure becomes a regression test: capture the call, reproduce it in simulation, generalize it into scenario variations, and add it to the suite so that class of failure can never silently ship again. This is the voice equivalent of Hamel's "look at your data, extract new assertions" advice, and it's how test suites grow to hundreds of meaningful cases instead of staying at twenty synthetic ones forever.

A Production Metrics Menu, Stage by Stage

It helps to keep one organizing question in mind when picking production metrics: what did the caller feel? Every stage of the pipeline maps to a feeling. The telephony layer decides whether they could hear you at all. STT decides whether they were heard. The LLM decides whether they were understood. TTS and turn-taking decide whether the agent felt human. Outcomes decide whether they got what they came for. A metric that doesn't ladder up to one of those feelings is probably vanity. Here's the menu I'd pick from, stage by stage:

Two notes on using this menu. First, don't dashboard all of it on day one; start with one metric per stage plus resolution and sentiment trajectory, then add depth where the failures actually cluster. Second, watch the pairs. High containment with low resolution means callers are trapped. Fast latency with a rising repeat-yourself rate means the STT is struggling. Good task success with declining sentiment means you're technically completing tasks while annoying people. The single metrics tell you what happened; the pairs tell you why, and the "why" is what makes the product better rather than just better-measured.

Notes from the Trenches

The vendor guides are polished, so it's worth balancing them against what people building these systems say when they're just comparing notes with each other. Reading through practitioner discussions (r/ai_agents has some good threads on voice evals), a few themes come up that the official literature underplays:

• Post-call user feedback is the most underrated signal. Several builders independently reported that simply asking callers about naturalness and frustration after the call taught them more about "does this sound human?" than any automated metric. Your users are grading you anyway; you might as well collect it.
• The teams that take this seriously run QA on 100% of calls. One team described their setup as automated analysis of every single call (interruption handling, latency, agent scoring), stress tests with accents, noise and rapid-fire questions, and human managers reviewing only the flagged edge cases. That division of labor — machines for coverage, humans for judgment — matches everything else in this post.
• Latency and conversational flow beat fancy voices. A recurring observation from people who have tested many platforms: agents don't fail because the voice sounds robotic, they fail at interruptions and contextual handovers. Boring infrastructure work (streaming, stable telephony) does more for perceived humanness than a premium voice model.
• There is still no standardized voice benchmark. Everyone feels this gap. Building one would require a shared dataset of audio with all its real-world messiness — noise, accents, interruptions — which, as one commenter put it, is a whole research project in itself. The academic attempts are starting to appear, but for now every team builds its own ground truth.

Where This Is Heading

Two developments are actively reshaping this playbook. The first is audio-native evaluation: judges that score the audio itself (prosody, awkward pauses, talk-over, tone under frustration) rather than the transcript. Given how much fails invisibly in text, this is quickly moving from novelty to necessity. The second is speech-to-speech and full-duplex models, agents that listen while they speak. These break transcript-based evaluation almost entirely, because timing is semantics when the agent can be interrupted mid-word. Roark's work on testing full-duplex agents proposes waveform-level metrics for this world: barge-in stop latency measured from the audio itself, false-interruption rate, backchannel handling. None of them exist in any transcript.

For sequencing all of this, Coval's maturity curve is a useful map: prompts alone get you to ~70% of the way; manual QA to ~85%; programmatic evals wired into CI get you to ~95% over a couple of months; and production feedback loops are what close in on 99% over six to twelve. The uncomfortable implication is that the last 4% costs more engineering than the first 95% — and at 200,000 calls a month, a 1% failure rate is still 2,000 people having a bad day. That's the whole argument for treating evals as infrastructure rather than a pre-launch checkbox.

Final Thought

Strip away the vendor branding — three levels, four pillars, five dimensions, six pillars — and every serious framework in this space is the same loop: define what good means for your calls, test it before every deploy, watch it continuously in production, and turn every real failure into a test. The teams that win at voice AI aren't the ones with the best agent on day one; they're the ones with the best evaluation infrastructure on day one hundred. Build the instruments before you fly the plane.

Sources & Further Reading

• Hamel Husain — Your AI Product Needs Evals — the foundational text on LLM eval systems.
• Coval — Voice AI Agent Evaluation Guide — maturity curve, personas, judge calibration, observability pillars.
• Langfuse — Evaluating and Monitoring Voice AI Agents — online/offline eval split and tracing.
• Hamming — Voice Agent Evaluation Metrics Guide — the most numbers-dense reference; see also their latency deep dive.
• ElevenLabs — Voice Agent Evaluation Framework: 6 Pillars.
• Daily/Pipecat — Benchmarking LLMs for Voice Agents and Benchmarking STT — open-source benchmarks, semantic WER.
• AssemblyAI — Word Error Rate Is Broken.
• Braintrust — How to Evaluate Voice Agents and their STT evaluation study.
• Cekura — What Is Voice Observability? and their evaluation metrics guide.
• Roark — Testing Full-Duplex Voice Agents.
• LiveKit — Understand and Improve Agent Latency.
• Arize — Voice Application Tracing and Evaluation — open-source audio evals with Phoenix.