Monitors showing call analytics dashboards on a dim operations desk
Two dashboards, one agent: the one engineering watches and the one finance believes.

Every voice AI deployment I have worked on eventually develops two dashboards.

One belongs to engineering. It has p95 latency, word error rate, tool-call failures, endpointing accuracy. It is correct, it is detailed, and no executive has ever read it twice.

The other belongs to the business. It has containment rate, cost per call, CSAT, and a number labelled “annualised savings” that someone computed in a spreadsheet in month one and has been quoting ever since.

The gap between those two dashboards is where voice AI programmes quietly fail. Engineering ships a latency improvement nobody credits. Finance celebrates a containment number that is actually a customer hanging up in frustration. Six months later the renewal conversation goes badly and nobody can say precisely why.

This post is the bridge: the full set of voice AI agent metrics worth capturing in production, organised in five layers, with formulas, published target ranges, and — importantly — how each one gets gamed. I write this from the seat of running deployed engineering for voice agents in home services, where a missed call is not a support ticket, it is a lost job worth several hundred dollars. That framing changes which metrics matter.

If you want the testing and observability side of this — simulation, LLM-as-judge, tracing — I covered that separately in Evals, Testing, and Observability for Voice AI Agents. This post is about what you put on the scorecard after the agent is live.

The Five-Layer Stack

Voice metrics form a causal chain, and reading them out of order is how teams misdiagnose everything.

Layer Question it answers Who cares Cadence
1. Mechanics Does it feel like a conversation? Engineering Real-time alerting
2. Comprehension Did it understand and do the right thing? Engineering + Ops Daily
3. Experience How did the caller feel about it? Ops + CX Weekly
4. Compliance Can this survive an audit? Legal + Risk Continuous, 100% audit
5. Money Is it worth doing? Finance + Exec Monthly / quarterly

The rule: a metric in layer N is only interpretable with the layer below it in view. Containment without repeat-contact rate is a vanity number. CSAT without abandonment is survivorship bias. Cost per contact without resolution is a cost that moved, not a cost that vanished.

Layer 1 — Conversation Mechanics

This layer decides whether the caller experiences a conversation or an interrogation. It is entirely technical and entirely felt.

Turn latency (the one that matters most)

Measure mouth-to-ear: from the moment the caller stops speaking to the moment your agent's audio reaches their ear. Not model time. Not time to first token in isolation. The whole gap, including endpointing delay, ASR finalisation, LLM inference, TTS synthesis, and network transit.

  • Human inter-turn gaps in natural conversation sit around 200ms.
  • Bluejay reports delays past 800ms associate with roughly 40% higher abandonment.
  • Report p50 and p95. Never the mean. The mean hides the 5% of turns where a slow tool call left three seconds of dead air, and those are the calls people remember.

Decompose it into spans so a regression is diagnosable: endpoint_detect → asr_final → llm_first_token → llm_complete → tts_first_byte → audio_out. Nurix calls this response assembly latency, and it is the difference between “the agent got slower” and “our CRM lookup p95 tripled after their deploy.”

End-of-turn detection accuracy

Two failure modes, opposite symptoms, same root metric:

  • Premature endpointing — the agent interrupts a caller mid-thought. Feels rude.
  • Late endpointing — the agent waits after the caller finished. Feels slow or broken.

Track both as separate rates, not one accuracy score, because the fixes point in opposite directions. This gets far worse with callers who pause mid-sentence to read a card number or check an address — exactly the moments where an interruption is most costly.

Barge-in and stop time

When a caller interrupts, how fast does the agent actually stop talking, and does it retain the context of what it was saying? Cekura notes agents typically resume within about two seconds. An agent that plows through an interruption is the single fastest way to make a caller hang up.

Pace and talk ratio

  • Words per minute. Production agents cluster around 200 WPM; Cekura's evaluation found more than half of agents pace above 190 WPM, at the top of natural conversational range. Fast reads as robotic and hurts comprehension for older callers and non-native speakers.
  • Talk ratio — agent speaking time over total speaking time. Past 0.80 the agent starts to feel domineering. A high talk ratio usually means over-long confirmations and prompt bloat, both fixable.

Dead air / silence rate

Percentage of calls containing a silence gap over a threshold. Retell suggests alerting on silences over 3 seconds. In practice I would alert lower — 2 seconds of unexplained silence on a phone call already reads as a dropped line, and callers say “hello? hello?” which then becomes a spurious turn your ASR has to handle.

Composite scores like Bluejay's Voice Agent Quality Index — a 0–100 blend of interruptions, missed response windows, and latency — are useful for trend lines and vendor comparison. They are useless for debugging. Keep the composite for the exec deck and the components for the on-call.

Layer 2 — Comprehension and Correctness

Mechanics decide whether the caller stays. This layer decides whether staying was worth it.

Transcription accuracy — but weighted

Word error rate is the standard ASR measure, and it is close to useless unweighted. Cekura puts it well: 99% WER is not enough if the missed word is “cancel” rather than “confirm.”

Track critical-token error rate alongside raw WER: errors on names, addresses, dates, monetary amounts, phone numbers, and the small set of decisive verbs in your domain. Weight those fully; weight filler words at zero. For tonal or morphologically rich languages, character error rate is the better base measure.

Intent recognition accuracy and coverage

Two different numbers that teams constantly conflate:

  • Accuracy — of the intents you support, how often do you pick the right one? Published targets: CloudTalk suggests 90%+ on top intents, 80%+ on the long tail; Bluejay puts enterprise launch expectations at 80–85%.
  • Coverage — (calls landing in a fully supported intent ÷ total calls) × 100. This is the ceiling on containment. You cannot automate what you have not modelled, and no amount of prompt tuning raises it.

Coverage is the metric that tells you what to build next. If 12% of calls are “reschedule an existing appointment” and you do not support it, that is a roadmap item, not a model problem.

Semantic accuracy

Distinct from both WER and intent classification: did the agent grasp the meaning, including implication and correction? Nurix's formulation is (correct intent or workflow selections ÷ audited utterances) × 100, with launch expectations in the 80–85% band and 90%+ as the mature target.

The failure this catches: a caller says “I need someone out tomorrow, actually make it Thursday.” Perfect transcription, correct intent, wrong date. WER says 0%. Intent accuracy says pass. The customer gets a truck on the wrong day.

Task success rate

The metric I trust most in this layer, and the one that requires real engineering to capture: did the thing the caller wanted actually happen in the system of record?

Not “the agent said it booked the appointment.” Did the appointment appear in the CRM, with the right customer, at the right time, and did the confirmation send? Nurix calls this outcome-verified — verify against downstream state, not against the transcript. Every voice team I know overestimates its success rate until it starts doing this, usually by 10–20 points.

Hallucination rate

Percentage of calls where the agent asserted a fact, policy, price, or capability not supported by its knowledge source. Bluejay cites research finding roughly 1% of audio transcriptions containing entirely fabricated phrases — and that is at the transcription layer, before the LLM adds its own.

In regulated or price-quoting contexts this is a release blocker, not a KPI. Grade it per node, not per call, so you know which part of the flow invents things.

Workflow adherence and instruction following

Did the agent follow the prescribed flow — collect consent before recording, verify identity before disclosing account details, offer the required alternatives before closing? Cekura reports that more than 20% of runs flag some form of workflow adherence gap, and that instruction-following failures at multi-step gates dominate the flagged categories.

That number should be sobering. Aggregate quality scores routinely look fine while the agent is quietly skipping steps.

Context retention

Does the agent remember what the caller said four turns ago? CloudTalk names forced repetition as “the single most consistent complaint” about voice AI. Measure it as the rate of calls where the agent re-asks for information already provided. It is a simple transcript-level check and it correlates brutally well with CSAT.

Fallback rate — read carefully

The share of turns where the agent asks the caller to repeat or rephrase. Some fallback is healthy: an agent that asks for clarification instead of guessing is behaving correctly. A fallback rate of zero next to a mediocre task success rate means your agent is confidently wrong. Read fallback against task success, never alone.

Layer 3 — Customer Experience

These are the metrics your customers would recognise as being about them.

CSAT, CES, NPS — and how to not fool yourself

  • CSAT — post-call satisfaction. Must be segmented by intent, by outcome, and by time of day. A single blended score hides the fact that your agent is excellent at appointment lookups and terrible at billing disputes.
  • CES (Customer Effort Score) — how hard was it to get the thing done. CloudTalk argues this is the most apt measure for voice AI specifically, because the entire promise of automation is easier, and I agree. If I could only keep one survey metric, it would be this one.
  • NPS — brand-level, slow-moving. Useful as a before/after guardrail for the deployment as a whole, not as an operational metric.

The trap in all three is survivorship. Surveys only reach people who stayed on the line. Your angriest callers hung up at turn two and will never take your survey. Always report survey scores next to abandonment.

Abandonment and disconnection reason

Percentage of calls the caller ends before resolution — and critically, where. Bucket disconnections by turn index and by preceding event:

  • Abandoned in the first 3 turns → your greeting, your latency, or the fact that they wanted a human immediately.
  • Abandoned mid-flow after a long pause → a mechanics problem, look at layer 1.
  • Abandoned after a repeat question → context retention.

Retell's framing of tracking disconnection reason is right: the raw abandonment rate tells you that you have a problem, the bucketing tells you which team owns it.

Sentiment trajectory, not sentiment

A snapshot sentiment score is nearly meaningless — people who call support are already annoyed. What matters is the slope. Nurix frames this as sentiment trajectory; CloudTalk as sentiment shift score. Negative-to-positive is genuine de-escalation and is worth celebrating. Neutral-to-negative at a specific turn index is a bug with a line number.

The high-value derived metric: percentage of calls with a negative sentiment slope after a specific event type (a transfer offer, an authentication step, a hold). That is a prioritised fix list generated automatically.

Frustration signals

The voice equivalent of Pendo's “rage prompting”: repeated reformulations, raised volume, profanity, and the universal tell — a caller saying “agent,” “representative,” or “human” repeatedly. That last one deserves its own counter. It is the cleanest signal of automation rejection you will ever get, and it needs no model to detect.

Time to first help

How long before the caller receives actual value rather than greeting, disclosure, and menu. Early friction is where callers bail. If your compliance disclosure runs 14 seconds before the caller can say anything, that is a measurable, negotiable cost.

Repeat contact rate

Percentage of callers who contact again within 72 hours about the same issue. CloudTalk targets under 10% and calls it “the audit trail,” which is exactly right. This single metric is what keeps containment honest, and it belongs on both the CX and the finance dashboard.

Layer 4 — Compliance and Risk

These are pass/fail, not KPIs to optimise. Treat any non-zero value as an incident.

Metric What it measures Target
Disclosure adherence Required statements (recording notice, AI disclosure, mini-Miranda, consent) delivered correctly 100%, 100% auditable
PII handling failures Sensitive data masked in logs, excluded from training, correctly scoped in transcripts Zero
High-risk escalation catch rate Fraud signals, vulnerable callers, safety/self-harm language correctly routed to a human 100% catch
Hallucination in regulated flows Fabricated policy, pricing, eligibility, or legal statements Zero
Recording & retention conformance Storage duration, jurisdiction, deletion-on-request honoured 100%

Cekura reports safety and compliance evaluators flagging more than 20% of calls in regulated verticals. If your compliance dashboard shows zero, the most likely explanation is that you are not measuring, not that you are clean.

One practical note: disclosure adherence is one of the few voice metrics where you can and should audit 100% of calls rather than sampling. It is a deterministic string-and-position check, it is cheap, and “we sampled 5%” is not a defence anyone accepts.

Layer 5 — The Money

Here is where most voice AI reporting goes wrong, so let me be blunt about the central failure.

Containment is not savings. Resolved containment is savings.

Containment, honestly

Raw containment rate is (calls handled end-to-end by AI ÷ total calls) × 100. Published ranges: CloudTalk puts mature deployments at 50–70%, Bluejay puts leaders above 70%.

Raw containment is also the easiest metric in this entire post to game. An agent that simply refuses to transfer contains 100% of calls. So does an agent so unpleasant that callers hang up — those count as contained, because no human touched them. The honest version:

Resolved containment = (AI-handled calls − abandoned calls − repeat contacts within 72h) ÷ total calls

Report both. The gap between raw and resolved containment is the most diagnostic single number on the finance dashboard. In deployments I have seen, a 70% raw containment can resolve to 55% honest containment, and the 15-point gap is precisely the work still to do.

First call resolution

(issues resolved on first contact with verified downstream action ÷ total calls) × 100. Published benchmarks converge: 70–85% for mature deployments, 80%+ as world-class (CloudTalk, Retell, Bluejay all land in this band).

The word doing the work is verified. FCR computed from “the agent did not transfer” is containment wearing a nicer name.

Escalation, split two ways

CloudTalk's split here is the most useful thing in any of these sources:

  • Planned escalation — the flow is designed to hand off (complex quote, legal issue, VIP account). This is a feature. It should not be minimised.
  • Forced escalation — the agent failed. Target under 10%.

Reporting them as one number means every product decision to add a deliberate handoff looks like a regression. Split them on day one.

Then measure transfer success rate — escalations where the human resolved it with full context carried over. CloudTalk targets 85%+. A transfer where the human says “can you tell me what this is about?” has destroyed most of the value of the AI leg and doubled the caller's effort. Also track time to handoff: a forced escalation at turn 2 costs 20 seconds; one at turn 12 costs the caller's patience and your margin.

Handle time — with a caveat

AHT should fall relative to the human baseline; Retell suggests effective agents reduce it 20–30%, and Cekura cites a customer case at 40% lower handle time. But CloudTalk's caveat is the one to internalise: “a 90-second average that resolves the issue beats a 60-second average that ends in a transfer.”

Never optimise AHT directly. Optimise resolved containment and let AHT fall out. AHT as a target produces agents that rush callers off the phone.

Cost per contact vs cost per resolved contact

Cost per contact is total operational cost ÷ contacts handled. Typical figures cited: AI-handled calls under $1, human-handled $5–15 (CloudTalk); Deloitte Digital's figure via Bluejay is a 9–25% cost-per-contact reduction within 3–6 months.

But the metric to actually run the business on is cost per resolved contact:

Cost per resolved contact = (AI platform + telephony + LLM/ASR/TTS usage + integration & tuning labour + human cost of escalations and repeat contacts) ÷ resolved contacts

That denominator change is the whole game. A cheap agent that resolves half its calls has a cost per resolved contact roughly double its cost per contact, plus the human cost it pushed downstream. I have watched a deployment report an 80% cost reduction while the contact centre's headcount did not move — because the failed calls came back as second contacts, and second contacts are handled by humans on longer, angrier calls.

The revenue side — the half everyone skips

Cost reduction is the default frame for contact-centre AI, and for a home-services business it is the less interesting half. When a plumbing company misses 30% of its inbound calls because the office is on a job site, the voice agent is not a cost-reduction tool. It is a revenue-capture tool.

  • Answer rate — inbound calls answered ÷ inbound calls received. Pre-AI baselines of 60–75% are common in SMB service businesses. Going to ~100% is the single largest financial effect of deployment and it never shows up on a cost-per-contact chart.
  • Booking / conversion rate per answered call — the AI agent's close rate on qualified demand. Compare against your human baseline honestly, segmented by intent.
  • Revenue per answered call — (bookings × average job value × show rate) ÷ answered calls. This is the number that makes a CFO care.
  • After-hours capture — revenue from calls outside business hours that previously went to voicemail. Usually pure incremental.
  • Lead qualification accuracy — of leads the AI marked qualified, how many actually were? An agent that books unqualified jobs is generating negative revenue via wasted truck rolls.
  • Recovery rate — for collections use cases, contactability, promise-to-pay rate, and value recovered per call (Rootlenses covers this pattern well).

The ROI equation, written out

Most published ROI formulas are just (gains − costs) ÷ costs, which is true and unhelpful. Here is the version with the terms teams actually forget:

Monthly value = [ resolved_AI_calls × (loaded_human_cost_per_call − AI_cost_per_call) ]
    + [ incremental_answered_calls × booking_rate × avg_job_value × margin ]
    − [ failed_containments × (rework_call_cost + churn_risk_cost) ]
    − [ monthly_tuning_and_ops_labour ]

Three notes on using it:

  1. Loaded human cost means salary plus benefits, supervision, QA, training, attrition, and idle time — not hourly wage. Contact-centre labour typically runs 60–70% of total cost, so this term dominates.
  2. The rework term is not optional. It is what converts a moved cost into an honest one. If you cannot measure it, use repeat contact rate as a proxy.
  3. Tuning labour is real and ongoing. Someone reviews flagged calls, adjusts prompts, adds intents. Budget it or your ROI is fiction. Published payback claims — Bluejay cites vendor figures of 200–300% ROI in 12–18 months and 60–90 day payback targets — are achievable but assume this work is happening.

Treat all vendor-published benchmark numbers in this post, mine included, as calibration ranges rather than promises. Every one of them comes from a company that sells voice AI, measured on deployments that succeeded.

The One-Page Scorecard

If you steal one thing from this post, steal this table. It is what I would put on a wall.

Metric Formula / definition Target Read against
Turn latency (p95) Caller speech end → agent audio out < 800ms Abandonment
Interruption rate Turns where agent cut caller off ÷ turns Trend down Endpointing config
Talk ratio Agent speech time ÷ total speech time < 0.80 CSAT, prompt length
Critical-token error rate Errors on names/numbers/dates ÷ critical tokens As low as achievable Task success
Intent coverage Calls in supported intents ÷ total calls Ceiling on containment Roadmap
Intent accuracy Correct intent ÷ classified intents 90%+ top, 80%+ tail Fallback rate
Task success (verified) Downstream state correct ÷ attempted tasks Highest-trust metric Repeat contact
Hallucination rate Calls with unsupported assertions ÷ calls ~0 in regulated flows Knowledge coverage
Workflow adherence Calls following required flow ÷ calls > 95% Compliance audit
Abandonment Caller-ended before resolution ÷ calls Trend down, bucketed by turn Latency, CSAT
Customer Effort Score Post-call ease rating Beat human baseline Abandonment
Sentiment slope Change start → end, by event Non-negative Turn-level events
Repeat contact (72h) Same-issue recontacts ÷ resolved calls < 10% Containment
Raw containment AI-only calls ÷ total calls 50–70%+ Always with the next row
Resolved containment Contained − abandoned − recontacts The real number Cost model
Forced escalation Failure handoffs ÷ calls < 10% Planned escalation
Transfer success Escalations resolved with context ÷ escalations 85%+ Handoff payload quality
Cost per resolved contact All-in cost ÷ resolved contacts Beat loaded human cost Repeat contact
Answer rate Answered ÷ offered inbound → 100% Revenue per call
Revenue per answered call Bookings × job value × show rate ÷ calls Grow Qualification accuracy
Disclosure adherence Compliant calls ÷ calls 100% Audit sampling = 100%

Instrumenting It Without Building a Data Team

Four things make this tractable:

1. One trace per conversation, spans per stage. Every call gets an ID that survives into your CRM, your ticketing system, and your billing records. Without that join key you can compute layer 1 and 2 forever and never reach layer 5. Getting this right on day one is worth more than any individual metric.

2. Verify outcomes downstream, not in the transcript. After the call ends, check the system of record. Did the appointment exist? Did the payment post? This is a background job, it is not hard to write, and it is the difference between measuring what your agent claimed and what your agent did.

3. Automated grading with a calibrated human sample. LLM-as-judge scales to 100% of calls for subjective metrics. It also drifts. Keep a weekly human-audited sample — 30 to 50 calls is enough — and measure agreement between judge and human. If agreement falls, your dashboard has been lying for however long it has been drifting. I go into calibration in more depth in the observability field guide.

4. Segment everything by intent. A blended dashboard tells you the agent is “85% good.” A segmented one tells you it is 96% on appointment lookups, 91% on rescheduling, and 48% on billing disputes — which is an actionable sentence and a decision about where to route.

The Metrics That Lie

A short list of traps, because most of the damage in voice AI reporting comes from a handful of repeat offenders.

  • Raw containment. Gamed by refusing to transfer, and inflated by callers hanging up. Always with repeat contact.
  • Average latency. Hides the tail that ruins calls. Use p95.
  • Unweighted WER. A 2% error rate concentrated on account numbers is a catastrophe; the same rate on filler words is nothing.
  • CSAT alone. Survivorship bias. The people who hated it left before the survey.
  • AHT as a target. Rewards rushing. Make it an output, not a goal.
  • Blended anything. One score across all intents is the statistical equivalent of an average temperature across all your customers' houses.
  • Cost per contact. Right denominator, wrong one. Resolved contacts is the honest denominator.
  • Month-one savings extrapolated to a year. The tuning cost is front-loaded and the failure cost is back-loaded. Both bite later.

How I Would Sequence This

You do not instrument twenty metrics on day one. Order of operations:

  1. Week 1 — trace ID, p95 latency, abandonment, transfer rate, and a 100% disclosure-adherence check. This is enough to know if you are hurting anyone.
  2. Weeks 2–4 — verified task success and repeat contact rate. Now you can tell resolution from deflection.
  3. Month 2 — intent coverage and per-intent segmentation. This generates your roadmap.
  4. Month 3 — the money layer: resolved containment, cost per resolved contact, revenue per answered call. Only now do you have a defensible ROI number.
  5. Ongoing — sentiment slope, CES, workflow adherence, judge calibration.

Anyone who quotes you an ROI number in week 1 is quoting containment. That is the whole trick.

A Closing Rule

The best voice AI scorecard I have seen fit on one screen and had a strict rule attached: every business metric had to be traceable to a conversation-level metric that engineering could move. If cost per resolved contact went up, you could click through to the intent, then to the flow, then to the turn where callers started leaving.

That traceability is the entire point of measuring five layers instead of one. Containment tells you what happened. Latency, comprehension, and sentiment tell you why. The money layer tells you whether to keep going.

Measure all five, keep the honest denominators, and never let a number reach the board that you cannot follow back down to a single bad call.

Sources & Further Reading

The published benchmark ranges cited above come from these vendor and analyst write-ups. They are the best public compilations available, and every one of them is written by a party with an interest in the answer — read them as calibration, not as ground truth:

And from this blog, the adjacent pieces: Evals, Testing, and Observability for Voice AI Agents for the pre-production side, What Are Evals? for the plain-English introduction, A Dummy's Guide to Evals for AI Agents if you are starting from zero, and Bayesian and Sequential A/B Testing for how to decide whether a metric actually moved.