IVR and voice AI solve different routing problems. A short keypad menu is predictable, accessible without speech recognition, and appropriate when callers need only a few stable destinations. A voice agent becomes useful when callers express varied intents, need approved answers, or must complete a bounded transaction. Replacing every IVR with a model adds cost and failure modes without necessarily improving the call.
Choose the simplest architecture that satisfies the call
Keep IVR for deterministic choices such as language, emergency instructions, or department routing. Use a voice agent when natural-language intent materially reduces menu depth or when the call requires a sequence such as identifying a customer, checking availability, and confirming a booking. A hybrid flow often works best: carrier and emergency controls remain deterministic, the agent handles the approved conversational branch, and a keypad or human fallback is always available.
Budget latency by segment and measure it in production
Do not publish one universal latency promise. Instrument caller-end timestamp, end-of-turn detection, model response start, first audio byte, and tool round-trip separately. The acceptable budget depends on network conditions, turn-detection settings, reasoning depth, speech generation, and whether a tool must run. Stream audio when the platform supports it, keep prompts and tool descriptions focused, cache stable context, and test from the actual telephony path rather than a laptop microphone.
A latency target is useful only when each segment is measured and the system has a graceful response when a dependency exceeds it.
Treat turn detection and interruption as product behavior
Voice activity detection decides when the caller has finished and when the model may respond. Aggressive thresholds interrupt thoughtful callers; slow thresholds create dead air. Test noise, speakerphone echo, short acknowledgements, long pauses, and mid-response corrections. When the caller barges in, stop playback, preserve the unheard portion as not delivered, and let the new utterance update the active state rather than continuing an obsolete answer.
Put transactions behind narrow, idempotent tools
The model should not invent availability or write directly to unrestricted CRM endpoints. Expose separate tools for identity lookup, availability search, provisional hold, booking confirmation, contact update, and message delivery. Validate required fields and authorization before each call. Give mutating requests an idempotency key tied to the conversation and proposed slot so a retry cannot create a second appointment.
Confirm consequential details before execution
Repeat names, telephone numbers, locations, time zones, prices, and appointment times in a compact confirmation step. If speech recognition remains uncertain, switch to keypad entry, send a secure link, or transfer to a person. Never let conversational fluency conceal missing evidence; the CRM record should distinguish caller-provided facts, retrieved facts, model interpretation, and the final tool result.
Design transfers and degraded modes before launch
Define triggers for explicit human requests, repeated misunderstanding, complaints, emergencies, policy exceptions, tool failure, and low-confidence identity. Pass a structured summary and captured fields to the receiving queue. If no person answers, return to the caller with an honest callback option and create an owned task. If the model service fails, preserve deterministic emergency and voicemail routes instead of dropping the call.
Evaluate outcomes, not just transcript sentiment
Create a test set covering accents, background noise, interruptions, ambiguous intent, no availability, stale knowledge, duplicate booking attempts, tool timeouts, and hostile instructions. Review task completion, booking accuracy, unauthorized-action rate, transfer success, abandonment by call stage, and correction rate. Compare the voice flow with the IVR or human baseline for the same call types before expanding traffic.