Jesse Hall of LiveKit explains voice-agent engineering for financial services in this twenty-one-minute AI Engineer talk.

Measure what the caller hears

  • A voice stack combines speech recognition, a language model and speech synthesis; speech-to-speech models offer another architecture.
  • Test the components together: mishearing “the 3rd” as “the 30th” can corrupt an otherwise fluent payment conversation.
  • Measure time to first audible speech and time to a speakable sentence, alongside provider first-byte and first-token metrics.
  • Hall describes roughly 1.5 seconds as an upper conversational budget, with 500–600 milliseconds feeling more natural.

Keep tools and conversation moving

  • Let slow back-end work run asynchronously while the agent acknowledges the wait or collects independent details.
  • In the hotel-booking demo, the agent continues asking questions while saving previous answers.
  • Handle failed writes explicitly and provide a human handoff when recovery fails.
  • Use audio rhythm and intonation alongside words to decide whether someone has finished speaking.

Benchmark the complete interaction

  1. Define the caller’s goal and the authorized back-end outcome.
  2. Simulate interruptions, changed minds, mumbling and background noise.
  3. Use text tests for business logic and audio tests for the speech pipeline and turn-taking.
  4. Check database state, unnecessary steps, invented tool calls and privacy safeguards.
  5. Hold the scenarios constant while swapping models and providers; preserve failures as regression tests.

Keep the claims in scope

  • Hall reports premature interruptions of 4.5% versus 9.9% for an alternative in LiveKit’s own tests.
  • Confirm dates and money amounts, verify identity before account access and keep listening during slow operations.
  • Caveat: the timing targets and comparison come from a LiveKit presentation. The selected hotel demo cannot establish reliability for financial transactions.

“Leaderboards, they score these models, these components. But we’re shipping a conversation.” — Jesse Hall, 5:19–5:25