Jonathan Osborne, lead AI engineer at NatWest Group, explains the bank’s evaluation and rollout architecture in this 18-minute AI Engineer talk.

Limit what reaches a model

  • Osborne describes Cora handling about 100,000 conversations daily across a bank with roughly 19 million retail customers.
  • A contextual filter removes events such as button clicks, followed by deterministic routing rules that use the message and customer metadata.
  • Remaining messages pass through gated relevance/rephrasing, an embedding-based intent classifier, and policy checks before reaching a backend.
  • Decisions carry audit codes; versioned configuration, fallback answers and access controls support operational changes.

Measure safety, task completion and experience separately

  • The agent can invent facts, follow malicious instructions, give accurate but circular answers, or behave differently on repeated inputs.
  • Osborne groups quality into three independently monitored dimensions: customer safety, task success and conversational experience.
  • Human validation covered 111 conversations, 37 payment journeys and four brands, with three independent reviewers per conversation.
  • The overall 83% pass rate hid 97% on happy paths, 76% on edge cases and 43% on topic changes.
  • Reviewer agreement also varied: 95% for bias/fairness, versus 64% for intent understanding.

Calibrate the judge against individual cases

  • A separate, live complaints-summary tool supplied around 14,000 cases scored by people and an AI judge.
  • Its faithfulness judge averaged about 0.97, versus 0.83 for people, with almost no correlation between their case-level scores.
  • Compare distributions and individual traces; a generous judge can miss the failures hidden by a reassuring average.
  • Treat the judge model and its thresholds as versioned configuration; changing the model can change release decisions.
  • Osborne reports better alignment after moving the judge to a different model family from the agent.

Make evaluation part of production

  • Capture traces using OpenTelemetry and OpenInference, then score a 5% random sample alongside failed handoffs, long conversations and angry-user cases.
  • Score offline every four hours and block regressions in the continuous integration pipeline when prompts or context change.
  • Osborne describes private model access, controls against personal data leaving the network, and Comet/Opik for generative-AI observability.
  • Monitor the conventional intent classifier separately for input drift and F1, the balance of precision and recall.

Observe safely, then release in small steps

  • Shadow mode runs the new classifier on real inputs while the existing scripted system still answers customers.
  • Roll out brand by brand and by traffic percentage, with explicit metrics and automated or manual approval gates.
  • Global and backend-specific feature flags allow fallback to the scripted system within seconds.
  • In the complaints tool, requesting another investigation requires feedback first, turning a routine workflow into labeled evaluation data.

“Continuous evaluation is part of your production infrastructure.” — Jonathan Osborne, 12:36

Caveat: Cora’s new customer-facing rollout was still ahead at the time of the talk. The production judge evidence comes from the separate internal complaints tool.