Doneyli De Jesus, who leads AI engineering solutions at Langfuse by ClickHouse, describes work with an unnamed top-five bank in this 19-minute AI Engineer talk.

Test the deployed workflow on representative data

  • The bank wanted evidence that a specific application could serve its users, including the firm’s data, context and compliance requirements.
  • A golden dataset contains inputs, expected outputs and metadata; evaluation runs produce traces showing what happened.
  • Keep the model, prompt, tools and other configuration tied to those traces so score changes can be explained.
  • The described deployment needed on-premises operation, controlled access and an isolated environment.

Fix the weak component, then measure again

  • De Jesus describes a model with 97% groundedness but only 60% numerical accuracy on the bank’s evaluation.
  • Adding a deterministic Python calculator moved the reported numerical result toward 95%, clearing an 85% gate for that use case.
  • A fluent answer can cite the right documents while still calculating the wrong number.
  • Changes to prompts, tools or models trigger another evaluation before promotion.

Give each failure mode its own gate

  • Use code-based checks for deterministic properties, trace checks for required steps, and AI judges for qualitative aspects.
  • Domain experts choose the scoring dimensions and thresholds for each application.
  • The examples include datasets of roughly 150 and 1,500 questions, rather than a universal test-set size.
  • A failed compliance check can block a release even when grounding, completeness and tool-use scores look healthy.

Feed reviewed failures back into the tests

  • Route failures into annotation queues where domain experts can inspect the trace and explain the problem.
  • Reviewed examples become candidates for the golden dataset, strengthening later regression tests.
  • De Jesus reports shortening evaluation from weeks to minutes, without publishing a timing study or the bank’s underlying results.

“Use your data, use your bar, use your gates.” — Doneyli De Jesus, 18:46

Caveat: This is a vendor-presented, anonymous customer case. Its numerical results and approval thresholds describe that application, not bank-wide safety or regulatory certification.