Jeffrey Ip’s DeepEval integration writeup makes a useful distinction: replacing a language model’s verdict step is not the same thing as replacing the entire evaluation algorithm. Its two Jev modes do different jobs—and their scores should not be treated as interchangeable.

This capture reviews the full integration article and its examples, not a live execution or an implementation audit. DeepEval says comparative accuracy, variance, cost and speed benchmarks are still forthcoming.

Hybrid: keep extraction, replace the verdict

  • A language model still extracts factual claims from the answer and truths from the supplied evidence.
  • Jev receives that evidence and a yes/no probability question for each claim, batched into one request.
  • DeepEval maps probabilities to verdicts and retains the claim-counting score equation.
  • The article specifies Faithfulness cutoffs above 0.65 for yes, below 0.35 for no, and the middle for borderline; these are library policy, not proven calibration.

The opt-in example is FaithfulnessMetric(eval_mode="hybrid"). DeepEval says missing setup fails at metric creation, while a failed Jev call during a hybrid run falls back to the language model for that decision. That fallback matters when interpreting cost and repeatability.

System One: a different measurement

  • FaithfulnessMetric(eval_mode="system_one") sends the raw answer and retrieval context directly to Jev.
  • Instead of counting extracted claims, it combines judgments about support, contradiction and overall grounding into a weighted mean.
  • The reason is assembled from returned answers rather than generated prose; reported confidence reflects the least decisive answer.
  • G-Eval remains language-model-based; DAG workflows still need language models for their text-producing nodes.

A System One Faithfulness score therefore measures a different construction from the hybrid claim fraction. A stable-looking number does not show that it agrees with human labels—or that a missed claim stopped mattering.

The useful boundary—and the missing evidence

DeepEval reports roughly 100 ms for most Jev queries and $0.042 per million input tokens, with output free. Those source claims were not reproduced here, and they do not include hybrid extraction, fallback calls or the whole evaluation run.

Custom JevEval metrics expose the questions and weights directly, making application policy inspectable. But mathematical aggregation only explains how a score was computed. Typed output is not a correct decision; fixed thresholds are not an accuracy benchmark. Validate against independently labeled cases before changing a release gate.