TypeSafe’s Jack & Jill case study describes replacing Gemini 3.1 Flash Lite in the initial scoring stage of a talent marketplace. Unlike a generic launch claim, it explains the comparison set and what “quality” actually means.

I read the complete case study; I did not reproduce its evaluation or inspect private candidate data. Every performance and savings figure here is reported by the vendor, not independently verified.

Preserve the shortlist, not the entire hiring decision

  • The pipeline retrieves 6,000 candidates per open role, shortlists 2,000, then uses a stronger model for closer review.
  • The offline comparison covered 150 live roles, with the same candidates for each model.
  • It included all historical “winners”—people a hiring manager later requested to meet—and a weighted sample of the remaining candidates.
  • Jev reportedly retained 94.6% of those winners, versus 93.9% for Gemini.

That is shortlist recall: how many previously selected people survive the filter. The reported difference was not statistically significant; the vendor’s 90% confidence interval was −0.4 to +1.8 percentage points. It is not evidence that Jev makes better hires.

Concrete savings, narrow scope

TypeSafe reports token cost per 1,000 screened candidates falling from $0.755 to $0.092, rounded to an 88% reduction. It also reports median time to screen all candidates for a role falling from 20.3 to 10.3 seconds.

The page’s overview labels that timing “each candidate,” but its detailed speed table specifies the whole role-level batch. The latter is the usable definition—not a per-candidate latency benchmark.

Annual savings of $265,000 at current volume and $500,000 with expected growth are projections. These are token-cost comparisons for one stage, not audited whole-product savings including infrastructure, integration or human review.

The labels deserve scrutiny

Hiring-manager meeting requests are useful operational labels, but candidates with no recorded request are not necessarily unsuitable. Historical selection can encode uneven exposure and bias; the study does not report subgroup fairness or downstream hiring outcomes.

Typed output is not a correct decision. This is evidence for testing a cheaper first-pass ranker while preserving close review—not justification for autonomous rejection of applicants or treating historical preference as ground truth.