GrowthBook’s Graham McNicoll describes an experiment-quality feature that asks Jev narrow questions and combines the answers in code. The checks assess how an experiment is framed, including whether its goal metrics measure the predicted effect.

Make the score explain its parts

The hypothesis check asks whether the text:

  • Names a concrete change, such as a new design or checkout flow.
  • Predicts a measurable effect with a direction.
  • Explains why the change should cause that effect.

GrowthBook averages the answers locally. Its Quality tab can then identify the missing part and rank fixes by the points they would add.

For example, “Redesigning the homepage will increase signups” supplies a change and an effect, but leaves out the causal explanation. Feedback can point directly to that omission.

Put feedback where the experiment gets edited

  • GrowthBook plans to rerun checks about a minute after hypothesis or goal-metric edits, including on drafts, running experiments and stopped experiments.
  • The team pins its Jev version to prevent a model release silently changing judgments.
  • Custom, open-ended checks may still use a chat model; Jev also helps decide which checks fit the decision model.

This separates reading comprehension from the scoring policy. Typed answers keep the interface predictable; a correct assessment still depends on the question and the evidence supplied.

Read the timing result separately from accuracy

GrowthBook reports hand-labeling 30 hypotheses and pairing some good hypotheses with mismatched metrics. It compared Jev with gpt-5.4-mini using its existing chat-model prompt.

  • Reported timing: about 0.1 seconds per experiment for Jev versus 1.3 seconds for the chat model.
  • Estimated model charge: roughly $0.25 for 10,000 passes, assuming 600 input tokens per pass at the article’s quoted TypeSafe rate.
  • Missing result: the article publishes no agreement rate, confusion table or accuracy comparison against the human labels.

The design supports specific, editable feedback. The published evidence leaves open how often that feedback is right, and a well-written hypothesis can still describe a poor experiment.

Caveat: timing and cost figures are GrowthBook’s unreproduced reports and estimates. The Enterprise feature is described as coming soon.