DSPy’s evaluation guide starts with a scoring function and a baseline. An optimizer can improve the behavior that function rewards. A poor scoring rule can therefore produce a better score and a worse product.

The pieces to understand

  • dspy.Example(...).with_inputs(...) declares which fields the program receives. A reference answer belongs to the metric, not the prediction’s inputs.
  • dspy.Evaluate runs a program across examples and applies your metric.
  • Deterministic rules can check properties such as an allowed decision or an exact label.
  • Human labels and model judges can assess meaning. A model judge still needs checks against human judgments.

The guide’s haiku example first checks whether the output repeats an input season. That makes the optimization loop easy to see. A poem can satisfy that rule and still be poor.

Try this: a digest classifier

Label a small collection of article titles and excerpts as keep, skip, or needs-review. Establish a simple label-accuracy baseline. Inspect errors by category, including useful articles rejected and irrelevant articles retained.

Keep training, validation, and final test examples separate. Avoid placing near-duplicate stories in different splits. Use validation for choices and the final test set only after those choices are fixed.

Checkpoint: explain why the selected metric represents the task. Record actual failures alongside the score, plus model, version, latency, and cost.

Caveat: this is a proposed exercise. No classifier accuracy or improvement is claimed here.

Next: GEPA and useful feedback. Return to the learning path.