Hamel Husain hosts Will and Florian from Prime Intellect — the open-source reinforcement learning training team — on using evals to actually improve models. ~36 minutes.

An evaluation has three parts

  • Task set — your data, prompts, and scoring methods (what everyone focuses on first)
  • Harness — the program that drives the LLM: Claude Code, Codex, or open-source harnesses like Prime / OpenCode
  • Environment — where it runs: Docker, sandboxes, your own infra

“If you are unable to express your task or your problem in any way or capacity, you’re also unable to improve your results.”

The hidden variables that skew scores

  • Inference engine / API: calling a closed model over Bedrock can give different results than the first-party API — gnarly to debug
  • Harness choice: minimal harnesses (small system prompt, a few tools) drive real cost reductions
  • Hardware & timeouts: Terminal-Bench results jumped 14-15 points just from raising timeouts — that’s two model generations of progress for free

Improve evals before you fine-tune

  1. Run several models and look at the rollouts — did it misinterpret the prompt, call tools wrong, go in weird directions?
  2. Fix prompts and tools first, then consider swapping the harness
  3. Fine-tuning is one tool in the belt, not the first one

Sampling parameter pitfalls

  • Temperature 0 is a bad default — models writing eval boilerplate love to pin it; use the model maker’s recommended config, and “temperature 1 is always better than zero” when unknown
  • Truncation — low max-tokens/max-turns cuts off reasoning models mid-thought
  • Not enough time/resources — give evaluations the highest settings initially

RL is just “make the eval go up”

  • Cheaper moves first: prompt optimization (DSPy-style), then SFT distillation from a bigger model
  • Reinforcement learning is the general version — hill climbing on your task distribution, with a train/test split
  • Architecture: inference engine (vLLM, SGLang) → orchestrator → traces → trainer (GRPO/PPO)
  • RL is unforgiving of sloppy evals: reward hacking is what happens when the eval is correlated with your goal but doesn’t measure it (e.g. “Claude says these words” instead of “writes well”)

Reward hacking and how to fight it

  • “The more clever the reward, the more likely there’s a problem” — if you can’t describe a correct answer in one sentence, it’s tricky
  • Use coding agents to find adversarial examples that score high but violate the spirit — then fix the eval
  • Linting-style deterministic checks + yes/no judge questions beat fuzzy “is this good or bad?”
  • Humans stay the arbiter: “You need to look at your data. Agents don’t solve the problem of looking at your data for you.”

“You need to look at your data. I think agents are great. They don’t solve the problem of looking at your data for you.”