An ICML 2026 spotlight from Princeton/UChicago (Wu, Liu, Bai & Griffiths) argues the dominant bias framing — remove the existing stereotypes models absorbed from training data — is incomplete. Put an LLM in a stateful decision-making loop and it will invent new social biases about groups that have no real differences, purely from the exploration-exploitation dynamics of learning from noisy outcomes.

The setup is a hiring game from the psychology literature: 40 rounds, four fictional demographic groups, all equally likely to succeed. LLMs over-fit early random feedback, lock groups into job categories, and end up more stratified than human participants:

  • Human baseline SI = .84; all frontier LLMs average 1.39; reasoning models worst (OpenAI o3 SI = 1.83)
  • Newer and larger models stratify more in every model family — Claude 4 Sonnet’s SI was >8× Claude 3 Haiku’s — inverting single-turn bias benchmarks like BBQ, where newer models score better
  • The biases are learned per-run from experience, not read off pretraining (no priors toward the fictional groups)
  • Same pattern in refugee-resettlement and conscription settings; gamified rewards not required
  • Standard fixes fail: CoT prompting, higher temperature, and compressed context leave stratification intact

The mechanism is insufficient exploration — the same “hot stove” trap documented in humans, amplified by better optimization. And the fix points at the problem: only changing the model’s objective — adding an explicit diversity/reward term — dropped nearly all models below both the human and random-assignment baselines.

The authors are careful about the caveat: that steer assumes equal base rates, and applied where groups genuinely differ it reduces overall success — classic reward misspecification. Their conclusion is a research program, not a solution: we need intrinsic, value-level guidance, not prescriptive prompts — “in social domains where truth is often indeterminate, it is often desirable to thoroughly explore candidate options before exploiting a seemingly optimal outcome.”

The unsettling part for anyone building agentic systems: capability and fairness diverge exactly where agent deployment begins. Stateful agents that retain beliefs across interactions inherit this amplifier — better evals on single-turn bias benchmarks say nothing about what they’ll do with a persistent memory and a reward to optimize.