LinkedIn’s Ziwei Guo and Saloni Gandhi explain a feedback loop for an agent that infers members’ career preferences. The current production change they describe is human-reviewed prompt optimization, not autonomous rewriting of model weights or the entire agent.

Start with a bounded improvement loop

  • A user-understanding agent assembles fields such as preferred job titles from profile, search and preference data, attaching evidence and confidence.
  • Score outputs on 15 product-defined dimensions; group failures, diagnose causes, propose prompt edits, validate candidates and test on a held-out set before review and deployment.
  • The team calls this recursive self-improvement, but says its active work is prompt updates. Workflow, harness and optimizer-code changes are future rungs, not deployed results in this talk.

Measure the evaluator before trusting its fixes

  • Gandhi reports reviewing about 180–200 human-labeled examples. Her initial judge agreed fully with human reviewers about 55% of the time for score-1 outputs, about 57% for score-2 outputs and about 80% for score-3 outputs.
  • Failure cases included plausible job titles that omitted important intent signals, blurry distinctions between domain/function/skill, inconsistent partial credit and penalties for harmless formatting.
  • Rubric changes reportedly raised overall judge–human alignment from 57% to 64% on the team’s comparison. That measures judge agreement, not a seven-point improvement in the agent’s user-facing quality.
  • A second “reflection” judge could misdiagnose the first judge’s errors as a bad rubric or wrong agent component. They added explicit error categories and human review rather than letting plausible explanations drive edits unchecked.

Ship carefully and define success

  • The speakers report landing an initial optimized prompt change in production after human review and measurement; they do not establish fully autonomous, continuous self-improvement.
  • Keep held-out checks separate from candidate selection, inspect judge disagreement by field and monitor whether changes help actual members, not merely the rubric score.
  • Standardized shared evaluation infrastructure is a proposed next step across teams, not evidence that the loop already operates enterprise-wide.

“Before we could trust a system to improve an agent automatically, we had to understand whether the feedback driving those improvements was also itself trustworthy.” — Saloni Gandhi