Microsoft product manager Keiji Kanazawa recounts rebuilding his personal research assistant after three weeks of coding-agent refactors in this AI Engineer talk.

Check what the architecture actually does

  • A misattributed digest sent Kanazawa looking for a trace; the supposed agent still ran as a deterministic workflow.
  • “Skills,” “actions,” and “agents” had plausible names but collapsed into overlapping implementation concepts.
  • The expected skill files and agent-runtime behavior were missing, despite plans he had approved.

Look beyond successful execution

  • Kanazawa says Codex connected unit tests to production inference and ran roughly 30 cycles, triggering a spending alert.
  • A scheduled daily reflection job continued running while producing empty reports.
  • He started a fresh implementation with clearer specifications, an explicit agent registry, and outcome rubrics.

Make the output criteria concrete

  • His video-analysis feature combines transcripts with selected frames, including equations and slides.
  • One lecture yielded about 440 keyframes; the resulting canvas paired visual material with explanations and links to relevant moments.
  • He revised the rubric when the output grouped equations separately from a wall of explanatory text.
  • Coding agents optimized against the rubric while he inspected the artifact and refined what the rubric should reward.

Review the rubric too

  • One generated rubric became tailored to a single lecture and performed poorly on five other videos.
  • Kanazawa reports better results after correcting that overfitting, but provides no complete benchmark or final held-out scores.
  • Claude and Codex also review one another’s work over multiple rounds; those reviews provide feedback rather than a correctness guarantee.

Delegate implementation while owning the goal

  • His framework comparison favored tools that fit his product-manager workflow, not a universal winner on coding quality.
  • The experience left him committed to delegation, with more attention to goals and evaluation.
  • The title’s code-reading incident is a diagnostic turning point; his conclusion is not to review every generated line manually.

“I own the goal.” — Keiji Kanazawa

Caveat: This is a personal project report and creator-presented demo, not a Microsoft production result or a controlled comparison of coding agents.