Coding-Agents

Ankur Sethi found the middle path between doing everything himself and letting coding assistants run free: he asks his LLM to generate code in chat, then manually types every line into his editor. His agent instructions forbid the assistant from creating, editing, or deleting project files — every proposed edit is shown in chat, and he types it in by hand. It’s “grossly inefficient and perhaps slightly comical,” but instead of being 10x faster he’s maybe 2x faster, and in exchange he keeps a real mental model of his codebase. Typing forces him to slow down, which makes him more likely to catch hallucinations and bad design choices, lets him clean up and refactor as he goes, and builds a spatial map of where everything lives — which makes his future prompting better too. He compares it to the old advice never to copy-paste code while learning: type it out, adapt it, understand it. Reviewing AI-generated PRs line by line, he argues, is a miserable way to work — “robots raise PRs, humans review them” — so the alternative isn’t heroic review, it’s never ceding the typing in the first place. His closing warning: the industry is piling up cognitive debt it will soon have to pay back, and shipping software you don’t understand is professional malpractice. ...

Shreya Shankar (UC Berkeley PhD) presents DAB — the Data Agent Benchmark (“Can AI agents answer your data questions?”), from her PhD work at Berkeley. ~27 minutes on Hamel Husain’s channel. Why data agents matter Huge real use case: AI answering business questions — much office work is this Enterprises struggle to use Claude Code / Codex off-the-shelf for BI tasks, so they build their own data agents: Uber’s Query GPT (1.2M queries/month at launch), OpenAI’s in-house agent, Databricks, Snowflake Existing benchmarks fall short: Text-to-SQL — assumes clean data in one Postgres/Snowflake; real enterprises have fragmented data; SQL alone isn’t enough (need domain knowledge, reasoning over outputs) Table QA — context table + question; doesn’t scale to enterprise data volume The formative study — 4 real-world challenges Interviews with enterprises + Hasura (PromptQL) surfaced four properties missing from existing benchmarks: ...

Hamel Husain demos a Codex capability most people don’t know about: Codex can control itself — spawning parallel threads, letting them coordinate, and driving computer use. ~5 minutes on his own channel. The demo setup Working project: AEO — optimizing one of his websites for AI discovery (making it findable by AI search engines) Codex listed 16 high-value tasks from the project file (aeo-to-do.md) — the fuel for the orchestration demo Spawning threads Prompt: “Open a new thread for each task and explain how you’d tackle it, along with prerequisite steps” — Codex spawns 16 parallel threads in the sidebar Codex can also rename and delete threads itself Value: manage separate tasks completely independently, no window-jumping Threads talking to threads Inside any thread you can query another: “What is AEO 1 doing? Does it need any help?” Great for orchestrating when things get stuck, or starting a supervisor thread that manages others and unblocks them Steering and queues Ask for a status table when threads finish: which can run in parallel, which need human intervention or input Broadcast guidance to all threads: “Direct threads that can work independently with computer use to start — don’t start work if you need other threads to finish first” One thread inventories the active threads and coordinates the rest — “this starts to become super powerful” Computer use A thread opens the browser itself: checks Bing Webmaster Tools, Google Search Console, etc. It tells Hamel what it needs (accepting a verification), keeps going, and reports when it gets stuck Mobile Same thread list appears on your phone — manage all parallel threads remotely, even away from the computer “You can have Codex control Codex and become a power user to do a lot of things faster and parallelize your work.” ...

Shreya Shankar (Stanford CS professor, co-creator of the AI evals course with Hamel) kicks off the 12-part AI product engineering series. 27 minutes on Hamel Husain’s channel. Why this matters Output quality is the biggest barrier to productionizing agents (LangSmith annual report) — and figuring out how to evaluate models is genuinely hard Vendors (LangChain, Braintrust, Arize) are selling end-to-end automated eval tools: point an LLM at your traces, it finds and fixes your bugs The catch is epistemic: what “good” means lives in your head, not in the traces — if a tool could fully fix your product, it could fix everyone’s, and there’d be nothing left to differentiate yours AI’s real job: help you express and apply your judgment faster, not replace it The eval lifecycle (analyze → measure → improve) Error analysis — the hardest step: take traces and find failure modes. No perfect definition of “mistake” (you can’t define slop, but you know it when you see it) Measure — how prevalent is each failure mode? Pareto applies: ~80% of issues come from ~20% of failure modes — prioritize those Improve — fix the product: prompt instructions, model switch, fine-tuning. Iterate forever AI is weak at the front (taste-specific error analysis) and strong at the back (measurement, prompt optimization, hill-climbing). ...