Hamel Husain on why 'it's hard to eval' is a product smell, verification-first design (the data agent, workers' comp report, and PE lesson planner redesigns), sharing skill shapes not skills, transition failure matrices, and why data science is more valuable than ever in the agentic era.
Radu, search engineer at Vespa, on why leaderboard picks leave cost, latency, and quality on the table — quantization, Matryoshka dims, tuning with Vespa Embed, and cheap-embedder + re-ranker architectures.
Cloudflare AI product lead Michelle Chen explains model launches, GPU economics, licensing, and why cheap inference subscriptions may tighten.
Daniel Bloom, a PM at Melio, walks through his self-improving Claude Cowork system: a Notion board his agent builds and manages for him, proactive context acquisition ('what does this term mean?'), a weekly prep and daily brief, a self-improvement loop that learns from his edits — and the Workstation plugin that onboards the whole company in 15 minutes.
Vectara's Ofer Mendelevitch (Hands-On RAG for Production) on building RAG systems that actually work at scale — retrieval, multimodal data, knowledge graphs, evals without golden answers — and why the skill that survives the AI shift is directing agents, not writing code.
Superwall's Joseph Choi interviews Welltory CPO Asya Polony on the onboarding and paywall framework behind an $18M ARR Apple Watch app: six pitch ingredients, syncing health data as the first CTA (90%+ do it), showing users their own data as the strongest proof, and why fear beats benefits at the paywall.
Replit's head of engineering on non-coders building real businesses with AI, the cost-of-code-zero test for what SaaS survives, and why app-layer companies are thriving.
Tara Seshan — OpenAI's product lead for Codex and ChatGPT Work — maps the third era of AI products (after chat and coding agents): persistent coworkers you steer, why ambition is the new differentiator, and why writing-as-thinking is the craft that survives.
A 7-hour build-in-public stream where Janet A. Carr vibes an AI auto-clipper into existence on Render Workflows — upload a VOD, transcribe it with Whisper, score it with an LLM, cut clips with parallel FFmpeg jobs — and actually deploys it and renders real clips before sign-off.
Weekly roundup of the 10 hottest GitHub repos: 2,282 security-scanned Claude plugins, open-source OpenAI Codex, Apache's incubating agent workspace, an open Logitech driver replacement, and more.
Peter Yang's head-to-head across design, writing, planning, coding, browser use, voice, image/video gen, and personal agents — with live demos and a clear overall winner.
GitHub Trending Weekly #46: 35 trending open-source projects — latticedb, Gradient, buzz, walgit, rome, CarWatch, selfdb, factory, and more. One-line takes on each, grouped by theme.
The hard part of background agents is the background: dev boxes over sandboxes, sessions over jobs, and keeping the agent in the control plane away from the risky work.
Shreya Shankar and Hamel Husain walk through the practical evals loop: top-down vs bottom-up criteria, an open-source Claude Code "error discovery" workflow that turns your taste into a rubric, and why automated eval tools get you a baseline but never replace reading your own data.
Why single-vector embeddings fail agentic retrieval, how late-interaction multi-vector scoring preserves low-level detail, and how sparse encoding makes it fast enough for production.
The Next New Thing's weekly Top 10 GitHub roundup: a flow-chart design system for Claude, OpenViking's agent memory, Omari v4, an on-device model, NVIDIA's model router, open-source Mojo, and more.
Evals are the bottleneck: the three parts of an evaluation, the hidden variables that skew scores, and when to reach for fine-tuning and RL instead of prompt tweaks.
Dillon Mulroy, Rhys Sullivan, and Sunil Pai debate AI containment, then inspect a coding workflow built around scoped access and visible end-to-end tests.
A full tutorial on Grok Bot's dedicated cloud computer: advisor, YouTube researcher, X Scout, digital Marie Kondo, and travel concierge — plus the trust question.
Unrequested merges, opaque model services, subscription economics, and why the terminal keeps surviving.
Riley Brown's full content stack on Codex: skills for research, hooks, thumbnails, and diagrams — plus his test-based, outcome-driven way of improving skills.
A live whiteboard lecture on model cascades: send every record to a cheap proxy model first, accept its answer when confidence is high, route the rest to the expensive model — and pick the threshold from a labeled sample to hit your accuracy target at minimum cost.
A practical LLM knowledge-base pipeline: voice-dictated raw notes, agent skills that enrich and backlink them, generated wikis, nightly cloud automation, and agent-built graph visualizations.
The enterprise adoption cycle, the fear×confidence axes, and a concrete playbook for making coding agents work for a whole team.
Dillon Mulroy, Rhys Sullivan, and Sunil Pai discuss AI burnout, lost collaboration, and the parts of making software that still bring joy.
Linear's Nan Yu and Jacob Shumway on the Linear Agent: the skills architecture, evals that grow from usage, minimal instruction, and agents tied to humans.
Zach Mueller (Lambda) on when open models beat APIs: the current open-model landscape, token-speed targets, quantization trade-offs, and why open weights are good enough for ~90% of tasks for ~90% of people.
A complete Product OS walkthrough: define, design, develop, and distribute a real software product with Claude Code — with four case studies of non-technical founders making real money.
Nous Research's Karan Malhotra on why Hermes outperforms native harnesses, reward hacking and sycophancy, the Hermes Curator, and a Sonic Chao Garden mod built entirely by his agent.
Diogo Almeida argues that pleasing a human and running reliable software are different objectives. His case for moving beyond AI assistance—and why he says Jev's approach is not simply RLVR.
The DAB benchmark for data agents: why text-to-SQL isn't enough, the four real-world data challenges (messy joins, multi-database, free text, domain knowledge), and why frontier agents only pass 57% of the time.
Codex's hidden superpower: threads that spawn threads, talk to each other, and orchestrate themselves. Hamel demos running 16 parallel coding tasks with computer use and mobile control.
A practical framework for picking an OCR model: text-blocks vs document structure, API vs self-host, and why you should test on your own data instead of trusting benchmarks.
Dillon Mulroy shows how Pi, Plannotator, and type-driven plans keep him involved in agent-written software.
The anti-generic Claude Design workflow: prepare three files (design.md → design system → template) so every deck starts on-brand and every feedback compounds into future designs.
Full Claude Design platform walkthrough: expanded templates, bundled usage, design systems for on-brand output, and a motion-graphics-for-videos use case that is genuinely surprising.
Maddy Zhang explains why basic document retrieval fails and when hybrid search, reranking, query rewriting and repeated searches can improve a RAG pipeline.
Lucas Rocha, an engineer at Brazilian edtech Nova Escola and alum of Hamel Husain's AI Evals course, tells his evals rollout story backwards: from a team that now runs LLM judges with no tickets and no engineer in the loop, back to the messy first spreadsheet and the three barriers — data, methodology, and judgment — that every team hits along the way.
Wayfinder is Matt Pocock's pre-spec orchestration tool: it charts a map of parallel decision/research/prototype tickets, spawns agents, and resolves open questions into a spec — demonstrated live building a TikTok creator feature.
Pi's creator joins a discussion of broken tool calls, model regressions, measurable agent loops, and the cost of renting intelligence.
Abi Aryan breaks down where LLM latency actually goes: the compute-bound prefill phase vs the memory-bandwidth-bound decode phase, and why decode — not prefill — dominates latency and cost, especially in multi-step agentic systems. A live Colab walkthrough with TinyLlama on a T4 measures both phases across chat, RAG, creative, and agentic request shapes.
Before worrying about evals, ask if you are building the right thing: AI that dumps an unverifiable answer creates more work. Hamel walks through design patterns — provenance, progressive disclosure, expert signals, modular steps — across three worked examples.
A walkthrough of Codex desktop app remote control: connect your agentic-coding servers, kick off sessions from your phone, and watch agents work anywhere — with live session views, subagent summaries, and follow-up prompts.
Part 1 of Hamel and Shreya's AI product engineering series: the analyze → measure → improve eval lifecycle, and three mistakes to avoid when using AI to automate evals.
Dillon Mulroy, Rhys Sullivan, and Sunil Pai argue for products that serve a customer's chosen agent. Their examples connect APIs, documentation, review loops, cloud environments, and burnout.
Hamel argues that hiring a consultant to implement AI and walk away is almost always a mistake — the underlying problems remain and the AI itself is designed to remove the middleman. Consultants work only for education or when they have skin in the game.
When a product seems impossible to eval, that is usually a sign the product itself is weak. Hamel suggests infusing user-facing signals into the product so users — and you — can assess quality.
Dillon Mulroy, Rhys Sullivan, and Sunil Pai ask why more generated code has not produced better software. They compare architecture, token costs, model access, and autonomous loops.
Hamel Husain counters the 'evals take too long' objection with one principle: if you can't eval your product, your users can't either — so evals are how you figure out how to convey trust in AI answers.
Hamel Husain explains how to scientifically measure an LLM judge's noise against ground-truth human labels — and why every LLM judge is a black-box classifier with edge cases, so you have to decide how much noise is tolerable.