YouTube videos

Evals are the bottleneck: the three parts of an evaluation, the hidden variables that skew scores, and when to reach for fine-tuning and RL instead of prompt tweaks.

Dillon Mulroy, Rhys Sullivan, and Sunil Pai debate AI containment, then inspect a coding workflow built around scoped access and visible end-to-end tests.

A full tutorial on Grok Bot's dedicated cloud computer: advisor, YouTube researcher, X Scout, digital Marie Kondo, and travel concierge — plus the trust question.

Riley Brown's full content stack on Codex: skills for research, hooks, thumbnails, and diagrams — plus his test-based, outcome-driven way of improving skills.

A live whiteboard lecture on model cascades: send every record to a cheap proxy model first, accept its answer when confidence is high, route the rest to the expensive model — and pick the threshold from a labeled sample to hit your accuracy target at minimum cost.

A practical LLM knowledge-base pipeline: voice-dictated raw notes, agent skills that enrich and backlink them, generated wikis, nightly cloud automation, and agent-built graph visualizations.

The enterprise adoption cycle, the fear×confidence axes, and a concrete playbook for making coding agents work for a whole team.

Dillon Mulroy, Rhys Sullivan, and Sunil Pai discuss AI burnout, lost collaboration, and the parts of making software that still bring joy.

Linear's Nan Yu and Jacob Shumway on the Linear Agent: the skills architecture, evals that grow from usage, minimal instruction, and agents tied to humans.

Zach Mueller (Lambda) on when open models beat APIs: the current open-model landscape, token-speed targets, quantization trade-offs, and why open weights are good enough for ~90% of tasks for ~90% of people.

A complete Product OS walkthrough: define, design, develop, and distribute a real software product with Claude Code — with four case studies of non-technical founders making real money.

The opening lecture of Stanford CS329A: scaling laws, inference-time scaling, reasoning models, and the self-improving loop that powers agents.

Nous Research's Karan Malhotra on why Hermes outperforms native harnesses, reward hacking and sycophancy, the Hermes Curator, and a Sonic Chao Garden mod built entirely by his agent.

Diogo Almeida argues that pleasing a human and running reliable software are different objectives. His case for moving beyond AI assistance—and why he says Jev's approach is not simply RLVR.

The DAB benchmark for data agents: why text-to-SQL isn't enough, the four real-world data challenges (messy joins, multi-database, free text, domain knowledge), and why frontier agents only pass 57% of the time.

Codex's hidden superpower: threads that spawn threads, talk to each other, and orchestrate themselves. Hamel demos running 16 parallel coding tasks with computer use and mobile control.

A practical framework for picking an OCR model: text-blocks vs document structure, API vs self-host, and why you should test on your own data instead of trusting benchmarks.

Dillon Mulroy shows how Pi, Plannotator, and type-driven plans keep him involved in agent-written software.

The anti-generic Claude Design workflow: prepare three files (design.md → design system → template) so every deck starts on-brand and every feedback compounds into future designs.

Full Claude Design platform walkthrough: expanded templates, bundled usage, design systems for on-brand output, and a motion-graphics-for-videos use case that is genuinely surprising.

Lucas Rocha, an engineer at Brazilian edtech Nova Escola and alum of Hamel Husain's AI Evals course, tells his evals rollout story backwards: from a team that now runs LLM judges with no tickets and no engineer in the loop, back to the messy first spreadsheet and the three barriers — data, methodology, and judgment — that every team hits along the way.

Wayfinder is Matt Pocock's pre-spec orchestration tool: it charts a map of parallel decision/research/prototype tickets, spawns agents, and resolves open questions into a spec — demonstrated live building a TikTok creator feature.

Abi Aryan breaks down where LLM latency actually goes: the compute-bound prefill phase vs the memory-bandwidth-bound decode phase, and why decode — not prefill — dominates latency and cost, especially in multi-step agentic systems. A live Colab walkthrough with TinyLlama on a T4 measures both phases across chat, RAG, creative, and agentic request shapes.

Before worrying about evals, ask if you are building the right thing: AI that dumps an unverifiable answer creates more work. Hamel walks through design patterns — provenance, progressive disclosure, expert signals, modular steps — across three worked examples.

A walkthrough of Codex desktop app remote control: connect your agentic-coding servers, kick off sessions from your phone, and watch agents work anywhere — with live session views, subagent summaries, and follow-up prompts.

Part 1 of Hamel and Shreya's AI product engineering series: the analyze → measure → improve eval lifecycle, and three mistakes to avoid when using AI to automate evals.

Dillon Mulroy, Rhys Sullivan, and Sunil Pai argue for products that serve a customer's chosen agent. Their examples connect APIs, documentation, review loops, cloud environments, and burnout.

Hamel argues that hiring a consultant to implement AI and walk away is almost always a mistake — the underlying problems remain and the AI itself is designed to remove the middleman. Consultants work only for education or when they have skin in the game.

When a product seems impossible to eval, that is usually a sign the product itself is weak. Hamel suggests infusing user-facing signals into the product so users — and you — can assess quality.

Dillon Mulroy, Rhys Sullivan, and Sunil Pai ask why more generated code has not produced better software. They compare architecture, token costs, model access, and autonomous loops.

Hamel Husain counters the 'evals take too long' objection with one principle: if you can't eval your product, your users can't either — so evals are how you figure out how to convey trust in AI answers.

Hamel Husain explains how to scientifically measure an LLM judge's noise against ground-truth human labels — and why every LLM judge is a black-box classifier with edge cases, so you have to decide how much noise is tolerable.

Hamel Husain argues that ROUGE/BLEU string-similarity scores and 1-to-10 rating scales are holdovers from traditional NLP that make evals harder to act on — nobody knows what a 4.2 means, and binary pass/fail evals are what actually let you ship.

Hamel Husain explains why he converts every 1-to-5 rating scale eval into binary pass/fail checks, scoping each failure into a specific, actionable criterion instead of hiding noise behind middle values.

Hamel Husain on evals for non-experts: why generic metric dashboards actively hurt you, how error analysis beats top-down guesswork, what guardrails and MCPs actually are, and the argument that AI competency can never be outsourced.

Paul Ford (former Harper's editor, co-founder of Abort) and Dan Shipper process the week Opus 4.5 landed in Claude Code — the step change, the grief, the discourse, and what actually matters.