Zach Mueller (head of DevRel at Lambda, ex-Hugging Face) on how to think about open models — which ones, where they run, what they cost, and how to serve them. ~41 minutes on Hamel Husain’s channel.

The big claim

  • Open models are good enough for ~90% of queries from ~90% of people — the 10% exception is people pushing the frontier (OpenAI, Google, Anthropic, frontier research labs)
  • Compared to a year ago, massive improvement in intelligence per parameter; small models now generalize, not just hyper-specialize

The current open-model landscape

  • Qwen non-MoE (27B, e.g. Qwen3.6 27B) — “the most capable model for an everyday agent”; 4-bit ≈ 14GB → fits a 16GB card; hundreds of people run Hermes agent purely on it
  • Qwen MoE (~30B) — better than bigger Qwen dense models
  • GLM-5.2 (~800B) — his daily driver for coding; needs 6×96GB cards (~$60k) or a big Mac
  • DeepSeek V4 Flash — capable smaller model, a solid Haiku replacement
  • Kimi (3T) — great for writing/review but 8×B200 ≈ 1.5TB VRAM
  • MiniMax — strong but enterprise license changed since 2.6
  • Nemotron — US-built, US data; easier to get past enterprise “scary Chinese model” objections

Speed targets (tokens/sec)

  • 20 tok/s — baseline usable: background tasks, CPU offloading of a giant model
  • 50 tok/s — usable workflow, roughly Opus-level; GLM 5.2 self-optimized its own deployment from 20→47 tok/s in a day (200–300M tokens through it in the first week and a half)
  • 100 tok/s — ideal for sub-agents
  • 750 tok/s (GPT-5.6 + Cerebras) — too fast to even review what the model writes; the joke is that becomes the chain of thought, presented at 50 tok/s

Quantization

  • FP8 native — most expensive; needed when RL rollouts must match training precision
  • NVFP4 / MXFP4 — near-lossless; NVIDIA ships its own NVFP4 weights via quantize-aware training; Kimi K2 (1T) goes from 8–16 B200s in FP8 to fitting on 4 in NVFP4
  • His setup: GLM 5.2 NVFP4 on half a B200 cluster, Kimi K2.7 code on the other half

Cost and security thinking

  • Don’t point company data at OpenRouter — it routes to providers anywhere (including outside the country) and routers can carry “upload your code” instructions; even American providers (Grok) did it. If you can run it yourself, that’s the lowest-risk variable
  • Buying hardware is usually a flawed equation — GPUs run 40–60% utilization, so “pays for itself at 100%” math doesn’t hold; try 2–3 month spot instances first, track token usage, then decide
  • Self-hosting skills advance your career (his path: Accelerate at Hugging Face → GPUs at home)

Serving

  • vLLM — out-of-the-box experience, broad model support, an “oracle” that auto-picks kernels; home/single-node default
  • SGLang — disaggregated inference, cache-aware routing for 100–1000 users; multi-node territory
  • llama.cpp — fine for single-user local; he ignores it for batched/multi-user work

Model routing and harnesses

  • KB cache + shared context are everything — cold caches rebuild from scratch on every provider switch, so he won’t use “magic routers” unless he picks the models, the tasks, and stays on one provider
  • Numena Ncode (XDR’s harness): a Claude Code fork with a model stack — Soul (driver) + GLM 5.2 (implementer/overseer) + DeepSeek V4 Flash (writes the code); XDR fine-tuned Kimi K2.6 for it and swaps models to measure how much code each frontier model removes
  • Pi — minimal open-source harness; he replaced 3–4 Claude Code workflows with GLM 5.2 + Pi
  • Amp — opinionated routing baked in (GLM 5.2 workhorse + a “Soul Oracle” for hard problems), API-priced; “the Puck” = an agent over agents
  • Evaluating open LLMs: “vibes” plus trusted people (e.g. XDR); Lambda publishes the LLM index — deploy recipes (Docker) + tokens/sec benchmarks

Codex side-note

  • Codex’s computer use operates any app on your Mac; mobile support shows all running sessions/threads on your phone; and Codex can control Codex — fan out a project into 12–15 threads, let them talk to each other, orchestrate

“Open weight models are good enough for about 90% of queries from 90% of people.”

Watch on YouTube — full summary in the vault note.