Open-Weights

Zach Mueller (head of DevRel at Lambda, ex-Hugging Face) on how to think about open models — which ones, where they run, what they cost, and how to serve them. ~41 minutes on Hamel Husain’s channel. The big claim Open models are good enough for ~90% of queries from ~90% of people — the 10% exception is people pushing the frontier (OpenAI, Google, Anthropic, frontier research labs) Compared to a year ago, massive improvement in intelligence per parameter; small models now generalize, not just hyper-specialize The current open-model landscape Qwen non-MoE (27B, e.g. Qwen3.6 27B) — “the most capable model for an everyday agent”; 4-bit ≈ 14GB → fits a 16GB card; hundreds of people run Hermes agent purely on it Qwen MoE (~30B) — better than bigger Qwen dense models GLM-5.2 (~800B) — his daily driver for coding; needs 6×96GB cards (~$60k) or a big Mac DeepSeek V4 Flash — capable smaller model, a solid Haiku replacement Kimi (3T) — great for writing/review but 8×B200 ≈ 1.5TB VRAM MiniMax — strong but enterprise license changed since 2.6 Nemotron — US-built, US data; easier to get past enterprise “scary Chinese model” objections Speed targets (tokens/sec) 20 tok/s — baseline usable: background tasks, CPU offloading of a giant model 50 tok/s — usable workflow, roughly Opus-level; GLM 5.2 self-optimized its own deployment from 20→47 tok/s in a day (200–300M tokens through it in the first week and a half) 100 tok/s — ideal for sub-agents 750 tok/s (GPT-5.6 + Cerebras) — too fast to even review what the model writes; the joke is that becomes the chain of thought, presented at 50 tok/s Quantization FP8 native — most expensive; needed when RL rollouts must match training precision NVFP4 / MXFP4 — near-lossless; NVIDIA ships its own NVFP4 weights via quantize-aware training; Kimi K2 (1T) goes from 8–16 B200s in FP8 to fitting on 4 in NVFP4 His setup: GLM 5.2 NVFP4 on half a B200 cluster, Kimi K2.7 code on the other half Cost and security thinking Don’t point company data at OpenRouter — it routes to providers anywhere (including outside the country) and routers can carry “upload your code” instructions; even American providers (Grok) did it. If you can run it yourself, that’s the lowest-risk variable Buying hardware is usually a flawed equation — GPUs run 40–60% utilization, so “pays for itself at 100%” math doesn’t hold; try 2–3 month spot instances first, track token usage, then decide Self-hosting skills advance your career (his path: Accelerate at Hugging Face → GPUs at home) Serving vLLM — out-of-the-box experience, broad model support, an “oracle” that auto-picks kernels; home/single-node default SGLang — disaggregated inference, cache-aware routing for 100–1000 users; multi-node territory llama.cpp — fine for single-user local; he ignores it for batched/multi-user work Model routing and harnesses KB cache + shared context are everything — cold caches rebuild from scratch on every provider switch, so he won’t use “magic routers” unless he picks the models, the tasks, and stays on one provider Numena Ncode (XDR’s harness): a Claude Code fork with a model stack — Soul (driver) + GLM 5.2 (implementer/overseer) + DeepSeek V4 Flash (writes the code); XDR fine-tuned Kimi K2.6 for it and swaps models to measure how much code each frontier model removes Pi — minimal open-source harness; he replaced 3–4 Claude Code workflows with GLM 5.2 + Pi Amp — opinionated routing baked in (GLM 5.2 workhorse + a “Soul Oracle” for hard problems), API-priced; “the Puck” = an agent over agents Evaluating open LLMs: “vibes” plus trusted people (e.g. XDR); Lambda publishes the LLM index — deploy recipes (Docker) + tokens/sec benchmarks Codex side-note Codex’s computer use operates any app on your Mac; mobile support shows all running sessions/threads on your phone; and Codex can control Codex — fan out a project into 12–15 threads, let them talk to each other, orchestrate “Open weight models are good enough for about 90% of queries from 90% of people.” ...

Shieldstral is Mistral’s 3B open-weights answer to the guardrail-model problem: instead of baking a fixed taxonomy of harm categories into the weights — which forces retraining every time a product, audience, or moderation policy changes — you hand it the policy as a plain-language question at inference time (“Does this content promote violence against a protected group? Is this image safe to show to a minor?”), and it returns a calibrated yes/no safety score from a single forward pass, covering text, images, and text+image pairs through one interface. The framing does real work: it unifies prompt classification, response moderation, refusal detection, and toxicity detection into a binary question-answering task, with policies living entirely in the prompt so one checkpoint adapts to novel policies at deployment without retraining. It’s small enough to run on a single 16GB GPU, yet Mistral claims it matches or beats open guard models up to 7x its size on text safety and sets a new state of the art on multimodal moderation — helped by training on deliberately similar, easily-confused policy pairs (teaching discrimination rather than memorization), LoRA fine-tunes merged via SLERP, and image–query pairs filtered through a vision-language reranker. Released under Apache 2.0 as an inaugural member of the Open Secure AI Alliance alongside NVIDIA; weights on HuggingFace, technical report on arXiv. ...

MiniMax H3 dropped today with open weights, and ComfyUI has native support on day zero. It’s MiniMax’s third-generation video model (after Hailuo 01 and 02) and the first released open-weights: feed it text, images, video, or audio and it generates video with real stereo sound — up to 2K, up to 15 seconds per clip. Modes include text-to-video, image-to-video, first-and-last-frame control, and reference-to-video, where a reference clip can carry a subject, a motion, or even a voice through the shot. Audio is generated in the same pass, not bolted on afterward. ...