How huge models fit across specialized chips

  • Cerebras explains how it spreads huge models across many chips — Its worked example explains why fast answers depend on both chip memory and the number of requests sharing it.
    • The October 22 engineering account uses Kimi K3, with roughly three trillion parameters, as an illustration. This is newly selected technical coverage, rather than a new model release.
    • Each WSE-3T wafer has 44 GB of on-chip memory. The example assumes four-bit weights, an eight-bit KV cache, the model’s stored working memory for a prompt, and 256,000-token average requests.
    • Its simplified sizing assigns about 16 GB of weights per layer. Two linear-attention layers fit per wafer because their request state stays fixed in size as prompts grow.
    • Full-attention layers need more memory as prompts grow, so each gets its own wafer. The example’s 69 linear-attention and 24 full-attention layers require 35 plus 24 wafers, or 59 altogether.
    • Weights stay on their assigned chips while intermediate results travel between them, with a claimed roughly two-microsecond network hop. Different requests occupy different stages concurrently to keep the pipeline busy.
    • Cerebras claims 1,000–2,000-plus output tokens per second for large models. Its account recommends comparing total output per unit of power at matched answer speed, rather than treating concurrency alone as efficiency.
    • Caveat: This vendor illustration supplies neither a reproducible end-to-end comparison nor a public Kimi service commitment, so the speed claim is not a deployment guarantee. (HN 185 points · 62 comments)

Show a coding assistant the interface problem

  • Scrawl bundles screen annotations and spoken explanations for coding assistants — A time-aligned bundle can show exactly which screen change your bug report refers to, including inside an iPhone app.
    • An agent is a model running a multi-step task with tools. Scrawl’s MIT-licensed source packages marked screenshots, a transcript, interaction events and a structured manifest for one to inspect.
    • Its v0.3.13 release, published October 22 UTC, adds macOS 15 support and a custom-command transcriber. Apple transcription stays on-device, while a custom program controls its own data handling.
    • The desktop app supports Apple Silicon Macs on macOS 15 or later and an unsigned Windows preview. The site still lists Linux as forthcoming.
    • ScrawlKit records inside an iOS 18-or-later app, while cabled capture provides another iPhone route. The assistant can request more frames around a narrated moment instead of receiving every frame upfront.
    • Default cleanup moves raw media to Trash after seven days and whole bundles after thirty. Retained screenshots and transcripts remain sensitive, and automatic OpenCode delivery can be disabled.
    • Caveat: On-device transcription does not keep the eventual handoff private, because the receiving coding assistant may send those screenshots and words to its model provider. (lobste.rs discovery · checked release, repository documentation and platform limits)

Custom developer tools need tests that outlast the demo

  • Dan Luu argues coding assistants make custom developer tools practical — His examples suggest testing narrowly useful tools against real workloads before committing to a general-purpose rewrite.
    • Luu’s essay identifies code search, Rust compilation and filesystem operations as bottlenecks once models write code faster. His examples target those specific waits.
    • He reports roughly 50-times faster cold builds and twice-as-fast warm builds from a Rust interpreter experiment. He explicitly describes quick comparisons without rigorous performance testing.
    • He points to pgrust’s source and its author’s upstream bug reports as inspectable evidence of testing. One August 1 PostgreSQL report includes a query sequence exposing history-dependent date parsing.
    • These are older examples supporting a newly circulated argument, rather than newly discovered PostgreSQL defects. The essay also acknowledges pgrust’s premature preview submission to a performance comparison.
    • Its counterargument is operational: a narrowly optimized tool can overfit one workload and become hard to maintain. Luu likewise separates personal-use quality from the work required for a reusable product.
    • Caveat: This is a practitioner’s argument with selected examples and informal timings, so it establishes neither a general productivity multiplier nor production readiness for the linked database. (HN 451 points · 397 comments · lobste.rs)
All gathered items - what was cut and why (8)