Michelle Chen, Cloudflare’s group product manager for its AI platform, joins Dillon Mulroy and Sunil Pai on The Next Token. Rhys Sullivan is away preparing Executor for YC Demo Day. Recorded August 24, 2026. Cloudflare sponsors the show, and all three speakers work there.

Workers AI runs models; AI Gateway connects them

  • 01:34: Michelle oversees Workers AI, AI Gateway, and AI Search. Her “arbiter of tokens and GPUs” description covers model selection, capacity, efficiency, and internal demand.
  • 03:55: She recalls an AI planning session becoming a product launch within weeks. Infrastructure staff carried GPUs in suitcases to install them in data centers.
  • 07:18: Workers AI operates the GPUs, model scheduling, and serving software. AI Gateway routes requests to providers and adds logging, caching, observability, and security.
  • The products are converging around shared interfaces. Michelle says dashboard and routing work remains, so this is a direction rather than a claim that every interface is unified.

Launching a model is easier than making its economics work

10:53: Michelle sketches the initial path: download weights from Hugging Face, store them in R2, and run a recipe from an inference engine such as SGLang or vLLM.

  • Serving flags control batching, memory use, and hardware utilization. Getting an answer is the first step; tuning the deployment is separate work.
  • Market prices then set a constraint. The team checks expected demand, throughput, chip costs, and whether serving at those prices can make money.
  • Michelle says Cloudflare initially declined to release a DeepSeek deployment because the prevailing price failed that test. She reports releasing it after prices rose fivefold. These are her historical account and pricing claims, not a current price comparison.
  • 15:00: Downloadable weights do not guarantee permission to operate a commercial service. Modified licenses and revenue thresholds can require negotiation.

Nearby GPUs and large-model clusters solve different problems

17:38: Cloudflare initially expected smaller, faster models to favor its global network. Michelle describes two hardware tiers after model demand grew in a different direction.

  • A latency tier puts smaller models near users. Voice sessions and fast vision requests benefit from shorter network trips.
  • A throughput tier supports large models across interconnected GPUs. It requires bandwidth, power, and cooling that ordinary edge facilities may lack.
  • 31:27: NVIDIA’s software ecosystem remains a major advantage, though Michelle points to AMD and heterogeneous serving as alternatives worth watching.
  • Procurement takes much longer than model releases. A chip that still works can become commercially unattractive when competitors serve the same workload more efficiently on newer hardware.

Open models challenge the old capability gap

20:06–29:37: Michelle describes strong competition among Chinese labs and researchers returning home for family, language, and well-paid work. She explicitly lacks inside knowledge of alleged distillation practices.

  • Her view is that the familiar “six to twelve months behind” shorthand no longer fits publicly available open models. Sunil backs that with his experience using GLM and Kimi.
  • They acknowledge that private lab capabilities are unknown. This is practitioner judgment, not a controlled comparison across every task.
  • Faster releases create a provider problem: users remain attached to old models that continue occupying GPUs.
  • Sunil jokes about distributing weights on USB drives and running community inference if access becomes restricted. Possessing the weights still leaves a large compute bill.

Capacity, throughput, and the subscription illusion

35:20: Michelle uses a past Kimi deployment to make the scale tangible: roughly 16 GPUs for one instance. At one earlier point, it handled about 0.5 requests per second. She immediately says the throughput has since improved; neither number is a universal sizing rule.

  • 37:27: Customers want huge context, fast output, low initial delay, full precision, serverless access, and deep discounts together. Michelle says the hardware costs force tradeoffs.
  • She sees many subscriptions as loss leaders. Automated token consumption is easy to sustain around the clock, unlike the low usage a provider might hope for.
  • A rival’s deep discount can reflect subsidy or an attempt to fill already committed GPUs. Its price alone does not reveal sustainable cost.
  • 40:42: Michelle predicts tighter limits and more usage billing. She also describes free-tier and subscription abuse feeding discounted token resale. Her forecast is not an announcement of another company’s plans.

Why operate inference, and what should builders make?

44:48: Michelle sees inference as part of Cloudflare’s full application platform. Agents need models alongside compute and storage. An existing supplier relationship can also save enterprise buyers another procurement process.

She and Sunil recall rapidly fixing tool-calling and provider bugs earlier in the year. The excitement was real, as was the temptation to stay inside building indefinitely.

50:02: Sunil asks what all that machinery produces. His software-factory joke ends with a factory that makes only better factories.

  • Build for family, friends, meetups, or everyday chores.
  • Treat privacy, trust, and ordinary people’s discomfort as product constraints.
  • His own counterexample is a digital mural using depth cameras. It expands what he considers possible, rather than merely raising pull-request volume.

Serving compute has hard costs. The closing challenge is to spend it on something people actually want.