Baseten co-founder and CEO Tuhin Srivastava joins Mario Gabriele on The Generalist for a 44-minute interview about inference, the work of running trained models.

The product that used to be free became the company

  • Baseten initially built tools around small models, data scientists and internal applications.
  • Srivastava says a fine-tuned Stable Diffusion customer’s traffic exposed the difficulty of serving larger models on GPUs.
  • The team discarded roughly three years of surrounding product work in about six weeks to focus on serving.

Why a specialist can compete with big clouds

  • Inference joins compute, networking, observability and workflow, often spread across separate cloud teams.
  • Customers need new models in hours or days; negotiating long-term cloud capacity can take months, Srivastava says.
  • He expects specialist inference platforms and GPU-focused clouds to move into each other’s territory.

Four ways customers run models

  1. Start with closed-model APIs and basic provider failover.
  2. Add open-weight models through shared APIs, accepting shared-capacity trade-offs.
  3. Use dedicated deployments tuned for response speed or total throughput.
  4. Post-train custom models using production inputs, outputs and feedback about quality.

The feedback loop matters more than a single model

  • Train routing logic to send tasks to suitable closed, open or customized models.
  • Collect production feedback, improve models, redeploy and repeat; Srivastava describes this as a compounding company advantage.
  • He calls proprietary data, workflows and customer feedback the ingredients of “owned intelligence.”

Capacity and reliability are part of the product

  • Srivastava reports operating across 20 clouds and 90 regions, presented to customers as one virtual cloud.
  • Regional routing supports failover, data-residency requirements and serving closer to users.
  • Hardware failures, high utilization and fast demand growth complicate day-to-day operations.
  • Long commitments, advance payments and emergency capacity make financing and forecasting engineering constraints too.

Customer service sits on the critical path

  • Srivastava draws on restaurant hospitality: show up during incidents and build relationships outside contract renewals.
  • Daily customer contact replaces formal quarterly reviews in his account.
  • He credits timing and luck heavily while treating execution skill as indispensable.
  • Caveat: company scale and competitive claims come from the interview; the “final market” thesis and trillion-dollar scenarios are speculative.

“We are critical path.” — Tuhin Srivastava, opening montage