Baseten co-founder and CEO Tuhin Srivastava joins Mario Gabriele on The Generalist for a 44-minute interview about inference, the work of running trained models.
The product that used to be free became the company
- Baseten initially built tools around small models, data scientists and internal applications.
- Srivastava says a fine-tuned Stable Diffusion customer’s traffic exposed the difficulty of serving larger models on GPUs.
- The team discarded roughly three years of surrounding product work in about six weeks to focus on serving.
Why a specialist can compete with big clouds
- Inference joins compute, networking, observability and workflow, often spread across separate cloud teams.
- Customers need new models in hours or days; negotiating long-term cloud capacity can take months, Srivastava says.
- He expects specialist inference platforms and GPU-focused clouds to move into each other’s territory.
Four ways customers run models
- Start with closed-model APIs and basic provider failover.
- Add open-weight models through shared APIs, accepting shared-capacity trade-offs.
- Use dedicated deployments tuned for response speed or total throughput.
- Post-train custom models using production inputs, outputs and feedback about quality.
The feedback loop matters more than a single model
- Train routing logic to send tasks to suitable closed, open or customized models.
- Collect production feedback, improve models, redeploy and repeat; Srivastava describes this as a compounding company advantage.
- He calls proprietary data, workflows and customer feedback the ingredients of “owned intelligence.”
Capacity and reliability are part of the product
- Srivastava reports operating across 20 clouds and 90 regions, presented to customers as one virtual cloud.
- Regional routing supports failover, data-residency requirements and serving closer to users.
- Hardware failures, high utilization and fast demand growth complicate day-to-day operations.
- Long commitments, advance payments and emergency capacity make financing and forecasting engineering constraints too.
Customer service sits on the critical path
- Srivastava draws on restaurant hospitality: show up during incidents and build relationships outside contract renewals.
- Daily customer contact replaces formal quarterly reviews in his account.
- He credits timing and luck heavily while treating execution skill as indispensable.
- Caveat: company scale and competitive claims come from the interview; the “final market” thesis and trillion-dollar scenarios are speculative.
“We are critical path.” — Tuhin Srivastava, opening montage