Mo Moustafa runs Olly, an AI assistant that lives in iMessage, on open-source models through OpenRouter — over 18 million messages to date, roughly a third of them on open models through the router. That is enough volume, as he puts it, to hit every edge case at least once. His post is the list of things he wishes he had known going in, and it is the most concrete public account of what a routing layer actually costs you.

He starts with the vocabulary that matters. The model is the weights. The provider is whoever OpenRouter routes you to: they host the model on their GPUs, at their chosen precision, with their own tool-call parsers and therefore their own list of bugs. Ask for deepseek/deepseek-v4-flash and you get one of roughly twenty companies you have mostly never heard of — “the same model on paper, but very different models in real life.”

The findings, in his order:

  • Identical weights, wildly different scores. On OpenRouter’s own per-provider board for DeepSeek V4 Flash 0731 (rolling 32-day average, 2026-09-07), first-party DeepSeek scores 90.2% on GPQA Diamond and 81.3% on TAU-Bench Airline. DigitalOcean, same weights, scores 75.3% and 58.4%. Most hosts cluster five to seven points below first-party on tool calling; four fall off a cliff on knowledge. In July it was worse — Fireworks scored 46% on TAU, a thirty-point gap. For an agent, the tool-calling number is the one that matters.
  • A vision model can have blind providers. He pushed three tiny images — a letter, a solid colour, a word on a background — through every host of two open vision models. DeepInfra’s Qwen endpoint read a “K” as an “R”, called red “blue” and described the word “umbrella” as “funny”, while four other hosts of the same weights got all three right. Venice and Together returned “no image provided” for every MiniMax image and still answered 200. He is careful to separate the two cases: MiniMax’s colour misses happened on every host including first-party, “so that one is the model, not the provider.”
  • reasoning.effort is accepted everywhere and ignored by some. He pinned every provider, sent the same prompt at low, high and max, three calls each, from a production machine, and measured reasoning tokens. Most respect the setting. DigitalOcean, GMI Cloud, Mancer and Venice do not.
  • Quantization filters do not buy quality. The intuition says fewer bits, dumber model. In practice the fp4 hosts sit in the middle of the fp8 pack, the three worst GPQA scores are one fp4 host, one fp8 host and one that declares nothing, and GLM 5.3 Flash’s best scorer on both benchmarks declares no precision at all. The filter also shrinks the pool OpenRouter can fall back to. His rule: “filter on the board, not the bits.”
  • The tool call leaks into the text. When a provider’s parser misses, the reply can contain raw markup like <use_skills><parameters>{"skills":["search"]}</parameters></use_skills>. How often depends on the provider, and it is common enough that you end up writing your own parser — with the awkward detail that wrapped tool calls and wrapped responses need opposite handling.
  • 200 OK, no answer. Reasoning models sometimes place everything in the reasoning field and return content: null with finish_reason: "stop" — his example is 345 completion tokens, HTTP 200, and nothing to show the user. A 200 means the request was served, not that an answer came back. No content and no tool call is a failure, so throw and retry.
  • Hollow completions are the sibling failure: null content, null reasoning, and no usage object at all. In July that was StreamLake on DeepSeek — about 20% of his traffic and 92% of his empty completions. A month later Together did the same thing on a different checkpoint.
  • History rules are per provider, not per model. DeepSeek in thinking mode emits a reasoning_content block, and in an agent loop the model often tool-calls with empty reasoning. Pass that history to SiliconFlow and it 400s with code 20015 — “the reasoning_content in the thinking mode must be passed back to the API” — while Baidu, Alibaba and Cloudflare accept the identical history without complaint.
  • Test from production, not your laptop. Venice and Novita worked perfectly for him from a Mac and 429’d nearly every probe from his infrastructure, same key, same minute. His read is IP-based rate limiting. Benchmark where the traffic will actually run.
  • Pinning does not save you. He once pinned three providers with fallbacks disabled — Cloudflare, Baidu, Alibaba — and two weeks later Baidu was rate-limiting everything, Cloudflare had stopped serving the model, and 100% of traffic had collapsed onto Alibaba, which then started 429ing. The most popular model on the platform, pinned to three of its most reliable providers, was down, “and so was Olly.”

One thing worth noting in the post’s favour, since it would be easy to read this as a takedown: his central evidence comes from OpenRouter’s own per-provider benchmark board, and his measurements are dated so they can be re-checked. That transparency is what makes the critique possible, and it is also why the useful lesson generalizes past this one router — “same weights, different product” is a property of routed inference generally, not a defect unique to one vendor.

The distilled version for anyone building on routed models: measure the provider serving your specific workload, on the benchmark closest to what you actually do, from the machine that will do it, more times than feels necessary. And treat a 200 as proof of delivery, not of an answer.