Sam Witteveen turns Jev’s typed decisions into a working model-router demonstration: a local endpoint chooses between small local models, cloud models, and image generation. He then replaces the hosted decision service with Semif, an open alternative running locally, to address the privacy problem in sending a sensitive prompt to a cloud classifier.

The router’s three main questions

  • What kind of task is this? A choice question separates chat, simple questions, rewriting or summarization, code, reasoning, and image requests.
  • How much model capability does it need? A score question places the request on a difficulty scale, with descriptions ranging from tasks a small local model can handle to work requiring a frontier model.
  • Does it contain sensitive information? A yes/no-style question checks for personal, confidential, client, financial, medical, or credential data. A high privacy result overrides the normal route and keeps the answering model local.
  • The demo also includes a decision about whether web access is needed. These judgments feed normal conditional logic rather than a free-form routing explanation.
  • Witteveen says the questions can be evaluated together against the same state. Low confidence should trigger a safe fallback instead of forcing a confident-looking guess.

The local-versus-cloud setup

  • A small local MiniCPM model handles lightweight text requests. More demanding or code-related work can go to DeepSeek through OpenRouter.
  • Local Qwen image generation handles image requests. Witteveen also describes optional stronger-model and web-enabled routes.
  • The interface supports both manually selecting a model and letting the router choose automatically. Manual selection makes it easier to compare behavior and test a particular backend.
  • Backend choices and routing thresholds are configurable; the essential pattern is a decision layer in front of several specialized destinations, not dependence on one fixed model lineup.

Following actual routing decisions

  • A simple greeting goes to the small local model, with low difficulty and no detected private information. Witteveen reports a Jev decision time of roughly 331 milliseconds for that example.
  • A Python deduplication request goes to the cloud code route, reflecting the policy that code tasks use the stronger backend even when a particular request is not especially difficult.
  • The inspection view shows the state, questions, category probabilities, difficulty judgment, and privacy and web signals. This makes it possible to distinguish the classifier’s output from the application rule that selected the final destination.
  • A fake API-key example and examples with personal details activate the privacy route. Witteveen shows a privacy value around 0.96 in one example; that is a demonstrated result, not a measured guarantee of detecting all secrets.

The privacy trap in a cloud privacy checker

  • Witteveen calls out the contradiction directly: if the prompt is sent to hosted Jev to ask whether it is private, it has already left the machine.
  • In the initial setup, the actual trust policy is therefore trust the Jev service with the prompt, but keep sensitive requests away from the downstream cloud answering model.
  • To make the routing check local too, he later swaps the judge to Semif. Keeping the classifier local removes that particular outbound disclosure; it does not prove that a probabilistic detector will recognize every sensitive request.
  • The distinction is central to the design: “answered locally” and “never sent to a cloud service” are not interchangeable claims.

Application architecture and policy rules

  • The browser interface is a Next.js app connected to a local FastAPI server. The server requests decisions, applies routing rules, calls the selected backend, and streams the response back.
  • Model connections use OpenAI-compatible endpoints where applicable. The demonstration includes a locally served text model and cloud access through OpenRouter.
  • The router has fallbacks, health checks, and SQLite logging. Witteveen describes storing decisions, messages, and images so the behavior can be inspected.
  • Routes have an order of preference, and ordinary code handles overrides: privacy, difficulty, web needs, and long prompts can all change the destination.
  • One example privacy rule uses a threshold above 0.5 to select a local model. This is an application setting shown in the video, not a universally safe threshold.

Image generation is a composed workflow

  • An image request selects the local image-generation tool rather than asking the decision model to create the image.
  • Before generation, the small text model rewrites the user’s simple prompt into a fuller description. The image model then produces the result shown in the UI.
  • A tool registry keeps this operation separate from the choice of decision backend. Jev or Semif selects what should happen; other components generate text and images.
  • This is the broader pattern of the build: classification supplies control signals, while execution remains the responsibility of application code and specialist models.

Swapping Jev for local Semif

  • Witteveen switches the judge to Semif and repeats a privacy example. He notes that the input and output formats differ slightly but expose the same kinds of signals needed by his router.
  • He reports an 88-millisecond local decision for the displayed run. That is a single demonstration, not a controlled latency or accuracy comparison with hosted Jev.
  • A web-search example selects the configured web-enabled route, showing that a local judge can still decide to use remote capabilities when the policy permits them.
  • Witteveen expects the original Jev to generalize better on harder tasks, but the video does not present an evaluation establishing that comparison.

What the demo statistics do—and do not—show

  • The dashboard snapshot reports 53 requests, with 75% answered locally. Witteveen presents that local share as a way to reduce spending on cloud answering models.
  • He distinguishes spending on the decision service from spending on the downstream model. Overall savings depend on both, as well as the request mix and local compute.
  • For longer conversations, he suggests retaining a strong model when appropriate rather than rerouting every turn, especially where prompt caching would favor staying with the same backend. He describes this as an extension to configure, not a feature he fully demonstrates.
  • The description says code will be uploaded later; this post does not claim that a complete repository for the demonstrated router has already been released.

Scope of this capture

  • This outline uses the entire transcript and the YouTube description, which identifies the local alternative as Semif. It omits channel promotion and avoids treating uncertain auto-caption model-version spellings as verified identifiers.
  • The timings, routing outcomes, cost discussion, and local-response share are Witteveen’s reported demo results. No router was installed or tested, and no independent privacy, calibration, or performance evaluation was performed for this post.