Daniel Lee’s djev-run is an inspectable route to a self-hosted, Jev-style decision service. It serves NVIDIA’s DiffusionGemma-Jev with vLLM, adds a /v1/systemone compatibility layer, and includes deployable Cloud Run configuration rather than stopping at a notebook demo.

What is actually in the repository

  • A pinned vLLM container, startup script, Cloud Run deployment script, middleware implementation, Colab notebook, and three small game demos.
  • The container copies roughly 18 GB of model weights from Google Cloud Storage into /dev/shm, then starts vLLM with bidirectional Triton attention and diffusion-specific settings.
  • The recommended Cloud Run shape is one RTX PRO 6000, 20 vCPUs, 80 GiB RAM, concurrency 32, and scale-to-zero.
  • I inspected the README and implementation files. I did not deploy or run the model because the documented configuration requires that GPU and weight setup; this is not a security audit.

How the compatibility layer works

  • The middleware accepts TypeSafe-shaped choice, score, and yes/no questions plus an application state.
  • It converts each allowed answer into a fixed label, builds a constrained response template, and finds the token positions where labels differ.
  • Non-answer positions are pinned while DiffusionGemma fills the answer slots in a single diffusion read.
  • Log probabilities at those slots are normalized across the allowed labels and returned as choices, scores, or yes/no probabilities.
  • Typed output keeps responses inside the supplied schema. It does not establish that the selected choice or probability is calibrated for a production workload.

The useful engineering details

  • Tokenized label templates are cached by the pair of baseline strings, avoiding repeated /tokenize calls for the same schema.
  • The server preserves ordinary vLLM endpoints while adding /v1/systemone, so callers can use either the low-level completion interface or the compatibility wrapper.
  • The Docker image pins its vLLM base image by digest, while the documented raw-container alternative uses a nightly image and is less reproducible over time.
  • Scale-to-zero lowers idle compute cost but introduces a claimed cold start of about 47.5 seconds. That trade-off matters for an interactive decision service.

Performance claims, not reproduced here

  • The repository reports 61 ms median server latency and 117 ms end-to-end latency for one-step, one-sample inference.
  • It reports roughly 100–123 requests per second at concurrency 32.
  • On the public JevBench v1.3.0 suite (N=231), the README reports 81.4% accuracy for one sample and 81.8% with automatic sampling.
  • It lists active Cloud Run GPU cost at $3.19 per hour and says the service scales to zero.
  • Those figures are the maintainer’s measurements. I inspected their implementation and configuration but did not reproduce the benchmark, cloud bill, latency, throughput, or accuracy.

Caveats before production use

  • The API wrapper derives answer probabilities from top log probabilities at selected token slots; that is an implementation choice, not a demonstrated calibration guarantee.
  • It fills missing labels with a low probability floor when they fall outside the returned top-logprob set, which can affect normalized confidence.
  • The service allows all CORS origins in the supplied launch configuration and exposes no authentication layer of its own; production deployment needs perimeter controls.
  • The public repository has no declared license, so “open source” reuse rights are unclear even though the code is visible.
  • Cloud Run’s scale-to-zero economics fit bursty use, but a large GPU instance is not a lightweight local alternative.

The project is valuable because it makes the mechanism inspectable: Jev-style typed decisions can be built from fixed answer slots and probability reads on an open-weight diffusion model. The next useful evidence would be an independently reproducible benchmark on representative decision tasks, including calibration—not only valid typed output.