Vercel’s Yury Selivanov tests Jev in two Python experiments: deciding whether partially typed input is Python or English, then trying to make a fixed-choice model write code one AST node at a time. The first looks useful with visible edge cases. The second produces valid-looking but mostly incorrect programs—and is more instructive because Selivanov says so plainly.

The Python API is deliberately small

  • Vercel’s experimental Python AI SDK exposes evaluate() with state plus typed questions.
  • ChoiceQuestion selects among fixed options, ScoreQuestion scores on a defined scale, and NoulQuestion estimates whether a statement is true.
  • The response is structured probability output. That removes parsing ambiguity; it does not establish that the choice is correct.

A plausible fit: live input classification

  • Selivanov wanted an agentic REPL to switch between Python and natural language without an explicit mode toggle.
  • His hand-trained classifier was unstable while a line was still being typed, repeatedly flipping labels on partial input.
  • Jev appeared steadier in the traces he published, but still called "what's" + " up English even though it is a partial Python expression.
  • No labeled test set, aggregate accuracy, latency distribution, or reproducible comparison is reported, so this is a product probe—not a benchmark.

A bad fit: generating Python through choices

  • Character-by-character generation barely produced an if statement, so the experiment moved up one level to Python abstract syntax trees.
  • GPT-5.6 first expanded the prompt into a detailed plan; Jev then selected each statement, expression, identifier character, and tree branch.
  • Host code—not Jev—provided punctuation, indentation, AST validity, retry limits, and rendering through ast.unparse.
  • The published Fibonacci trace misspelled fibonacci as fibonacio; Selivanov describes the outputs as syntactically valid but “still mostly incorrect.”
  • The source displays generated code rather than executing it. A later generative-model review is an assessment, not a test result.

The useful boundary

  • Jev fits bounded judgments where software can enumerate the legal actions and independently validate outcomes.
  • Turning every token or AST slot into a classification does not make generation disappear; it moves generation into a long sequence of dependent decisions.
  • More structure can guarantee valid syntax while errors accumulate in names, algorithms, stopping decisions, and semantics.
  • Vercel’s examples were inspected through the article and linked source. I did not run the paid Jev calls or reproduce either experiment.

“Jev is built for making narrow decisions. Whether it’s accurate enough for your use cases is something you’ll need to test.”

The strongest lesson is not that Jev can or cannot write Python. It is that typed output works best when the action space is genuinely bounded, the question is locally meaningful, and correctness is measured outside the model.