Kevin Liu’s Relay is a Slack-like workspace for studying computer-use agents. Its Jev adapter illustrates a useful design boundary: the harness builds a menu of permitted actions, Jev picks one, and the application—not the model—executes and grades it.

The actual decision loop

  • runner/typesafe.mjs accepts accessibility or visible-page JSON, not screenshots. It builds actions from enabled controls, plus fills using quoted text or search tokens from the task, keys, scrolling, waiting, and finishing.
  • The adapter sends that menu as a TypeSafe Choice question. It rejects invented choices, incomplete or invalid probability distributions, missing usage data, and menus above 255 options rather than silently truncating or substituting another model.
  • Requests, candidate labels, selected action, returned probabilities, model identity, and outcomes become inspectable receipts. The task grader checks final workspace state, not a model’s claim that it finished.

Where the comparison breaks

  • Jev selects a provided action; a generative agent composes one. Comparing them changes the action policy as well as the model, so it is not a clean model-only contest.
  • The text-only menu cannot compose arbitrary new answers from retrieved facts. Relay explicitly excludes Jev from its free-composition and multi-step workflows, and from pixel/API conditions.
  • Relay’s Jev contract tests use a fake TypeSafe transport with real browser interactions. Its documentation says no TypeSafe key was supplied for that implementation test: live Jev task accuracy, billed latency, and API compatibility were not demonstrated there. I inspected the README, Jev design note, verification record, and adapter source; I did not run Relay or call Jev.

The useful artifact is a reproducible boundary for decision-model agents: disclose the observation and menu, validate the receipt, keep the action gateway constrained, and grade the resulting state. A typed answer or a probability bar is not evidence that the action was correct.