Ian Langworth’s jev-nethack is a useful stress test for decision models because NetHack is not a single classification. It is a long, partially observed sequence where one bad action can invalidate hours of progress. The program reads the terminal, constructs legal concrete actions, asks Jev to choose, and turns the selected label into keystrokes; no generative LLM plays the game at runtime.

I inspected the README, API client, main bot, tests, and the project’s unusually detailed development journal. I did not build NetHack, supply a paid API key, or reproduce the reported games.

  • Host code parses the terminal, remembers levels, paths around obstacles, identifies inventory and monsters, and creates actions such as attack, retreat, eat, pray, explore, use stairs, or manipulate an item.
  • Jev receives a choice question over those actions plus separate danger and safest-action questions.
  • When danger crosses the project’s threshold and Jev’s safest choice differs, the bot uses the safer selection.
  • A single legal option skips the paid call. Every multi-option call records state, criteria, probabilities, confidence, model, latency, screen, and the action’s observed outcome.
  • Returned choices are validated against the offered labels; an unknown option cannot reach the motor layer.

The experiment’s real result is the surrounding code

  • The maintainer reports 570 games, a deepest level of 11, twelve runs reaching level 10, a best score of 18,708, and no ascension as of October 2.
  • Those figures come from the repository’s own run ledger and README; I did not independently replay the logs.
  • The bot has nevertheless solved the first Sokoban level and sustained a longest reported run of 34,423 turns.
  • A 333 KB bot.py, a 403 KB journal, game-specific parsers, pathfinding, motors, strategy rules, and regression tests surround the small Jev call.

Failure reports are more valuable than the headline

  • Jev repeatedly chose “eat,” but a missing word boundary made the motor interpret “dwarvish spear” as containing “pear.” The typed decision was reasonable; the deterministic action builder was wrong.
  • Repeated prayer eventually angered the character’s god because prayer cooldown state was lost across restarts.
  • A trap-door refusal, blocked pet, locked shop door, stale map memory, invisible monsters, and a misspelled Elbereth each produced different loops or deaths.
  • The project added frozen-turn guards, budget limits, persistent state, action blacklists, source-derived game rules, and tests as each failure appeared.

What the run does and does not prove

  • Jev is genuinely called through TypeSafe’s /v1/systemone endpoint, and the selected ID maps to a concrete action. This is not a repository that merely mentions Jev.
  • The experiment shows that typed choice can control a complicated environment when software enumerates and validates the available moves.
  • It does not show that typed choices make the whole agent simple. Most competence comes from state extraction, legal-action construction, memory, pathfinding, safety rules, and post-action feedback.
  • The reported scores do not isolate Jev from continual engineering changes, and there is no fixed-seed comparison against random choice, a heuristic policy, an LLM, or an open decision model.

“Jev was right and my option builder was wrong.”

That sentence is the project’s strongest lesson. Typed output prevents malformed actions; it does not guarantee that the host supplied the right action set, represented the world correctly, or attached the right motor to the winning label.