Frat Quintero’s jev-typesafe-spike is a sprawling research notebook rather than a packaged library. Its value is methodological: dozens of small probes, cached raw responses, revised interpretations, and a long critical guide ask what Jev is actually judging when software presents a bounded question.

I inspected the repository, probe code, cached responses, and guide. I did not issue fresh TypeSafe API calls, so every model result below is the author’s recorded result unless explicitly described as a local check of the saved artifact.

Jev as a judge, not an answer machine

The guide’s central model is precise: Jev receives an expediente—the case material—and one or more narrowly written judgments. It returns typed support for those judgments; ordinary code owns composition, policy, thresholds, and action.

  • Noul weighs one judgment in instructions.
  • Choice distributes support among unordered candidate judgments in criteria.
  • Score distributes support among ordered levels and reports their weighted position.
  • The output’s type says the response has the expected shape. It does not prove that the designer asked the intended question or included the right possible answers.

This framing avoids a common category error: reading 0.86 as “86% true” when it is support for one particular sentence against one particular case file and model version.

The question determines which facts matter

One of the strongest sections varies the question while holding a scene fixed. A man enters a hot house with closed windows and says, “It’s cold, isn’t it?”

  • Asked about sentence form, Jev concentrates support on interrogative.
  • Asked what the speaker literally does, it favors asking whether it is cold.
  • Asked what the sentence is relative to the situation, with irony available as an answer, it favors irony.

The repository’s conclusion is not that one answer is universally correct. The question defines which part of the case can count as evidence. The answer menu then defines where support is allowed to go.

A generic “other” option is not enough protection. In the recorded scene, when irony was omitted, a plausible literal option won and “other” received little support. Typed choice can therefore be confident inside an incomplete frame.

Policy must be written somewhere visible

Words such as “urgent,” “grave,” “important,” or “acceptable” smuggle organizational policy into what looks like a factual classification.

The guide proposes three honest placements:

  • put the policy in the rubric, with each level describing an observable situation;
  • put the applicable policy in the case material when it varies by customer or context;
  • ask Jev for component facts and apply the policy deterministically in code.

The bad alternative is leaving policy implicit and allowing the model’s generic assumptions to fill it in. A legal output can still encode the wrong institution’s values.

Small wording changes are substantive

The probes repeatedly show that apparently adjacent formulations are not interchangeable:

  • “the note states that…” and “the note says that…” can receive materially different support;
  • Noul and yes/no Choice can disagree because one weighs a judgment independently while the other forces a distribution among alternatives;
  • an answer set with two options and one with four options are different questions, even when they share a label;
  • a Score’s numeric mean can land on a level with almost no direct support when probability mass is split around it.

The guide is careful not to infer an internal mechanism from these black-box effects. It turns them into design precautions: name the material, state exactly what is being judged, inspect the full probability distribution, and calibrate each primitive separately.

The 50-sentence probe

One concrete script submits 50 Spanish sentences across seven grammatical modalities in a single Choice request:

  • 8 declarative;
  • 7 interrogative;
  • 7 exclamatory;
  • 7 imperative;
  • 7 desiderative;
  • 7 dubitative;
  • 7 nominal phrases.

The committed response selects the expected class for 50 of 50 sentences. I independently parsed the saved request and response and confirmed the 50/50 correspondence. The ambiguous subjunctive sentence “Que el jurado delibere con serenidad y justicia” received 0.83 for desiderative and 0.17 for imperative in the artifact.

That is a sanity check on a hand-written, form-heavy set—not an estimate of general Spanish-language classification accuracy. The labels are strongly signaled by punctuation, verbal mood, and cue words, and the examples were created by the researcher.

Confidence is concentration, not correctness

The repository reconstructs confidence as a function of how concentrated a Choice or Score distribution is. That produces several operational lessons:

  • high concentration does not guarantee the winning judgment is correct;
  • low confidence can reveal overlapping criteria, missing evidence, or a badly framed question;
  • thresholds must be calibrated on the final wording and language;
  • a measured example moved around a 0.70 threshold across repetitions, which would route the same case differently;
  • changing the prompt design can change confidence enough to alter the workflow even when the source case is unchanged.

For production, the guide recommends storing model version, judgment version, case identifier, complete probabilities, confidence, and the path taken—not merely the winning label.

Recorded failure modes and corrections

The repository is unusually useful because it preserves claims that were later weakened or withdrawn:

  • It retracts an earlier “context contamination” story when the evidence better supports reference disambiguation.
  • It rejects a simple inferred knowledge-cutoff date after noticing that low support can mix ignorance, old knowledge, base rates, and temporal semantics.
  • It distinguishes probes designed to study the model from realistic tests designed to validate an application.
  • It notes when a baseline was not actually measured with an empty state.
  • It separates documented TypeSafe behavior, third-party reports, local measurements, and architectural inference.

Those revisions are evidence of a better research process, not noise to remove.

What the spike is building toward

Jev is one component of a broader document system called Zettel. The current design extracts structured claims and evidence from documents with a generative model, then considers Jev or explicit questions for later relevance and judgment steps.

The design’s key distinction is between retrieval and information: retrieving a passage is not the same as showing how evidence changes the set of admissible answers to a question. That makes selection itself part of the case framing. If retrieval omits the best evidence, Jev cannot recover it and the omission may leave no trace.

The repository therefore recommends broader and diverse candidate selection, written selection criteria, system-level abstention, and—where feasible—letting the decision model judge all candidates in stages.

Caveats

  • The repository is a personal working lab with 269 commits, many historical folders, and changing terminology; it is not a stable SDK.
  • Most experiments are small, hand-built probes on one proprietary model version and are not held-out evaluations.
  • The author frequently reports two-decimal API probabilities, limiting analysis near zero and one.
  • Several useful rules are explicitly heuristics, not validated universal laws.
  • The cached artifacts establish what the repository recorded; this capture did not authenticate the external API response or reproduce it.
  • No security audit was performed. The repository’s API scripts depend on credentials injected by the author’s environment.

Why it matters

The durable lesson is upstream of model accuracy: whoever writes the question, answer set, rubric, and retrieval rule has already shaped the decision. Type safety makes that framing executable; it does not make the framing neutral or correct.