AI-PM Wiki uses Jev to suggest highlights in technical documents. Its annotation backend shows a concrete integration mistake: uncertainty about a highlight’s colour was being treated as a reason to replace the whole judgment.

Ask which probability controls the decision

The Jev adapter asks three questions per paragraph:

  • Worth: a yes/no probability for whether the paragraph deserves highlighting.
  • Purpose: a choice among the supplied colour meanings, such as a definition or conclusion.
  • Importance: a score used to rank suggestions.

The returned confidence is the smaller confidence from the colour and importance answers. It says nothing directly about the separate worth question.

The maintainers report that an earlier average-confidence cutoff sent whole chunks to an expensive fallback. Their recorded page averages straddled that cutoff. Those deployment observations are source reports, unreproduced here.

Fall back on failure, filter on worth

The current implementation keeps colour and importance confidence as diagnostic metadata. It switches providers after errors such as timeouts or unusable responses, then filters suggestions using the worth answer.

  • A local synthetic control-flow check retained a high-worth suggestion despite low ancillary confidence, without calling the fallback.
  • Separate offline checks exercised low-worth filtering, error-triggered fallback, and reuse of a covered cache entry.
  • A partial cached page correctly required another judgment when the request expanded to additional paragraphs.

These checks used explicit test doubles. They establish code behaviour, not Jev accuracy or calibrated probabilities. Typed output is not a correct decision.

Keep the paid suggestion separate from the write

The highlighting service returns suggestions for a separate adoption flow. Ordinary annotations remain available when model services are unconfigured.

Before model calls, it checks paragraph text against the site’s document index and applies simple exclusion rules. The service also implements call limits, estimated-cost reservations, and persistent page caching. Its budget and rate counters reset with the process, so they need shared durable storage before scaling across instances.

The repository’s own coverage map leaves several live-provider checks unverified. Local execution passed TypeScript checking and fourteen unit-test groups; the HTTP group could not start because the parent wiki’s search index was absent.

Caveat: source inspected and offline control flow exercised; no live Jev calls, highlight-quality evaluation, deployment reproduction, or security audit.