Solace’s Amir Ghasemi tests two open-weight decision models as local alternatives to hosted Jev in an Agent Mesh ticket router. A compatible endpoint lets the workflow keep the same decision interface; it does not make the models equally accurate or their confidence scores interchangeable.

What Solace tested

  • The company used 40 synthetic tickets, each with department, urgency and severity questions, and three runs per model. The test was not independently reproduced here.
  • For department routing, Solace reports 87.5% accuracy for H2O-Lightning-4B and 85.0% for Perplexity’s Decider Flash, versus 92.5% for Jev 1.13 and 88.3% for Jev 1.14.
  • Escalating answers below 0.7 confidence to a self-hosted Qwen reviewer brought both open models to 95.0%, according to Solace. The company selected that threshold after seeing the same tickets, so this is not a held-out result.
  • Solace ran local models on one Mac but Jev over the public internet with different concurrency. Its latency measurements do not establish a fair speed comparison.

The operational details

  • The workflow sends typed questions to a /v1/systemone endpoint. H2O uses a compatibility shim; the article supplies a small HTTP wrapper for Perplexity’s package.
  • Solace says its H2O shim’s default yes/no probability floor obscures uncertainty; the test disables it. The company also found that changing the urgency cutoff affected agreement with its labels.
  • The sample wrapper processes requests one at a time and does not check an API key. Solace recommends an authenticating proxy and a restricted network; its sample is not a production server.
  • Keep the reviewer local too if data must remain inside the boundary. Solace says an outbound-blocking proxy logged no external requests during its end-to-end demonstrations; I did not verify that claim.

The synthetic sample is small, and a valid typed answer can still route a ticket incorrectly. Test thresholds and escalation on separate, representative labeled data before relying on the workflow.