Enclave runs a hacking benchmark: eleven deliberately vulnerable copies of real software — Grafana, Jenkins, Nextcloud — plus four patched versions as controls. A model gets a shell inside an isolated copy and has to prove it reached code execution on the vulnerable ones without breaking the patched ones. DeepSeek V4.1 Flash scored 11 out of 11. Yanir Tsarimi’s writeup is about what happened when the team re-read every run line by line.

The raw numbers:

  • Eleven vulnerable targets exploited, all four fixed controls left secure.
  • Accepted runs cost $4.65. Including failed attempts and reruns, $5.14.
  • 2,349 shell commands and about two hours and 38 minutes of active model time. The median successful run took 4 minutes and 38 seconds.
  • 268.3 million input tokens, of which 266.2 million were cache hits — reused input billed at a lower rate. That caching is the reason a two-and-a-half-hour attack session cost less than a sandwich.

The audit is the more useful half:

  • The benchmark originally graded only the outcome: did the proof command run on the target? DeepSeek met that condition in every run.
  • Reading the transcripts, only six of the eleven wins followed the weakness the challenge was built to measure. Five found shorter routes that happened to exist in the test environment.
  • Grafana is the clearest example. The intended bug was in plugin-install path handling; DeepSeek instead dropped executable files into a temporary plugin folder and asked Grafana to load that folder as a normal plugin. It did this three times, in 52, 64 and 90 seconds.
  • Two Jenkins runs skipped the intended upload-timing race entirely and used plain file links instead, which was quicker. The proof command still ran, so the old scoring accepted them.
  • Enclave’s own caveat: those five alternate routes are artifacts of its private test environment, not new vulnerabilities in the upstream products. Every model was given the same test code.

Why it matters beyond this result:

  • A hacking agent has no reason to take the path a test author expects. It takes the fastest route that works.
  • An outcome-only score cannot separate “found the intended bug” from “found a different bug that also passes the check.” Both look like an 11.
  • Enclave closed the shortcuts and re-versioned the challenges, so results are no longer comparable across versions — earlier leaderboard entries need fresh runs.

The honest framing is that a benchmark tested the model and the model tested the benchmark back. Enclave published the finding even though it undercut its own headline.

What the thread adds

The 66-comment thread on Hacker News

  • TuxSH — a counter-test on a different target entirely, a hand-annotated decompilation of the Nintendo 3DS kernel, run with subagents. “GLM 5.3 founds almost all the vulns in 30min for $22, while DS only found one vuln for $2 in 40min.” Their proposed explanation is framed as a question: whether DeepSeek does better where “targets have low-hanging fruits than can be found fast.”
  • jrflo — the scope objection, and it landed: “Seems pretty bold to claim deepseek is the ‘best hacking model’ while providing zero comparisons to other models…” sva_ supplies the defense: “Notice the qualifier ‘our’, that is the one they have access to.”
  • The thread’s most portable point — the score belongs to model plus harness plus provider, not to the model alone. gertlabs measured a “~20% higher” average percentile “when given a harness vs a one-shot solution”; mariopt writes that “DS becomes really good if you provide custom tools for the task, on it’s own it’s mediocre”; tacomagick reports that “Claude Code (TUI Harness) does not work well with Deepseek, but Opencode is surprisingly good.”
  • cregy — a confound for any head-to-head: which provider served the weights. Some routes quantize the model to a lossier format, and “The fp4 ones glitched on anything complex,” so they now allow only one provider class.
  • gertlabs, on price: the model “sits near Gemini 3.7 Flash on the Pareto front (not as cheap as the price suggests in practice)” once its heavier reasoning is paid for.
  • pimeys — corroboration from paid evals: “I’ve eval’d these models for weeks against Gemini 3.8, Kimi K3 and Opus 5, and DeepSeek absolutely wins these evals,” at “15-30 cents” per task against $3-15 for a comparable run on another model.
  • Dissent from people using it daily: habosa — “I just haven’t found them to be very good? … DS models seem to get stuck in loops or tell me nonsense. GLM feels like budget Claude.” Grimblewald — “I’m a deepseek fanboy, but has anyone else found flash to not meet expectations? … feels like a strict regression on previous offerings.”

The question the thread kept asking

Best against what? The writeup measures DeepSeek only against its own benchmark, and two commenters raised it unprompted — jrflo (“providing zero comparisons to other models”) and Art9681 (“As opposed to what? Is enclave.ai signed up for GPT Cyber or Glasswing?”). Nobody resolves it in the thread; the runs that exist are DeepSeek’s, on that version of the benchmark. The follow-on question every reader has — pelzatessa asking whether you just point a coding agent at your repo and tell it to find vulnerabilities — gets a concrete answer from AshamedBadger56, who names two open-source pentesting harnesses that supply the tools, prompts and framework around the model.

A note on reading comments as evidence: HN handles are pseudonymous and the site publishes no per-comment scores, so the ordering here is HN’s own ranking, not a vote. This is a slice of the thread. The counter-tests and evaluations quoted are commenters’ own reports, offered as theories where they say so — not verified findings.