Benchmark

We care about real vulnerabilities, not leaderboards. 0sec’s proof is the running, verified track record of real bugs disclosed in the Linux kernel and widely-used open source, with maintainer review — that lives at 0.security, not on this page.

Public CTF benchmarks are a secondary signal. They tell you the core pipeline isn’t broken; they don’t tell you the agent finds real CVEs. CTF challenges are far smaller than real repos, and cross-project scores are protocol-sensitive (different fork, model, turn cap, retry protocol), so treat every number below as specific to its test conditions, not a like-for-like leaderboard.

Where 0sec stands (honest, condition-specific)

Benchmark	Score	Conditions & caveats
XBOW web CTFs	93 / 95 = 97.9% black-box	gpt-5.4 model-specific cohort — per-model single-shot solve rate, not a best-of-N union. ~$0.48/run, $5.20/flag.
Cybench	36 / 40 = 90.0%	First full-suite run, single-config (Azure gpt-5.4), single-shot, 3 retries. BoxPwnr’s 40/40 is best-of-N across ~10 configs — not directly comparable.
npm audit (81 packages)	F1 = 0.973	none profile, 100% TPR, FPR 0.11. Self-published ground-truth set; see the ablation log.
AI/LLM suite (10 challenges)	10 / 10	Self-authored regression suite, not an independent benchmark.
AutoPenBench / HarmBench	Not scored yet	Harness built; no published score.

Read the numbers honestly. The XBOW figure we lead with is the per-model gpt-5.4 cohort (93/95), because it is a stable single-model solve rate rather than a union over an aging artifact-retention window. A wider retained-artifact aggregate exists but is rotation-volatile and is not the headline. Benchmarks here are single-model, single-config; cross-model cost isn’t published; and the 10/10 AI suite is self-authored. For the full measurement discipline — per-attempt rate, Wilson confidence intervals, and why a single solve is an anecdote — see Methodology.

Running the canonical harness

0sec bench run is the single benchmark orchestrator. Integrations own only suite-specific target lifecycle and official grading; every run still produces the same manifest, attempt receipts, scorecard, tournament, and evidence contract.

# Core web/source-audit corpus.
0sec bench run --integration core --variants variants.json

# XBOW: Docker lifecycle + fresh per-attempt flag, scored by the shared oracle.
0sec bench run \
  --integration xbow \
  --xbow-path /path/to/xbow \
  --variants variants.json \
  --attempt-policy independent-repeat \
  --pass-at-k 10 \
  --schedule case-major

# CyberGym: official differential oracle, strict one graded submit per task.
0sec bench run \
  --integration cybergym \
  --cybergym-harness /path/to/cybergym \
  --cybergym-subset results/cybergym-fair-v1.subset.txt \
  --variants variants.json

--attempt-policy pass-at-k is the default and stops a case after proof. independent-repeat retains every scheduled fresh attempt for a per-attempt rate. case-major interleaves variants by task while keeping Docker and CyberGym execution serial.

Cybench, npm audit, AutoPenBench, and HarmBench retain their specialized suite commands until they are migrated through the same integration contract.

Related

• 0.security
• Methodology — per-attempt rate, Wilson CI, single-model caveats
• XBOW Analysis — how the XBOW score is built, and its limits
• Competitive Landscape — where other agents sit, briefly