Introducing EchoBench: A Human Calibrated Benchmark for Autonomous Pentesting
Evaluating Autonomous Pentesting Tools
If you have spent any time evaluating autonomous pentesting tools, you have seen the numbers: “A model finds 42 percent of the available issues in this open source repository,” “An agent completes 18 objectives on this benchmark,” “A new harness improves recall by 11 percentage points”, etc.
Every one of those figures can be perfectly accurate and still leave you with the same lingering questions: “Is that any good?” and “What does 80 percent on this benchmark even practically mean?” Those questions are hard to answer because the numbers arrive without a reference point. 42 percent of what, measured against whom?
A score without a denominator is a score without a scoreboard.
Underneath all the vendor claims are questions that actually matter.
Those are the question we built EchoBench to answer.
TL;DR
- Empirically Calibrated Human Outcome Benchmark to evaluate agentic AI web application pentesting.
- A score of 100 marks parity with a measured human cohort, associate pentesters from NetSPI University
- Four components combine by geometric mean: finding fidelity, difficulty-reach, OWASP breadth, and repeatability. No single strength can hide a weakness.
- Models are scored with their harnesses. every published score ships its cohort, configuration, component values, and evaluation rules alongside the headline number.
How does an autonomous AI system compare to human pentesters?
( On the same applications and under the same finding definitions )
How faithfully can they reproduce validated human findings?
One scoping note before we go further: EchoBench measures application pentesting, specifically.
Vulnerability research, fuzzing, and depth of exploitation are all important capabilities, and existing benchmarks already cover them well.
What has gone unmeasured is the job itself: can an autonomous system perform an application pentest the way a person does?
EchoBench supplies that missing denominator, and the denominator is people.
An ECHO score of 100 marks parity with a measured human cohort across the dimensions included in the benchmark. Today, that cohort consists of associate pentesters, and every score is calibrated against their reviewed results. The score describes performance under a defined application assessment protocol. Broader professional activities, such as client communication, architectural review, remediation planning, and novel vulnerability research, remain distinct areas of practice.
That scope is what makes the result useful. A score has a clear subject, a clear reference group, and a clear set of conditions. It also saves everyone from measuring against an unspecified security expert who apparently has unlimited time, perfect recall, and no calendar invitations.
What EchoBench measures
EchoBench evaluates a model and harness configuration together, because that is how these systems actually ship. The model matters, and so do the prompts, tools, browser environment, context handling, execution policy, report generation, and orchestration logic.
A capable model with a browser, durable memory, specialized tools, and a disciplined reporting workflow is a different evaluated system from the same model thinking hard in a minimal shell. Configuration details belong beside every score, and in EchoBench they always appear there.
The benchmark scores four things: finding fidelity, difficulty-reach, OWASP breadth, and repeatability. Agent results are compared with human results on the same application, those target level comparisons feed the four components, and the components combine into the headline ECHO score. The graphic below walks through the flow.
F : Finding fidelity
Strict F1 against the humans who tested the same target. Discovery and reporting discipline, together.
D : Difficulty-reach
Recall parity across easy, moderate, and hard findings. Can the system get to the hard stuff at all?
O : OWASP breadth
Macro category recall parity across the applicable OWASP 2025 categories.
Q : Repeatability
Lower quartile agent F1 against the lower quartile of the human reference. What happens on a bad day.
Public and proprietary targets are scored separately long enough to reveal any provenance related differences, then combined into the final result. Every publication shows the intermediate component and provenance results, keeping the evidence on the table next to the headline number.
The math behind the score
Under the hood, the score is a chain of simple ratios. For every target and every metric, the agent’s statistic is divided by the human cohort’s statistic on that same target, so a ratio of 1.0 means the system matched the humans there. Those ratios feed the four components: F for finding fidelity, D for difficulty-reach, O for OWASP breadth, and Q for repeatability. The components combine within each target group, and the groups combine into the headline number:
Both combinations are geometric means, and that choice does a lot of work. With a regular average, one great score can hide one terrible score and the overall result still looks respectable. A geometric mean refuses to do that: if any of the four measurements sits near zero, the whole score sits near it. Strong finding fidelity cannot make up for minimal OWASP breadth, and excellent easy finding recall cannot erase an unreliable lower tail.
The same property is why the scale gets steeper as you climb. Moving from 40 to 50 can come from cleaning up one weak component. Moving from 90 to 100 requires every component to approach parity on every target group at the same time, so each remaining weakness costs more as the score rises. Nobody coasts to 100.
In this first version, component credit is capped at parity, so the headline tops out at 100 while performance beyond the cohort stays visible in uncapped companion ratios published beside the score. Scoring above 100 is absolutely possible in principle, because the score is relative to a comparison baseline rather than an absolute ceiling, and that is exactly what later versions are being designed for as the industry progresses.
Versioning is part of the design. Every published score pins its benchmark version, human cohort, ground truth, and evaluation rules, so results stay comparable within a version. Future versions will raise the bar deliberately: real world applications, active open source projects, and human baselines that step up in seniority from associates toward experienced consultants. Same yardstick, taller measurements.
The Benchmark Corpus
The current corpus contains five web applications and 184 canonical finding IDs. A canonical finding ID represents one specific, reportable security opportunity under the benchmark criteria, not a general theme.
That precision matters. A report that gestures at access control in broad terms only reproduces a validated authorization finding when it brings meaningful evidence about the relevant endpoint, role, object, and impact.
Security researchers curate and version the canonical corpus, and it is allowed to grow. When a tester uncovers a valid issue that prior work missed, that finding can enter review, because realistic ground truth should improve over time. The corpus is a reviewed reference with explicit versioned boundaries, not a museum.
| Benchmark composition | Current Value |
|---|---|
| Web applications | 5 |
| Canonical finding IDs | 184 |
| Associate application assessments in the program data | Roughly 300 |
| Years of NetSPI University cohorts in the broader program data | 5 |
| Findings with human assigned difficulty labels | 173 of 184 |
| Applicable OWASP 2025 categories | 8 |
Difficulty levels are assigned by humans on a scale that runs from 0 through 10, and the current labeled findings occupy levels 1 through 10. Of the 184 findings, 173 carry labels so far: 117 easy findings at levels 1 through 3, 40 moderate findings at levels 4 through 6, and 16 hard findings at levels 7 through 10.
The distribution is useful and mildly sobering. Easy findings make up most of the labeled corpus, which means an overall recall number will naturally lean toward that larger group unless the benchmark deliberately exposes performance by difficulty.
EchoBench keeps those bands visible. Reliably sweeping up easy findings delivers real operational value, especially across a large application portfolio. Moderate and hard findings test different muscles: sustained exploration, state tracking, careful validation, and longer reasoning chains. The score preserves those distinctions rather than blending them into a single flattering average.
The corpus also spans eight applicable OWASP 2025 categories, with applicability determined at the target level. An application with no opportunity in a category creates no missing finding for either the human or the agent. Nobody gets penalized for failing to find what was never there.
The human reference
The human reference comes from NetSPI University, whose program data spans five years of cohorts and roughly 300 completed application assessments. The current matched snapshot draws on nearly one hundred of those, and the rest are being aligned to canonical finding IDs so the full data set can stand behind the reference.
NetSPI University trains associate and entry level consultants for six months before they perform assessments on these applications. Participants receive structured technical preparation, then work through a consistent assessment program. The result is exactly what a benchmark needs: a structured, repeatable performance baseline.
The current aggregate metric snapshot reflects the matched records available today, while the active reference is defined by the matched assessments included in its published analysis. As the remaining assessments from those five years of cohorts finish alignment, the reference grows toward the full data set of roughly 300.
Before the numbers, a quick primer on the metrics. Precision is the share of reported findings that turn out to be valid. Recall is the share of available findings that actually get found. F1 blends the two into a single score that only stays high when both are high, which punishes the tester who reports everything and hopes, as well as the tester who reports one perfect finding and calls it a day.
Human results are aggregated so each application receives equal influence, which stops an application with more completed assessments from quietly taking over the benchmark. Across the five applications, the application equal human means land at 64.6 percent precision, 57.9 percent recall, and 59.3 percent F1.
Precision
The share of reported findings that turn out to be valid.
Recall
The share of available findings that actually get found.
F1
Blends the two, and only stays high when both are high.
Precision, recall, and F1 are aggregated separately across assessments and applications, and the displayed F1 is the mean of individual F1 values. These are actual assessment level results, not idealized ones. Application level performance shows why target specific calibration matters.
| Web Application | Associate Human F1 |
|---|---|
| Public Application A | 71.2 percent |
| Public Application B | 59.8 percent |
| Public Application C | 38.9 percent |
| Proprietary Application A | 64.2 percent |
| Proprietary Application B | 62.5 percent |
Human F1 ranges from 71.2 percent on Public Application A down to 38.9 percent on Public Application C. The same raw agent F1 therefore means very different things on different targets. A result may trail the associate cohort on one application and match it on another.
EchoBench preserves the application specific denominator before combining anything, which gives difficult targets their proper context and keeps strong human performance visible where it occurs.
The reference also captures normal human variation. Associates omit findings, submit claims that fail strict validation, spend time on unproductive paths, and occasionally discover something clever five minutes after the assessment ends. Experienced consultants will recognize the phenomenon, possibly from personal experience.
Human parity, in other words, represents measured cohort performance, complete with the variation and missed findings that come with real people doing real assessments.
Finding fidelity
Finding fidelity asks the most direct question in the benchmark: on this exact target, how does the system’s strict F1 compare with the humans who tested it?
Useful pentesting output requires both discovery and reporting discipline. Recall measures how many canonical findings the system reproduced. Precision measures how much of the submitted output survived validation. F1 combines the two, rewarding systems that find real issues and describe them with enough specificity to be matched and reviewed.
Component F
Agent strict F1 ÷ human strict F1, on the same application. A ratio of 1.0 is parity. Unsupported claims stay attached to the system that produced them.
This component keeps unsupported claims attached to the system that produced them. A generous reporting strategy may boost recall, but the resulting false positives still count against fidelity. That mirrors the reviewer experience, because someone eventually has to validate the report, reproduce the evidence, and explain why a confident paragraph about remote code execution contains neither code execution nor much remoteness.
F1 cannot express every quality of a professional finding. Severity, clarity, remediation value, and business context all matter. Strict F1 simply provides the focused core measure: did the system reproduce validated findings under common criteria?
The comparison stays application specific. Agent fidelity on Public Application A is measured against human fidelity on Public Application A, and the same alignment holds across all five applications.
Difficulty-reach
Difficulty-reach asks whether the system can get to the hard stuff. It measures recall parity across easy, moderate, and hard findings.
Human assigned difficulty is contextual, and everyone in this field knows it. A revealing error message can make a difficult path much easier. An apparently simple issue can consume an afternoon when authentication state develops opinions of its own. Even with that variation, reviewed labels provide a structured view of investigative reach.
The three bands receive separate attention, which prevents the 117 easy findings from drowning out the 40 moderate and 16 hard ones. A system that reliably collects common issues will show that strength. A system that also reaches findings requiring deeper exploration will show a broader capability profile, and the difference will be visible in the score.
Difficulty-reach uses recall because its question is about access: can the system reach the finding opportunities in each band at all? Precision already lives in finding fidelity, and keeping the signals distinct keeps both easier to interpret.
The remaining 11 findings still need labels before they join the full difficulty analysis. Current reporting identifies the labeled subset clearly.
OWASP breadth
OWASP breadth measures macro category recall parity across the applicable OWASP 2025 categories. Translated: does the system explore everywhere, or does it just have favorites?
Each applicable category carries equal, meaningful influence. That keeps the common vulnerability classes from controlling the whole result and reveals whether a system actually ranges across different classes of weakness.
Component O
Recall parity per applicable OWASP 2025 category, weighted equally. Applicability is decided per target, so breadth is measured against opportunities that actually exist.
Autonomous systems develop recognizable habits. One configuration may inspect injection paths diligently while giving authorization a polite wave from across the room. Another may enumerate broadly but struggle with business logic or stateful workflows. Category breadth turns those patterns into visible evidence.
OWASP categories are a practical organizational lens rather than a perfect one. Finer application security distinctions stay visible in the finding level results, while the category view offers a familiar, reviewable summary of coverage. Only applicable categories count on each target. Breadth measures performance against opportunities that actually exist.
Repeatability
Autonomous testing is stochastic. Repeated executions explore different routes, chase different hypotheses, and produce reports of varying quality. One excellent run is valuable evidence. It is not, however, a service level. The primary view compares the lower quartile of agent F1 with the lower quartile of the human reference. Every assigned attempt stays in the distribution, and failed, timed out, and unusable attempts receive zero under the declared evaluation rules. There is no quietly discarding the bad runs.
Variation itself can be productive. Diverse exploration gives repeated strategies a chance to reach different findings. Repeatability simply checks whether that diversity still produces a dependable floor. The statistic requires adequate repeat coverage. Sparse cells support engineering checks and pipeline validation. Balanced repeated cells support public claims, and EchoBench keeps that line bright.
Component Q
Repeatability asks the question every deployment eventually asks: what happens on an ordinary bad day?
What ships with every score
Every published score identifies the human reference cohort, model, harness, target versions, run budget, and evaluation rules, along with target metrics, component values, uncapped human relative results, provenance views, and repeat coverage. Readers with different operational priorities can take the component values and weight them however their world requires. If that sounds like a lot of disclosure, that is the point.
Cost and runtime stay separate from the score. ECHO measures reproduction of a named human outcome, and resource reporting shows what the system consumed getting there. Readers can then judge whether a strong result fits their own economic and operational constraints.
- Reference:
the human reference cohort, benchmark version, ground truth, and evaluation rules. - Configuration:
model, harness, target versions, and run budget. - Results:
target metrics, component values, uncapped human relative results, provenance views, and repeat coverage.
Provenance and recognition
The benchmark includes public recognizable applications and proprietary ones. Public targets support reproducibility and comparison. Proprietary targets provide evidence under different familiarity conditions.
Target provenance deserves careful treatment. A proprietary application may still share libraries, patterns, descriptions, or derived material with content encountered during model training. Provenance records describe what is known about origin, distribution, access, and benchmark handling.
Run traces add another useful layer. Models sometimes identify a target with great confidence after the hostname or branding already supplied the answer. This can resemble a detective announcing a discovery while the subject’s name remains printed on the mailbox.
Recognition from visible evidence is a valid capability. Hostnames, page titles, logos, certificates, source comments, and default content can all provide strong clues. Preserved run traces let reviewers separate those observations from prior familiarity. The result is a positive and explainable provenance record: reviewers can see what the system observed, when it observed it, and how the conclusion followed.
Sometimes the model performs impressive inference. Sometimes the application is wearing a name tag. Both are easier to understand with a trace.
Repeated runs and pass at k
The primary ECHO operating point is one completed run and one submitted report. That matches how a normal autonomous assessment launches, and it aligns naturally with one completed human assessment. Pass at k provides the companion view for repeated attempts, and EchoBench uses three distinct flavors of it.
Standard oracle pass at k
Each candidate gets a full budget, and hidden ground truth picks the winning report after the fact. A candidate counts as a success when its strict precision and strict F1 both meet the target human medians. A useful upper bound with an honest label: the oracle part tells you that benchmark knowledge chose the winner, which is not something you get to do in production.
Fixed total budget strategy success at k
The one you could actually run. A complete retry policy gets one shared budget and must submit a single final report, blind, using only what the evaluated harness can see. Agent execution, selection, merging, reconciliation, and report generation all draw down that same declared budget. There are no free retries.
Benchmark union F1 at k
Answers the coverage question. True findings are deduplicated by canonical finding ID, while every distinct false positive and unsupported claim stays attached to the union, shown separately for full and fixed budget regimes. Claim volume and duplicate volume appear beside the curve, so readers can estimate the review burden before volunteering for it.
The three views answer different operational questions. Oracle pass at k measures the benefit available when hidden truth can pick the winner. Fixed total budget success measures a retry policy you could actually deploy. Union F1 shows the cumulative coverage and the reporting cost that comes with it.
Pass at k stays outside the four ECHO components because retry allocation and report selection are separate operating policies. The headline score describes the core human relative assessment result. Repeated run curves show what additional resources and orchestration can buy.
Balanced comparison cells must align target, model, harness, prompt, budget, timeout, and version. The current snapshot has no cells with five comparable runs yet, so the first pass at k curve arrives after balanced repeat collection. The existing repeats are already earning their keep by validating software behavior and scoring logic.
Publication protocol
The first numeric release will complete the alignment and repeat coverage needed for defensible ECHO reporting. Here is what that involves.
The methodology will spell out validation criteria, applicability decisions, human aggregation, component construction, repeatability statistics, report matching, and configuration reporting. Judge agreement and uncertainty intervals ride along with the primary results, because a benchmark that scores other systems should be comfortable being scored itself.
- Finding level human alignment
will confirm how human reports map to canonical finding IDs, and the remaining difficulty labels will finish out the scale. - Provenance records
will document the public and proprietary target groups. - Balanced repeated cells
will unlock repeatability and the first pass at k curves, and fixed total budget experiments will put complete blind retry strategies to the test.
Sensitivity analysis will show how the conclusions change when each target is removed in turn. With five applications, any single target can swing a compact benchmark, so the public results will identify assessment counts and repeat coverage clearly enough for readers to judge each comparison for themselves.
Component results and uncapped human relative views will appear beside the headline score. Readers will be able to see whether a system reaches parity through balanced performance or approaches it while hiding a distinct weakness in fidelity, difficulty, breadth, or repeatability.
Cost, runtime, model charges, and infrastructure use will appear as separate evidence. A performance and cost view can then show which configurations provide useful operating points while keeping quality and expense legible as separate axes.
EchoBench covers validated web application findings under defined assessment conditions, and that boundary is what keeps the claims testable. Professional pentesting also includes scoping, client communication, architectural judgment, remediation advice, novel research, and creative investigation beyond the current corpus. Future work can measure more of those skills with the same commitment to measured cohorts and paired evidence.
A more useful denominator
EchoBench gives autonomous pentesting a practical human reference.
- Agents and humans face the same five applications.
- Their reports map to the same 184 canonical finding IDs.
- Difficulty comes from human assigned labels, and breadth follows applicable OWASP 2025 categories.
- Repeatability reflects lower tail performance, and public and proprietary provenance remain visible.
- Models are evaluated with their harnesses. Retry policies and cost stay explicit.
The reference draws from roughly 300 associate application assessments across five years of NetSPI University cohorts, and the cohort behind every future score will be measured with the same care.
The benchmark is designed for scrutiny and useful disagreement. Researchers may prefer different category systems, severity treatments, or operating budgets. Practitioners may value precision, hard finding reach, consistency, or cost differently. EchoBench gives those discussions a common factual base.
We invite researchers, builders, consultants, and security teams to examine the methodology, challenge assumptions, contribute evidence, and help expand the measured cohorts. The goal is a benchmark that improves through review and remains clear about what every score means.
EchoBench makes the denominator human
Put an autonomous AI next to a measured human cohort
EchoBench Empirically calibrated human outcome Benchmark
Explore More Blog Posts
Verifying the Verifier – Assessing Identity Verification Services
As AI-driven identity verification becomes the front line of defense against fraud, this piece breaks down how to stress-test these systems, from building synthetic IDs and deepfake voices to uncovering the gaps attackers are most likely to exploit.
BOFScale: A CDN-Fronted Tailnet from a BOF-PE
Discover how BOFScale leverages a modified Tailscale daemon compiled as a BOF-PE to seamlessly hide C2 traffic and DERP relays behind CDNs using standard WebSockets.
Stealing the Artifact – JFrog Artifactory Vulnerability
Discover how NetSPI uncovered and reported vulnerabilities in JFrog Artifactory that allowed unauthenticated attackers to bypass authentication and steal an arbitrary artifact.