Pentest reporting can now run without a human writing it. The whole document: executive summary, severity scores, evidence, remediation guidance, tickets opened in Jira.
Scanner-generated reports have existed for decades, and producing a document was never the hard part. The hard part was deciding which findings were real, proving them, and explaining their impact.
What agentic pentesting changes is where pentest report automation gets its quality. When validation and evidence capture happen during testing, the reporting layer assembles verified data instead of reconstructing it afterward.
One survey of pentest teams put reporting at 20 to 60% of testing time, with 40% naming copy-pasted CVE descriptions as their biggest time sink. The hours are real. But hours were never why teams kept doing it by hand. Trust was, and automated reporting only holds up if the pipeline is built to earn it.
If you’d rather just look at one, there’s a full report at the bottom of this page.
The term gets used for at least three different things, and they aren’t equivalent.
Templating tools speed up document assembly. You still write every finding by hand, and the saving comes from not rebuilding the same structure every time.
AI-assisted write-up goes further, generating finding descriptions and executive summaries from a tester’s notes, scanner output, and screenshots. This works without the testing itself being automated, which is why it’s the most common form on the market. It also inherits whatever was wrong with the input, because a model composing prose from notes has no way to tell whether the finding underneath is true.
Only the third kind touches that problem. It moves validation and evidence capture into the testing process, so the reporting layer receives findings that are already proven. Which raises the obvious question of where in the process that actually happens.
A finding passes through a lot of hands between the moment it’s discovered and the moment someone reads about it. Most reporting tools own the tail end. Quality gets decided long before that.
Here’s the full sequence:
A candidate finding is created from a test result
A validation state is assigned: exploited, detected but unconfirmed, or informational
Exploit evidence is stored during the run: request, response, payload, state change, timing
Duplicates are correlated so one root cause doesn’t appear as five findings
Business context is attached: what the asset is, whether it’s exposed, what it touches
Enrichment runs: CVSS vector, CWE, OWASP category, exploitability data
Executive and technical narratives are generated from the same record
Sensitive values are stripped before generation, not after
Release conditions are checked, whatever yours are
Outputs are produced: client report, dashboard view, engineering tickets
Retest appends state to the same finding rather than replacing it
Everything up to enrichment is a testing-time decision. Everything after assembles what already exists.
Notice where the work sits. Everything up to enrichment is a testing-time decision. Everything after is the reporting layer, and by the time it runs, every question that matters has already been answered. A narrative generator can’t decide whether a finding is real. It can only describe a record that was either built properly or wasn’t.
Which is why exploitation-first changes the outcome rather than the schedule. When exploitation happens before reporting, evidence exists as a byproduct of the exploit instead of something someone has to remember to collect. Validation state gets set by what actually happened, not by a judgment made three days later against a screenshot folder.
We walked through a full run of this in Agentic Pentesting with Strobes AI, including the evidence package the reporting phase draws from.
Get that order wrong, and the failures are predictable.
Every failure below traces back to the same thing. A claim gets made that the evidence can’t support, and nothing in the pipeline stops it.
A tool spots a pattern that resembles a vulnerability, writes it up with the confidence of a fact, and ships it. The client’s engineer then spends an afternoon proving it isn’t exploitable. That afternoon costs more than the automation saved, and it takes the report’s authority with it. One bad Critical and the reader starts checking your Mediums.
Findings that were exploited, findings inferred from behavior, and findings matched off a version string all get the same format and the same badge. Nothing tells the reader which is which, so they either trust all of it or verify all of it.
A screenshot of a login page proves nothing about an authentication bypass. The test is whether an engineer who doesn’t trust you can reproduce the issue from what’s in the report. Request and response pairs pass. A terminal capture with the interesting line scrolled off the top does not.
Base scores are context-free by design. Publish them as final and an internal admin panel carries the same number as an internet-facing payment endpoint.
Every report arrives fresh with no memory of the last one, so nobody can see what’s new, what came back after a fix, or what stayed closed. Retest reports contradict originals because findings got edited after publication with no version history behind them.
None of this is new. Manual reports fail the same ways, just slower and less consistently.
What changes is that hand-writing findings was doing quality control as a side effect. It’s hard to write a convincing description of a vulnerability you don’t understand, so the writing itself caught things. Take the writing away and that check goes with it, which means the validation that used to happen by accident has to be built in on purpose.
Five checks decide whether any of that holds up. Run them against whatever you use now, including your own process.
What happens to a finding the platform detects but cannot exploit? It should be held back, or ship with an explicit unconfirmed state. If it ships looking like everything else, you don’t have this gate, and you can’t retrofit it. It’s decided by whether your platform exploits or scans.
Can you sort your findings by confirmation state? State needs to be a field, not a sentence buried inside each finding. Business logic flaws will always need a human look, and the client has to see which ones without reading all forty.
Hand your last three high-severity findings to an engineer who wasn’t on the engagement. Can they reproduce the issue without contacting you? If they come back with questions, the evidence documents rather than proves. OWASP calls these reproducible test artifacts and names curl commands, proof-of-concept code, and HAR files capturing request and response flows.
Two findings, same base score, different assets. Do they land at the same priority? If yes, the environment never entered the pipeline.
Pull this quarter’s report and last quarter’s. Can you tell which findings recurred without reading both? Findings need an ID that survives the report they first appeared in.
The last three gates live in the reporting layer, which is why reporting tools cover them. The first two sit upstream, which is why no reporting tool can give them to you.
The architecture behind the first two gates, including scoped boundaries and human-in-the-loop approval, is covered in our complete guide to agentic pentesting.
Clear all five and the document itself becomes a question of format.
A pentest report has always had a structural problem, and it isn’t about which sections it contains (we covered that in the essential elements of a penetration testing report). The CISO needs a page. The engineer needs the HTTP traces. Ship one PDF and both scroll past most of it to find their part.
The reason it stayed that way is cost. Writing a second version of the same findings for a second audience means writing it twice, so teams wrote one document and let everyone share it.
That constraint disappears when findings are structured data:
| Output | Contents | Reader |
|---|---|---|
| Executive summary | Severity counts, top priorities, posture verdict, one page | CISO, board |
| Technical report | CVSS vector, CWE and OWASP mapping, evidence, proof of concept | AppSec, engineering |
| Findings matrix | Sortable table of finding, endpoint, severity, classification | Program owner |
| Remediation guidance | Per-finding fix detail, grouped into priority tiers | Developers |
Then there’s the section most reports don’t have at all. A coverage map records what was tested and held, not just what broke. Horizontal IDOR attempts blocked across fourteen resource types. JWT algorithm attacks rejected. Price manipulation ignored server-side. Card data masked to the last four digits.
That’s the part auditors ask about first, and it’s the part a findings list can’t answer. Eleven findings tells you what’s wrong. It doesn’t tell you whether anyone looked at authentication. OWASP suggests exactly this as an appendix: a checklist of all the tests conducted.
Findings also leave the report entirely, into Jira, GitHub Issues, or ServiceNow with reproduction steps already attached. Nobody retypes a vulnerability into a ticket.
Every finding carries the request and response that proved it. Twenty-five wrong OTP codes, all rejected, no lockout, and response times uniform at 427 to 438 milliseconds across every attempt, which is what rules out throttling. That’s the level of detail an engineer needs to reproduce the issue without calling you.
You can download a full report from a Strobes run if you want to look at one. It’s a test app, not a customer engagement, so it’s useful for judging the format and not much else.
With the document handled, the remaining manual work is easier to see.
Scoping is the obvious one. Someone has to define what’s in and out, what the perimeter is, and what testing would break. An agent shouldn’t decide its own boundaries.
Business context for severity is the one people underestimate. Base scores are context-free by design. Whether an affected asset is internet-facing, what data it touches, what a compromise costs this particular company, none of that is derivable from the vulnerability itself. It gets stated once during scoping and applies to everything after.
Some exploit payloads shouldn’t run without a person signing off, which is a configuration decision rather than a limitation. Any platform worth using lets you set where that gate sits. And some findings can’t be safely confirmed at all, where exploitation needs production data or would cause damage the scope doesn’t permit. Those still belong in the report, labeled as unverified rather than dropped or dressed up as proven.
None of that is the write-up. The document assembles itself. What needs a person is the judgment before testing starts and the calls that can’t be made from evidence alone.
Which is roughly the division an auditor cares about too.
An automatically generated report can satisfy audit requirements when the underlying assessment and the report itself carry the evidence the applicable standard and the auditor ask for. There’s no general rule that AI-generated reports are accepted or rejected. The question is what’s in the document.
OWASP’s Web Security Testing Guide sets out what a well-formed pentest report contains. It describes these as suggestions rather than strict rules, which is worth knowing before anyone treats the list as a compliance checklist:
The team, with their expertise and qualifications
Scope: the boundaries and needs of the engagement, agreed with the organization
Limitations, including out-of-bounds areas, broken functionality, lack of cooperation, lack of time, and lack of access or credentials
Timeline: the duration of the engagement
Test methodology used
Severity and risk rating explanations
Per finding: what the vulnerability is, how to exploit it, and the damage that may result
On a retest: the updated status of previously identified vulnerabilities, cross-referenced with the current test
Most of that is data an automated pipeline already holds, which is why the compliance case for automation is stronger than it sounds. Scope, dates, methodology mapping, and per-finding evidence are structural outputs rather than things someone has to remember to write down.
That last item is where automation clearly wins. Carrying previous status forward means findings have to persist across engagements, which is the fifth gate above. A pipeline that regenerates a fresh document each quarter can’t answer what happened to last quarter’s criticals. One that appends state to a persistent record answers it automatically.
None of which helps if the report itself leaks.
A pentest report is a document explaining exactly how to break into something. Reports routinely contain internal endpoints, session tokens, working exploit payloads, architecture detail, and test credentials.
OWASP is direct about this. Its reporting guidance says possibly-sensitive data should be masked, naming passwords, personal information, and credit card details, and that it’s advisable to secure the report and encrypt it so only the receiving party can use it.
Worth thinking about before you automate the pipeline that produces it, because automation means that data moves through more systems than it used to. Ask where captured evidence lives and how long it’s kept, who can open a generated report before the client does, whether sensitive values get stripped before generation or after, whether any of it reaches a third-party model and under what terms, and whether the record of who accessed what is itself retained.
Redaction has to happen inside the pipeline. Stripping tokens out of a finished PDF is a manual step, which means it’s the step that gets skipped on the fourth report of the week.
Minutes, when the findings are already validated and evidenced. The generation step is fast because it’s assembling structured data rather than composing anything. The time in an engagement goes to testing and validation, not writing.
Yes, and tools have done a version of this for years. The useful distinction is whether the AI is composing prose from a tester’s notes or assembling a document from validated findings with evidence attached. The first can produce a confident write-up of something nobody proved. The second can’t.
It depends on the contents, not the generation method. If the report carries scope, methodology, limitations, dates, attribution, and per-finding evidence sufficient to reproduce and remediate, the way it was produced is not usually the question. Confirm requirements with your auditor and the applicable standard.
They should stay in the report with an explicit unverified label. Dropping them hides real risk. Presenting them as confirmed is worse.
Yes. Report structure and which sections appear are configurable, and severity thresholds can be set per engagement.
Report quality was never a function of how well the report was written. It’s a function of whether the finding was validated, evidenced, and structured before the reporting layer touched it. Get that right, and the document is close to a formatting problem.
Run the five gates against your current process. If the first two fail, no reporting tool will fix them.
Gates one and two are decided by whether your platform exploits or scans. If you want to see what the second kind produces, point Strobes AI pentesting at one application and read the report it writes.
OWASP Web Security Testing Guide, Reporting Structure — owasp.org/www-project-web-security-testing-guide/latest/5-Reporting/01-Reporting_Structure
NIST SP 800-115, Technical Guide to Information Security Testing and Assessment — nvlpubs.nist.gov/nistpubs/legacy/SP/nistspecialpublication800-115.pdf
Cyver Core, pentest reporting time survey — core.cyver.io/how-long-do-you-spend-on-pentest-reporting/
