Most of the conversation around AI agents in security is about which model is driving. That's the least interesting variable, honestly. The two changes that moved our agents the furthest this year had nothing to do with the model. They changed how the agent acts on the target, and how it records why it believes something.
We call them tooling as code and structured scratchpads. This post walks through both: what was broken before, what the setup looks like now, and what actually changes in practice.
Give a web application pentest agent the usual toolset and on paper it looks fine:
http_request to send an HTTP requestgraphql_request to send a GraphQL querybrowser_click, browser_type, browser_snapshotcrawl_url, extract_endpoints, decode_jwtUnderneath, each of these is a thin wrapper around an HTTP library. The wrapper isn't pointless. It holds the session, keeps the cookie jar, sets a real browser TLS fingerprint, sends traffic from the right network identity, and records the exchange as evidence. That's real value, and it's the reason you don't just hand the agent curl.
The problem was never the wrapper. It was the calling convention.
Say you're testing an object reference flaw and you need to sweep five hundred identifiers. With one tool call per request, that's five hundred round trips through the model, five hundred response bodies pushed into context, and a run that burns its whole budget during recon and never even reaches analysis.
Every obvious fix fails the same way:
iterations argument. Works fine, right up until each iteration needs a different path, a different header, or a decision based on the previous response.range argument. Then you add filter, stop_on, vary_header, follow_redirect_if, and so on. Every real testing pattern becomes another field, and the tool schema keeps growing. Remember, you pay for that schema in tokens on every single request the agent makes, whether it uses the tool or not.fuzz_parameter, swap_object_id, replay_as_role. Now you've got thirty tools, the model has to pick the right one every time, and the second a target does something slightly unusual, none of them fit.The real tension is simple. A tool schema is a fixed shape, and security testing is not a fixed-shape activity. You're always one loop, one conditional, or one intermediate transformation away from whatever the schema anticipated.
And sitting on top of all of it is one obvious fact: the model is already excellent at writing Python. Handing a fluent programmer thirty rigid buttons instead of a keyboard is a strange trade to make.
We stopped shipping actions and started shipping libraries.
Each discipline gets a Python SDK that the agent imports and drives inside a sandbox:
| Discipline | Library | What it covers |
|---|---|---|
| Web / API pentest | strobes_pt |
HTTP, GraphQL, sessions and login, browser control, crawling, injection oracles, authorization matrices, JWT |
| Source code review | strobes_cr |
Repository entry points, taint tracing, dependency inventory, citation, coverage accounting |
| Network pentest | strobes_net |
Host/port/service discovery, TLS inspection, content discovery, template scanning |
| Cloud / CSPM review | strobes_cloud |
Account verification, prescan, public-exposure derivation, identity trust paths |
The agent doesn't call a tool per request anymore. It calls one tool, "run this code," and the code does the work.
Five hundred requests. One tool call. Three lines of output.
The filtering happens where the data is, not after it's been dragged back through the model. That's the whole economic argument, and it holds for every discipline. A dependency inventory, a port sweep, a cloud prescan, a taint trace across ten thousand files: all of it is loop-and-filter work that has no business being narrated one request at a time.
The value the old tools provided didn't disappear. It moved inside the library, where it's stronger.
Three properties matter here.
Egress identity is structural. All target traffic leaves from the sandbox, never from the worker. That isn't a convention we ask the agent to follow. It's a property of where the library runs.
Everything is recorded, automatically. Every request the web SDK sends, every analysis the code-review SDK runs, every scan the network SDK launches, every check the cloud SDK reads: each one appends a row to an evidence trail that gets drained into queryable tables. The agent doesn't opt in, and it can't forget.
This gives us something no findings list can. We get a record of what was examined, not just what was reported. "Was this host ever scanned, with what, and when" is a question a client asks the moment something gets missed, and you can't answer it from a report. You can answer it in one query from a coverage table.
Absence is loud, never empty. This is a rule we enforce across every SDK, and if I could keep only one design decision, it'd be this one.
A security tool that returns "no findings" because it couldn't run looks identical to one that ran and found the target clean. That confusion is the most dangerous failure mode in the whole domain, because "clean" is what gets reported to the customer.
So every function returns an explicit status. A missing binary, a dead credential, an unopened account, a scan killed by a timeout: all of them return ok: false with a reason. They never return an empty list.
We hit four separate instances of this while building the code-review toolkit alone. One static analysis engine analyses git-tracked files only by default, so point it at an untracked directory and it examines zero files, exits successfully, and reports nothing. Another silently skips tests/, vendor/ and build/. A single unparseable rule aborts a whole ruleset, so one broken pattern zeroes out every unrelated language's rules too, and the whole thing reads back as "no attack surface."
Through a naive tool, every one of those is a clean bill of health for code nobody looked at.
A few things we didn't design for but got anyway:
Tooling as code fixed how the agent acts. It did nothing for how the agent concludes.
Scratchpads aren't a new idea. Reasoning-and-acting agents have kept notes for years, and you'll see the same thing in Claude Code. What's new here is what the notes are for.
Think about how an agent actually reaches a finding. It probes forty endpoints. On the eleventh it sees something odd. It chases it across six more requests, gets a signal, and files a vulnerability. Between the observation and the submission sit thirty more requests worth of unrelated traffic.
By the time it writes the finding, the evidence that justified the claim has been diluted, truncated, or evicted from its working context. So the write-up isn't read from a record. It's reconstructed from a degraded memory.
That reconstruction is what a false positive usually is. Not a hallucination in the dramatic sense. A confident summary of something the agent half-remembered.
You can try to fix this with prompting. We did. "Explain your reasoning before filing" gives you a paragraph of reasoning-shaped text written after the decision was already made. It's a rationalisation, not a record, and there's no way to falsify it.
Our own production data made the point pretty sharply. In one corpus we found findings whose own write-up said exploitation wasn't possible. The analysis was correct and sitting right there in the submission, and they got filed anyway, and every one was rejected by a human. The missing piece was never knowledge. It's that the analysis wasn't load-bearing on the decision to file.
So we gave the agent a method, and then made the method structural instead of advisory:
The pad opens before the probing starts, not after. That ordering is the whole point. A pad written afterwards is a summary. A pad written during is the reasoning.
A pad is just a plain markdown file in the workspace. The agent edits it with the normal file tools it already has, so it stays completely freeform. It can write anything, anywhere, at any point.
The shape is the same for every investigation, whatever the vulnerability class:
One pad per finding, or per scenario. A pad covering four findings answers cleanly for none of them.
The Evidence section is where the two features meet. Every request the SDK sends gets recorded and stamped with a short, content-addressed id, handed straight back to the agent in its output:
[evidence] 2 exchange(s) recorded:
ev-39ef025a66c8 GET /api/orders/1042 as bob -> 200
ev-8ed4a9dfb1fa same request unauthenticated -> 401 (the control)
The agent cites those ids inline as it reasons. A reader, human or machine, goes from "the control returned 401" straight to the actual bytes.
And because the ids come from the content of the exchange itself, an invented citation can't resolve. A hallucinated reference stops being an invisible problem and turns into a visible signal.
Here's the part that makes the pad more than a diary.
Every pad carries a validation checklist: the questions that actually decide whether the thing being claimed is a vulnerability. Four apply to any finding at all:
Those four are where findings actually die in human review. We ask them of every finding, including the big chunk whose class we can't confidently identify. Not knowing what kind of flaw something is has never been a reason to ask nothing.
On top of that, each class adds its own discriminators. We maintain roughly three hundred class checklists across web, API, authorization, authentication, cloud posture, mobile, dependency, secrets and business-logic families.
The cross-origin sharing checklist is a good example, because it's the worst false-positive class we measure. In one judged sample, 89% of these findings were rejected by human reviewers, and every rejection had the same shape: someone saw a permissive header with a command-line client and called it data theft. Two facts kill most of them, and the checklist asks about exactly those two:
Answer wildcard, and the checklist tells the agent, in plain language, why that isn't a disclosure. A browser refuses to attach cookies to a wildcard, so the attacker page receives exactly what anyone could fetch unauthenticated.
Two design decisions here matter more than the questions themselves.
The answer is a small controlled vocabulary; the reason is free text. Prose can't be validated. A one-word answer from a fixed set can. The why: line is where the agent says what it observed that makes the answer true, and a reason under a minimum length isn't a reason.
The questions get seeded into the pad, not stored in a document the agent is told to go read. A mandate inside a document nobody opened isn't a control. We measured this. On findings of one high-rejection class, the relevant guidance skill was loaded on 23% of them. The other 77% were filed by an agent that never saw the guidance. So the checklist arrives in the pad, at the moment the pad opens.
When the agent files, the pad travels with the finding. Before anything gets written, a gate reads it.
Three things about this flow are deliberate.
Corroboration checks the reasoning against the bytes. The checklist proves the agent reasoned. It doesn't prove the reasoning was true. An agent can answer "the origin was reflected" about an endpoint whose recorded response says wildcard, and every rule in the validator would pass, because the validator only ever saw the answer.
So for the mechanical questions (what header came back, what status was returned, how many distinct requests were actually made) the answer isn't a matter of judgement at all. It's in the evidence the SDK already recorded. Where the claim and the bytes disagree, the agent gets told exactly how. A claim the evidence contradicts is a stronger false-positive signal than any unanswered item, because the agent has asserted something the data it collected doesn't support.
This only works because of tooling as code. The evidence trail exists, in a queryable form, with no cooperation required from the agent.
A "stop" answer doesn't delete the work. When the agent's own answers say this isn't exploitable, there's nothing to correct. But the observation is often still worth keeping. A permissive header is worth a hardening note. So the finding isn't discarded. It's refused at that severity and recorded as an observation. The problem was never that hardening notes existed. It was that they showed up at Medium and above, which is the band humans dismiss.
The gate can't lose findings. This is the constraint every other decision bends around:
Neither feature is that interesting alone. Together they close a loop.
Tooling as code produces a durable, citable record of everything the agent did, as a side effect of the agent doing it. Structured scratchpads make the agent reason against that record while it works, and make the conclusion checkable before it gets filed.
Without the SDKs, the pad has nothing solid to cite and the gate has nothing to corroborate against. It can only check that the agent answered, not that the answer was true. Without the pad, the evidence trail is just a pile of exchanges nobody connected to a claim.
On cost and reach. Sweeps that used to be unaffordable (five hundred identifiers, an authorization matrix across four roles, a dependency inventory across a monorepo, a full cloud prescan) are now single operations that return a summary. The budget goes into analysis instead of narration.
On honesty. The interesting shift isn't that fewer false positives get filed. It's that the agent's uncertainty became visible. "Control not run" is now a recorded answer instead of an absence nobody noticed. "Filed with unresolved items" is a state a reviewer can see. An unproven claim gets capped instead of dressed up.
On what a finding is. A finding used to be a paragraph the agent wrote. Now it's a paragraph, plus the notes written while the work happened, plus the specific questions that decide the class with the agent's own answers, plus the exact exchanges that prove it. All bound together so someone who wasn't there can check the reasoning instead of trusting it.
On what "we found nothing" means. With a coverage record, "no vulnerabilities found" is finally a claim with content behind it: here's what was examined, here's what was tried, here's what got explicitly ruled out and why. Without it, that sentence means nothing at all, and it's the sentence customers act on.
Both features are live across the Strobes agent platform: web and API pentest, source code review, network assessment, and cloud posture review.