Everyone is building right now, and most of it is good. You give a coding agent a few prompts, it wires your security tools together, and you watch the terminal fill with activity. By Sunday, your team has something that maps a target, runs a scanner, reads the output, and drafts a report good enough to send.
So the question comes up on its own.
"We built a working pentesting agent. Why would we buy one?"
The answer has nothing to do with features. A demo proves an agent can do a task. A production system has to do the right task, on the right target, and stop at the edge of the scope you approved. It has to leave evidence an auditor will accept. And it has to keep doing that after every model update, tool change, and new release.
Building has become easier. Owning what you build has not.
That second half is where the boring questions live, and every SaaS bill you've ever argued about is paying somebody to have already answered them. Your team's skill was never the issue. The real question is which of those jobs you want to own for the next two years, one system at a time.
Put them to the pentest agent your team built last weekend. Each one carries a cost that arrives long after the demo.
Is it scalable. Scale changes what the system is. A prototype gets pointed at one target. A program gets pointed at thousands of assets, on a schedule, with concurrent runs. Queue depth, rate limiting, retry behavior, and result storage all become real problems somewhere between the demo and the tenth engagement, and none of them are visible on the Friday it works.
Is security in place. The tool holds credentials for your whole estate, because testing requires them, and it reaches into places normal services cannot go by design. Read that back from an attacker's side. Privileged, unmonitored, unpatched, broad access. The thing that maps your attack surface is worth more than most of what it maps.
Does it stay up when real traffic hits it. Internal uptime becomes production uptime. Downtime on an internal tool is a shrug, right up until a client engagement or an audit window depends on it running Tuesday. Then it matters as much as anything you ship, and you find out whether anyone is on call for it.
None of that is free because the first version was.
The demo works because the conditions are kind. The target is known, access is set up, scope is one application, one engineer watches every action, and nothing that breaks costs the business anything. Production takes all of that away, and early success makes the work that is left look smaller than it is.
People misjudge this badly. In a 2025 METR randomized controlled trial, experienced developers working on their own repositories predicted AI would speed them up by 24 percent. Measured, they were 19 percent slower, and afterward still believed they had been faster.
Seven things have to be true before that harness touches a real engagement.
It can never reach production. An allowlist enforced at the network layer, not a line in a prompt, plus a tested kill switch and a way to stop mid-run when scope moves. In July 2025 an AI agent at Replit deleted a live production database during an explicit freeze, then misreported what it had done. The freeze existed in the instructions. It did not exist in the execution path. That distinction is the entire control.
The over-correction costs as much as the failure. A team that gets burned clamps scope to the assets nobody worried about and gates every step, then operates something a person could have done faster. Scope control has to leave the agent somewhere useful to go.
Findings get verified before a developer sees one. The first false positive that reaches engineering costs somebody a sprint. The third costs you the program, because after that developers close your tickets without opening them. A practitioner testing open-source pentest agents this year caught an invented finding only because he knew the vulnerability class well enough to force a retest. The failure worth planning for is quieter still: a tool returns malformed output and the model reads it with complete confidence.
An auditor accepts the evidence. A console log is not proof, and neither is a model reporting that something appears exploitable. What counts is the request and response, a timestamp, what ran the test, the methodology, and a traceable line from test to report. Under PCI DSS, requirement 11.4 also wants evidence of retesting after remediation, held for twelve months.
The loop stops. An agent hitting a rate-limited endpoint at 2am will spend the night retrying, and each attempt carries the accumulated context of every attempt before it, so spend climbs faster than the retry count.
It holds at estate scale. One target is a demo. Four thousand assets on a weekly schedule is a system. Concurrent runs compete for the same credentials and rate limits, and results have to be deduplicated or the same finding lands in your backlog forty times.
Coverage survives a model change. Anthropic commits to at least 60 days of notice before retiring a public model, OpenAI's floor is six months. Those are your revalidation windows, and they arrive whether or not you have capacity. Newer models reject sampling parameters older ones accepted, so a working call errors after a version bump. A regression suite needs clean targets as well as flawed ones, or you measure misses and never false positives.
Someone owns it by name, with a backup. It will break mid-engagement and need fixing that day. If the answer is the one engineer who built it, you have a bus factor of one on a system holding credentials for your entire estate.
Every one of those is a normal operating condition. If you are working through them as an evaluation exercise rather than a build one, the same list appears as seven checks to run before letting an AI agent exploit your systems.
Everything above assumes a decision. Somebody weighed it, somebody approved it, and the seven requirements were at least considered before the thing went near a target.
Retool surveyed more than 800 professionals in 2026. 60 percent had built software outside IT oversight in the past year. A quarter did it regularly.
Read that against the seven. A harness nobody reviewed has no allowlist enforced at the network layer, because nobody asked for one. It has no spend cap, no evidence retention, and no named owner, because there was never an approval step where those get assigned. It holds credentials to your estate on the say-so of whoever wrote it over a weekend.
This is the version security teams should worry about most, and it is the one nobody plans for. The build versus buy conversation happens in a meeting. The builds that should scare you never reach a meeting.
So if you own security, start by finding out what your team has already built and pointed at something. The decision comes after that.
You are not paying for features. Your engineer can rebuild most of what a security tool does now, everyone in this market knows it, and features stopped being the moat a while ago. The invoice buys the layer underneath. Someone is on call, and it is not your team. Someone patches dependencies on a schedule. Someone keeps the SOC 2 report current so procurement has something to read. Someone answers the question about how a finding was produced.
Build instead and that layer does not disappear. It moves onto your headcount and off the invoice. Invisible to finance. Extremely visible to your engineers. Four numbers describe it, and usually only the first one gets estimated.
The build. Two or three weeks of engineering, less with a coding agent. This is the figure that goes in the proposal, and the only one anyone can defend on the spot.
Recurring maintenance. No good research exists for security tooling specifically, so borrow the planning convention services firms quote, 15 to 25 percent of build cost per year. It covers dependency upgrades, API changes, and connector work that arrives as incidents rather than on a schedule.
On-call loading. Price it as the fraction of a rotation this system will consume. If nobody can name that fraction, it will be consumed anyway, by whoever built it, unscheduled.
The regression suite. A project at first and a standing cost after, and the first thing cut when a quarter gets tight.
Maintenance has been the majority of software cost for as long as anyone has measured it. Robert Glass put it at 40 to 80 percent of total cost, averaging around 60, drawing on survey work going back to the 1970s. That number predates AI coding by decades, and generating the first version faster does nothing to it.
Add the last three, then compare the annual total against your license quote. Well under it, building may pay. Above it, you are buying the same thing at a worse price.
There is a fifth number nobody writes down. Whatever your security engineers were going to get done this year, they are not getting it done, because they are running a platform instead.
Your budget got cut, and around the same time leadership started asking what you were doing about AI. Neither is unusual right now. ISC2's 2025 workforce study found 36 percent of security teams took a budget cut that year, and Retool found three quarters of the people building internal software had been told to use AI.
The conclusion is easy to reach, and plenty of teams have reached it. Build what you can no longer afford to buy.
And for a few weeks that works. Then maintenance arrives, and it arrives as scope. Somebody wants Jira tickets, then Slack alerts, then SSO, then role-based access, then an export to wherever compliance keeps things. Every request is fair on its own. Together they turn a weekend project into a platform with users, and the engineer who built it is on an incident somewhere else, because you hired them to do security work.
Three to six months in it goes one of three ways. It becomes important enough to need a permanent team. It stays small and never reaches the scale somebody promised when they pitched it. Or it degrades quietly until people stop using it and nobody announces that. In all three the headcount does not move, the technology does not move, and the story you take back to management is a mess.
That third path is the common one, and it does not feel like failure while it happens. Internal tools rarely die in one bad afternoon. They fade. The owner moves teams, the backlog grows, and people go back to doing the work by hand without ever deciding to. What you are left with is code, infrastructure, valid credentials, and nobody who can tell you whether any of it still works.
The staffing problem underneath is worse than the budget one. Your strongest offensive engineer can build this and will not want to be maintaining it in eighteen months. Platform engineers will happily run it and cannot tell a real finding from a plausible one.
You did not make a bad call. You made a call with one number in front of you. Build savings land this quarter and get credited to the decision. Ownership costs land three quarters later, charged to something else.
Criticality is the question that comes before everything else, and it is one question rather than a framework.
Critical means the obligations never end. Output reaches an auditor, a regulator, a customer, or a board, and you have a compliance obligation. A deadline depends on it running, and you have an uptime obligation. It holds credentials or touches production, and you have a security obligation. None of those come with a completion date.
Do not take those on in-house to save cost. That is the whole trap in one sentence.
If it is simple, one person or one team using it, build it. Failure means somebody does an hour of manual work and nobody outside the team notices. Build plenty of those.
Case by case, not a default.
There is a version of this question that has nothing to do with cost, and it is the one that carries in a board conversation. Owning the harness means running a software product whose only customer is your own security team, with a roadmap that competes for the same engineers as your security work. That is a reasonable thing to choose. It is a bad thing to discover in year two.
Six questions, each with a build answer and a buy answer. Pick a side on every row.
| Build side | Buy side | |
|---|---|---|
| Compliance | Nothing it produces carries an audit or contractual obligation | Output reaches auditors, regulators, customers or the board |
| Deadline | Failure means manual work, with no date attached | Failure has a date attached, like an audit window or a release gate |
| Production | It holds no production credentials and reaches no production system | It touches production, or holds credentials that reach production |
| Dependency | One team uses it and nobody outside depends on the output | Teams outside yours depend on what it produces |
| Load | Predictable and small, and it stays that way | Volume grows with your estate |
| Continuity | You can name the owner and the backup owner today | It has to work while the person who built it is away |
The rows are not equal, and that is the part most frameworks get wrong. The first three describe what the thing does. You cannot choose your way out of them. The last three describe gaps in how it is resourced, and gaps can be funded.
Any of the first three on the buy side. Buy it, or narrow the scope until that row moves. Compliance, deadlines and production access are obligations somebody else sets, and they do not expire.
First three on the build side, one or more of the last three on the buy side. You can build this. Fund the gap before it runs against anything real, because an unfunded gap is how a good build turns into the fade.
All six on the build side. Build it and get on with it.
A row you cannot answer. You have not scoped it yet. Find out what it touches and who depends on it before you decide anything.
When two rows pull against each other, the tie-breaker is who gets paged. Answer that out loud, with a name, without looking anything up. If you cannot, you are not ready to own it.
Sometimes nothing on the market fits, and that case deserves more than a footnote.
It is real when the constraint is technical rather than procedural. An air-gapped estate. A protocol no vendor supports. A regulator asking in writing for something specific and unusual. A test methodology that is your actual product, where handing it to a vendor means handing over the thing you sell.
It is ordinary when the constraint turns out to be a naming convention, an internal process, or a workflow you could change. Most teams believe their environment is unique. The cheap way to find out is to write the requirements down and send them to three vendors as a scoping question. If two say they already handle it, the requirement was ordinary.
Where it is real, fund it like a product and keep it narrow. Plenty of internal harnesses are excellent at one thing, usually recon or a single vulnerability class, and unreliable everywhere else. Fund the part that earns its maintenance and stop calling the whole thing a testing program. The regression suite comes before the first engagement, not after the first miss. A well-funded internal build beats a badly fitted purchase, and teams who choose this deliberately tend to do it well.
Teams do not get hurt by building. They get hurt by building something critical and then funding it like a side project.
Build more things. The cost of building fell, and you should spend that.
Then go back to the tool you thought of in the first minute.
Is it scalable?
Is security in place?
Does it stay up when real traffic hits it?
If the answers depend on one engineer staying interested, you have not decided to build. You have decided to find out later.
Building is easy now. Owning what you built is still the hard part.
Usually yes, and it is the most common good answer. Put the obligations with continuous liability on the vendor side and keep the thin layer that encodes your environment in-house. Buy the execution harness, the evidence store, and scope enforcement. Build the connector to your asset inventory, the custom auth flow nothing else understands, the report template your regulator expects. That layer is small, carries no compliance obligation of its own, and is genuinely yours.
That is the real risk on the buy side, and it deserves a straight answer. You are exposed to their roadmap, their pricing, and their acquisition. Reduce it the way you would any dependency: own your data on the way out, keep the integration layer thin enough to re-point, and read the exit terms before you sign rather than after. Open source shifts the same risk without removing it, since you inherit the project's release cadence and its maintainers' attention span, and a breaking change upstream becomes your weekend. The build cost falls either way. The ownership cost does not.
Yes. Auditors assess whether a finding can be reproduced and whether the methodology was documented, not who wrote the tool. What fails an audit is missing evidence, an undocumented method, or no record of retesting after remediation. If your harness captures the request and response, the timestamp, what ran the test, and the retest result, provenance is not the problem. Building that capture layer properly is the work.
No. It moves most of it and leaves you some. You still own scope decisions, the integration into your ticketing and asset systems, the review of findings before they reach engineering, and the vendor relationship itself. Anyone claiming a purchase takes that to zero is selling. The difference is that uptime, patching, audit posture, and model changes sit with someone funded to carry them.
Scope it rather than reopening it. Ask for a named owner, a maintenance budget line, and a place in the on-call rotation, all of which are reasonable asks for any system going into production. If those three are approved, the build is properly funded and may well succeed. If they are declined, you have your answer without having argued against anyone, and it is on the record.
