What to ask any vendor about data access, model training, storage, and deletion.
Your application’s responses can contain plenty you wouldn’t paste into a chat window. Session tokens, customer records, internal error traces.
An agentic pentest reads all of it. Which parts actually reach the model, where does that processing happen, and who still has a copy when the test is over?
These are the questions that come up in almost every security review we go through. We answer them one deal at a time, in questionnaires, on calls, in documents that get emailed to one procurement team and read by nobody else.
Those are fair questions when your name is on the approval. Getting clear answers shouldn’t require piecing together a privacy policy, a sales call, and a spreadsheet passed between procurement and engineering.
Here are the questions worth asking any agentic pentesting vendor, and how we answer each one at Strobes.
Most concern about agentic pentesting starts from one wrong idea: that the model is the thing doing the testing. Reading your application, probing endpoints, deciding what’s exploitable.
That isn’t how it works. The checks themselves aren’t generated by the model. Scanning, enumeration, and exploitation checks are performed by conventional security tooling and connectors, the same category of tools your team already runs. The model decides which to run and in what order, then reasons over what comes back. It correlates output across tools, triages findings, judges whether something is genuinely exploitable, and writes it up.
That distinction is about who does what, and it is worth establishing early because most of the worry assumes the opposite. It does not by itself tell you how much of your data reaches the model. Tool output can carry complete response bodies, so “a conventional scanner ran the check” and “the model never saw the response” are two separate claims. The second one has to be answered on its own terms, which is section 1.
It also means “what the tool saw” and “what the model saw” are two different questions. Most of this page is about the second one.
What to ask: what actually reaches the model, and what can the agent do without a human saying yes?
Start with a concrete case. An authorized test checks whether one account can read another account’s order. The response comes back with an order number, a customer name, a delivery address, and an email.
You now have evidence of an access control problem. The question is what the platform does with that evidence. Does the full response go to the model? Selected fields? Are personal details replaced first? Is only a summary passed along? Those choices decide what the model processes. A scanner touching a record tells you nothing about whether that record entered a model request.
Three groups of data to account for:
| Data group | Examples | What to establish |
|---|---|---|
| What you supply | Target URLs, API specs, scope instructions, test accounts | Which inputs enter prompts, and which are handled only by tools |
| What testing returns | Response bodies, headers, error traces, screenshots, downloaded files | What is selected, filtered, or masked before the model sees it |
| What the platform generates | Findings, evidence, summaries, reports | Whether these are reused in later model requests or stored as agent memory |
Credentials are their own question, and a vendor’s answer here separates the strong from the weak. Requiring approval before a credential gets used is a workflow control, and it’s worth having. The model never receiving the credential at all is an architectural one. They sound similar in a sales call and they are not the same guarantee.
But credential handling and response filtering are separate controls, and vendors conflate them. A token can appear in an application response or in tool output without ever passing through the credential store. Ask how those paths are handled, including screenshots and downloaded files if the platform captures them.
“Sensitive data is masked” needs detail behind it. Which fields? Before or after transmission? What happens when content can’t be inspected? Summaries need scrutiny too, since a summary can repeat the sensitive content of the response it summarizes.
Then push past access to authority, because that’s where the real risk sits. An agent that reads your findings is a small decision. An agent that can close a finding, change a severity rating, or use a stored credential is a different one. Ask for the specific list of actions the agent takes without approval. “There’s always a human in the loop” is not that list, and until you have it you don’t know where the loop sits.
The single best thing to ask for: a sanitized example showing a tool result next to the model request built from it. That makes the conversation concrete without anyone exposing customer data.
The non-answers to watch for: a compliance badge offered in place of an architecture, or any sentence containing “industry-leading AI.” Neither tells you where your data went.
The agent never receives credentials in plaintext. It works with a reference, and the value is decrypted on our side at the point it’s used, then masked in output. Guardrails also run detection and masking for personal data on agent input, agent output, and tool arguments.
What the platform processes during an engagement: security findings and vulnerabilities, asset inventory, engagement and assessment data, and output from connected integrations such as AWS or GitHub. It processes that data when a user invokes it, not on a schedule of its own.
On the response bodies themselves, the answer is blunt. Captured responses reach the model in full rather than as a filtered subset or a summary. That is what makes the reasoning work, because a model triaging an access-control finding needs the response that proves it. The design assumption is that the model reads what the tooling returned.
Guardrails run alongside that, covering personal-data detection and masking, prompt-injection detection, dangerous-command blocking, and system-prompt-leak protection across agent input, agent output, and tool arguments. Treat them as a control on what gets acted on and surfaced, not as a filter that decides what the model reads.
So if your test data is cloned from production, those records reach the model. Where that is not acceptable, the answer is to scope the engagement to non-production data rather than to rely on filtering.
On authority, here’s the list.
| Action | Human approval required |
|---|---|
| Adding or using credentials | Yes |
| Modifying finding status | Yes |
| Creating tickets | Yes |
| Changing severity levels | Yes |
| Reading and analyzing data | No |
| Generating reports | No |
Reading and writing up are autonomous. Anything that touches credentials or alters your record of risk stops and waits for a person.
Read that table for what it is. Every row is an administrative action inside the platform. It says nothing about execution: which hosts the agent may reach, what commands it may run, what leaves your environment. Those limits are enforced separately, and they are the ones that matter if a target ever succeeds at redirecting an agent. Ask any vendor for both, and do not let an approval table stand in for scope enforcement.
One boundary to be clear about. AI features are available to any authenticated user inside a tenant, so the control on who can invoke an agent is your existing user provisioning rather than a separate AI permission layer. Tenant administrators can disable AI features entirely. If you need role-level restriction on who can run agents, raise it during evaluation rather than assuming it.
What to ask: is our data used to train or fine-tune any model, and who does that commitment actually bind?
“We don’t train on your data” is a useful commitment. Read it closely enough to know who “we” includes and which uses it rules out.
Three different things get blurred together here. Training the foundation model is one. Fine-tuning a private copy on your data is another, and some vendors do this deliberately and present it as a feature, which it may be for you, but it means your findings shaped a model artifact that outlives your subscription. Retaining prompts for internal evaluation or quality review is a third, and it’s the one that hides in a privacy policy rather than a sales deck.
Ask about all three separately. Then ask who holds the contractual commitment. If a vendor reaches a model through a hosting provider, the guarantee comes from that provider’s terms, not from the vendor’s intentions, and they should be able to name the document and hand it to you. A vendor who says “we’d never do that” without pointing at a term of service is telling you what they intend, not what they’re bound to.
Ask about exceptions specifically. If a commitment contains “except where authorized,” find out what triggers that and who can authorize it.
Training and retention are also different questions, and a vendor answering one as though it settles the other is worth noticing. A provider can process your data without training on it. A platform can store a conversation so an engagement resumes tomorrow without that conversation touching model weights. AWS documents model-dependent retention modes for Bedrock. Zero retention exists, but it has to be set explicitly; the default mode may allow retention for safety and abuse prevention, and AWS reviews content for models whose provider requires human review. Which applies depends on the specific model and endpoint, so a vendor’s blanket retention claim should be checked against the models actually enabled in your tenant.
Ask for the written training commitment, the provider settings that apply to your configuration, and the exceptions. One privacy statement usually blurs all three.
Strobes does not train, fine-tune, or build any model using customer data. No custom models are built from customer conversations or engagement data.
The commitment on the provider side sits in AWS Bedrock’s data protection terms: customer prompts and completions are not used to train or improve the underlying foundation models, and are not shared with the model provider. AWS operates Bedrock model deployment accounts and model providers cannot access them, which is why the model vendor’s name doesn’t by itself tell you who can see a request. Bedrock also applies logical isolation between customers. Read that as a control with a documented scope rather than an unconditional guarantee, and check AWS’s own terms for that scope.
One scoping note, and it applies to any vendor who supports customer-supplied models. The Bedrock commitments above cover the built-in catalog. If you register your own endpoint, inference runs on infrastructure you control or contract for, and the applicable terms are your provider’s rather than AWS’s. Our own commitment holds either way: we don’t train, fine-tune, or build models on your data regardless of which endpoint serves the request.
What to ask: which region does inference actually run in, and where does anything persisted end up?
Residency is where vendor answers get slippery, because there are two separate facts and vendors often give you only the easier one. Where your data is stored is usually clear. Where a model processes a request is frequently not.
This isn’t hypothetical. AWS documents that Bedrock global cross-region inference can route requests to supported commercial AWS regions worldwide. So a platform that stores your data in one region may still send inference requests elsewhere. “Hosted in the US” and “hosted in India” are storage statements, not processing statements.
Ask both explicitly. Next, how is the region chosen? If it’s set when your tenant is created, you know where you stand. If it’s a default that can shift, find out what triggers the shift.
Then ask which kind of inference profile the platform uses, because the two behave very differently and a vendor may not volunteer the difference. A geographic profile keeps requests inside its geography, so a request made in the US is served from US regions. A global profile does not. It routes to supported commercial AWS regions worldwide, and the destination list can grow as AWS adds regions. AWS also documents that a request can be routed to a destination region your own account never opted into. If residency matters to you, “we use AWS Bedrock in your region” is not the answer you need. Ask for the profile type.
Ask about fallback too. If the selected model is unavailable, does the platform stop, switch models, or switch regions? Any permitted alternate route belongs in what you approve.
A customer-hosted model changes the picture again, and not always as much as it appears. It may put inference inside your environment while the platform still stores findings, evidence, and execution records elsewhere. Follow the whole path before concluding your data stays put.
Also ask what persists between sessions. Agent memory, conversation history, cached context, and stored embeddings all live somewhere, and they may not be covered by whatever you were told about findings storage.
Ask for a deployment-specific data flow diagram showing processing countries, storage locations, which company operates each service, and permitted fallback routes. A cloud logo and a region label aren’t enough to evaluate.
For regulated environments, get it in writing. Residency is one of the few areas where a verbal assurance is worth nothing to your auditor.
Strobes runs in two regions, the United States and India. The region is configured when your tenant is created, not selected per run. All data is encrypted at rest with AES-256 and in transit with TLS 1.2 or above. Tenant isolation runs through a checkpoint architecture, which keeps conversation state and agent memory separated per tenant. Customer data is never shared across tenants.
On processing location, the honest answer has two parts. Built-in models are invoked through AWS Bedrock, and a request on the standard configuration may be served from a different AWS region than the one your data is stored in. Storage stays where your tenant was provisioned; the inference call may not.
India customers who need processing to stay in India can request India-exclusive models through support. That is a paid option rather than the default, so raise it during evaluation rather than after contracting if data residency is a requirement for you.
Conversation history is stored for continuity, so an engagement can be picked up where it was left. Which raises the next question.
What to ask: what’s retained after an engagement closes, and what happens to all of it if we leave?
The test finishes, you download the report, your team revokes the test account. That ends the account’s access. It doesn’t tell you what copies remain.
A real answer is a table with data types, retention periods, and deletion triggers. A vague answer is a sentence about retaining data only as long as necessary. Ask for the breakdown rather than one period covering “AI data.”
| Category | What the answer should cover |
|---|---|
| Findings and evidence | How long reports, response captures, screenshots, and attachments survive |
| Conversations and agent memory | Whether deleting a chat also removes checkpoints, summaries, and retrieval records |
| Temporary execution data | What ends a session, and when workspaces and artifacts are removed |
| Prompt caches and model requests | Which service holds them and which retention settings apply |
| Logs and diagnostics | Whether they contain request or response content, and when they expire |
| Backups and exported copies | Which deletion rules apply, and who controls the destination |
Logs deserve an explicit question. AWS offers optional model invocation logging that captures supported request and response content into CloudWatch Logs or S3, disabled by default. Whether that second copy of your data exists depends entirely on how the vendor configured it. This is why a statement about the inference provider’s retention doesn’t settle retention across the product.
Deletion also means several different things. Deleting a conversation, closing an engagement, and terminating an account can trigger different processes. Ask which action removes which records, what the deadline is, and how exceptions get communicated.
Backups need a straight answer. If a deleted record survives in a backup until expiry, find out the retention period, who can access it, and how deletion is reapplied if that backup is ever restored. And for findings exported into your own ticketing system, confirm who owns removing those copies, because it’s probably you.
Last, ask whether AI features can be switched off entirely. If not, that’s a real constraint on how you deploy.
| Data type | Retention | Deletion trigger |
|---|---|---|
| AI conversation history | Duration of subscription | User deletion or subscription termination |
| Checkpoint and memory data | Duration of subscription | Subscription termination |
| Ephemeral execution data | Session only | Automatic purge after session |
| Code interpreter artifacts | Session only | Automatic purge after session |
On termination, associated AI data is permanently deleted within 30 days, subject to two exceptions we would rather state here than a paragraph later. A legal or regulatory retention obligation can require us to keep specific records. And data sitting in business continuity backups that aren’t reasonably accessible stays until those backups expire; where that happens we tell you rather than claim it’s gone, and the data stays confidential and unprocessed. You can request earlier deletion in writing at any point, and we act on it promptly within those same limits.
Users can delete their own AI conversation history at any time. Tenant administrators can disable AI features entirely for the organization.
What to ask: the agent reads content from the target it’s testing. What stops that content from telling the agent what to do?
It is also the question most often missing from a security questionnaire, which is why you have to raise it yourself.
An agentic pentest works by reading what your application returns. HTTP responses, error messages, page content, API payloads. All of it is untrusted input, and a model is making decisions based on it. Untrusted input is the entry point rather than the attack itself; the attack is an attempt to plant instructions inside that content and redirect the agent. That is indirect prompt injection, and the stakes are higher here than in a chatbot, because the agent holds credential references and can invoke tooling. A crafted response that convinces the agent to run something outside scope, or to dismiss a real finding, is a meaningful attack path.
This isn’t speculative. Prompt injection is the top-ranked risk in the OWASP Top 10 for LLM Applications 2026, and that edition is explicit that malicious instructions don’t have to come from a user: they can be hidden in websites, documents, emails, images, audio, tool output, or data retrieved through RAG. Tool output is exactly what an agentic pentest consumes.
Ask what inspects agent input before the model acts on it. Ask whether dangerous commands are blocked independently of the model’s own judgment, because a model asked to police itself is not a control. And ask whether any of it can be turned off, since a vendor who lets you disable injection defenses has built a configuration option where a control should be.
Then ask about scope, which is a different control from guardrails and often gets discussed as though it were the same one. Guardrails filter content. Scope decides which hosts and paths an agent can reach and which commands it can run at all. Ask how scope is set for an engagement, whether it is configured per run or inherited from a default, and who can change it. Ask at which layer it is applied, since a scope the agent is told to respect and a scope enforced by the environment around it are not equivalent, and only one of them holds if a model is successfully redirected.
A vendor who says the model is instructed to ignore injection attempts has confused a prompt for a control.
Guardrails run on agent input, agent output, and tool arguments, not just on the prompt. They cover prompt injection detection, PII detection and masking, dangerous command blocking, system prompt leak protection, and content policy checks.
Security-critical guardrails are always on and cannot be disabled by an organization. The rest are configurable, so you can tune what fits your environment without switching off the controls that keep a run inside its boundaries.
Engagement scope is configured per sandbox rather than being something the agent decides for itself. Each run operates inside the boundary you set when the engagement is scoped.
The approval gates in section 1 sit downstream of these guardrails, so an instruction that survived them still cannot change a finding or use a credential without a person. Those gates cover administrative actions rather than execution, which is why scope configuration and guardrails are the two things to ask us about rather than the approval table.
What to ask: after a run, can you reconstruct exactly what the agent did?
Approval gates tell you what the agent needed permission for. They don’t tell you what it did with the permission it had. One is preventive, one is forensic, and you need both to answer a question after the fact.
Ask what’s recorded per run and at what granularity. Which model handled it, the steps taken, every tool invoked, the arguments passed. Ask whether the log is queryable or merely retained, and whether it’s exportable, because an audit trail you can’t get out of the platform is an audit trail you can’t show anyone.
Then the harder one. Can a finding be traced back to the evidence that produced it? A conclusion you can’t tie to captured output is a conclusion you can’t defend to an auditor, a developer, or a customer.
Each run carries a full audit trail, recorded in detail: the model used, every step taken, every tool invocation with its arguments, token counts, and cost. Nothing about a run is reconstructed after the fact from summaries.
Findings are tied to the tool output and artifacts that produced them, so any conclusion can be independently verified rather than taken on the model’s word. That’s also why human validation is required before a finding counts as confirmed.
What to ask: if something goes wrong with the AI system, when do we hear about it and from whom?
Ask whether AI systems are covered by the vendor’s general incident response process or treated as an exception. Ask what counts as an incident, since an agent behaving unexpectedly is not the same thing as a data breach and a vendor should have thought about both. Last, what triggers notification and who owns it.
AI and ML systems are covered by Strobes’ documented incident management process, which addresses unauthorized access, misuse, data breach, and compromise of AI infrastructure. Where an incident involves personal data, notification follows applicable data protection law and the terms of your agreement. Response procedures are periodically reviewed and tested.
You can report a suspected incident involving AI systems to security@strobes.co.
Agentic testing has real limits, and a vendor that won’t name them is a vendor you’ll discover them from later.
Output is non-deterministic. Two runs against the same target can investigate in different orders and describe the same finding in different words. Tool results can differ between runs too, since application state, timing, authentication, configuration, and network conditions all change what a check returns.
Models can assert things that are plausible and wrong. That’s why every finding is tied to captured tool output and requires human validation before it counts as confirmed. AI output is decision support, not a verdict.
Training cutoffs mean the model doesn’t natively know the newest CVEs or techniques. Current threat data reaches the agent through live tooling, connectors, and feeds rather than from model memory.
Reasoning quality is strongest in English and on mainstream technology stacks. Bespoke or legacy systems benefit from more operator guidance.
And agentic testing broadens and accelerates coverage, but it does not replace a scoped manual penetration test where you need contractual or regulatory assurance. If your auditor requires a human-led test, you still need one.
Agentic pentesting data security comes down to four things you can check rather than take on trust: what reaches the model, which services process it and where, what uses are permitted, and what remains afterward. A useful vendor answer is specific enough to compare against the deployment you’re actually buying.
| Question | Evidence to ask for |
|---|---|
| What reaches the model? | A sanitized tool output shown beside the model request built from it |
| Who processes it, and where? | A deployment-specific data flow diagram with permitted routing and fallback |
| Can it be used for training or accessed later? | Written commitments, the provider settings that apply, and the exceptions |
| What remains, and how is it deleted? | A retention schedule with triggers, deadlines, and backup handling |
| Can the target manipulate the agent? | The guardrails that inspect agent input, and which ones can be disabled |
| Can you reconstruct a run? | A sample audit trail, and whether it exports |
Documentation should match product settings. Revisit the review when a deployment adds a provider, changes routing, or enables a feature that changes data handling.
If you’re evaluating Strobes, bring these to the technical review and ask for the supporting documentation for your proposed configuration. You should leave with answers you can take back to your team.
Request the SOC 2 Type II report, the Data Processing Agreement, and completed security questionnaires in SIG or CAIQ format at security@strobes.co, subject line “AI compliance documentation request.”
