Is an agent security audit a penetration test?

No. A penetration test is somebody attempting to break a running system with permission to do so, and it produces findings about that system as it stood on that day. An audit of this kind is an examination of design, permissions and recovery. The two answer different questions and a company with real exposure usually wants both, in that order: fix what you already know about before paying somebody to find what you do not.

Whoever can see both the agent's configuration and the systems it reaches, which is usually one engineer and one person who owns the affected process. Running it inside the team that built the agent produces accurate answers about the code and optimistic ones about the recovery, because the recovery lives in somebody else's systems.

When what the agent can reach changes, not on a calendar. A quarterly review of an agent nobody has touched tells you what you already knew, and an agent that gained a connector last week is a different system from the one you reviewed.

What is an AI agent security audit?

An AI agent security audit examines what an AI agent is permitted to reach, what it is able to cause, and what happens when it acts wrongly. It covers the connectors and credentials the agent holds, the third-party components it loads, the rules that decide which actions are allowed, whether a run can be stopped while it is happening, whether its effects can be put back, and whether anybody can say afterwards who authorized what it did. It is not a review of the model. It is a review of the authority the model was given.

It is not the same as auditing the model

Most published work on AI security is about the model layer: whether somebody can make a model misbehave by writing something clever into a prompt, a document or a tool result. That work is real and it matters. It also answers a different question from this one.

An agent does not have to be attacked to do damage. A correct agent working from a stale record, a retry loop that keeps going, a connector scoped by whoever clicked through the consent screen, and a successful prompt injection all produce the same morning. The cause differs and the recovery does not. So an agent audit is indifferent to why the agent was wrong, and asks instead how far the effect reaches and what it costs to get back.

That is why the two reviews find different things. A model review can pass a system that will empty a table on a bad Tuesday, and an agent review can pass a system that is trivially talked into saying something embarrassing. Neither substitutes for the other, and a company running agents against real systems generally needs both.

The three moments

The useful way to lay out agent controls is by when they act, because a company that has one of them usually believes it has all three.

Before is authorization: which actions are allowed, in which circumstances, and which of them a person has to approve. It is the most written-about layer and it is the weakest one on its own, because approval happens at the moment when nobody knows anything is wrong yet.

During is the stop: whether you can halt a run that is already going. This is the layer teams are least likely to have, and it is the only one that operates in the window where you know something is wrong and it is still happening. Pulling a credential is not a stop. It is breaking something else in the hope that the agent notices.

After is reversal and attribution: whether the effects can be put back, and whether anybody can say who authorized them. Both are usually discovered to be missing during the first incident, which is the worst possible time to find out that the rollback has never been run.

What an audit of this kind cannot tell you

It cannot tell you whether the agent is good at its job. Reach, stopping and recovery are questions about consequence, and a useless agent with a tested stop scores well on all of them.

It also cannot tell you that nothing will go wrong. Everything on this page is about the size and duration of a mistake, not its likelihood. That is a deliberate narrowing: likelihood is a forecast and the size of the blast radius is a fact about your configuration, which means one of them is checkable today and the other is not.

Common questions

Is an agent security audit a penetration test?: No. A penetration test is somebody attempting to break a running system with permission to do so, and it produces findings about that system as it stood on that day. An audit of this kind is an examination of design, permissions and recovery. The two answer different questions and a company with real exposure usually wants both, in that order: fix what you already know about before paying somebody to find what you do not.
Who should run one?: Whoever can see both the agent's configuration and the systems it reaches, which is usually one engineer and one person who owns the affected process. Running it inside the team that built the agent produces accurate answers about the code and optimistic ones about the recovery, because the recovery lives in somebody else's systems.
How often?: When what the agent can reach changes, not on a calendar. A quarterly review of an agent nobody has touched tells you what you already knew, and an agent that gained a connector last week is a different system from the one you reviewed.

Find out where you stand

The assessment on this site asks up to 24 questions about what your agent can reach and what happens when it is wrong, and returns a score, a verdict and, where a control is short, the one thing to fix first. It takes about ten minutes, and questions your agent has no surface for are dropped rather than scored against you.

It is a self-assessment of design, permissions and recovery, not a penetration test, not an audit of a running system, and not a certification.

Run the assessment