Can a Prompt Prove That an AI Agent Behaved Safely?

Imagine an AI agent with access to customer records and email.

Its prompt contains three instructions:

• disclose only authorized information;
• obtain human approval before sending an email; and
• record every consequential action.

During an evaluation, the agent states that it followed these instructions. The transcript looks reassuring. It may even explain which policy applied and confirm that approval was obtained.

What does that transcript prove?

At most, it shows that the model can produce an answer consistent with the expected behaviour. The missing evidence concerns the surrounding system: whether access was restricted, execution genuinely paused, approval came from an authorized person and a durable audit record was created.

As agents gain the ability to retrieve data, select tools and initiate actions, evaluations need to follow the complete execution path and verify the controls, decisions and actions recorded along the way.

Knowing the rule does not ensure safe action

A 2025 research paper, LM Agents May Fail to Act on Their Own Risk Knowledge, examined the difference between understanding a safety rule and applying it during execution.

The researchers evaluated agents across three progressively demanding questions:

1. Does the agent know that an action is risky?
2. Can it recognize that risk within an execution trajectory?
3. Does it avoid performing the risky action?

The agents achieved pass rates above 98% on questions testing risk knowledge. Performance deteriorated when the same risks appeared within operational scenarios. Pass rates for avoiding risky actions fell below 26%.

These figures relate to the models, scenarios and evaluation design used in the study. They should not be treated as a universal failure rate. The wider finding is nevertheless important: an agent may articulate the correct safety principle and still fail to apply it while acting.

The researchers also tested an independent verifier that reviewed proposed actions before execution. Their system reduced risky-action execution by 55.3% compared with agents relying on prompting alone.

This helps explain why a safety instruction inside the agent’s prompt offers limited assurance. The model participates in interpreting the rule, proposing the action and reporting what happened. Independent controls are needed to evaluate and constrain that conduct.

A control instruction is different from control evidence

Prompts remain useful. They define objectives, provide context and guide behaviour. Their presence should form part of an evaluation record.

Their evidential value has boundaries.

A prompt can instruct an agent to use approved tools. Technical enforcement determines which tools are actually available.

A prompt can require human approval. The execution layer must prevent the action from proceeding until that approval is recorded.

A prompt can tell an agent to respect a user’s permissions. Identity and authorization services must enforce those permissions against the requested data and operation.

A prompt can require an audit trail. The system must create, protect and retain the relevant record.

This distinction is especially important when the agent produces its own account of compliance. A statement such as “approval was obtained” remains model-generated output. Independent evidence must establish who approved the action, what they saw, which operation they authorized and whether execution remained blocked until that decision.

Assurance requires substantiated claims

The UK Department for Science, Innovation and Technology’s Introduction to AI Assurance describes assurance as the use of techniques to measure, evaluate and communicate the trustworthiness of AI systems.

This framing moves governance from broad claims toward demonstrable properties.

An organization may claim that an agent is safe, accountable or subject to human oversight. Assurance asks how that claim was tested, which evidence supports it, what limitations remain and whether another party can scrutinize the result.

Different claims require different evidence. Model testing may help assess the quality of an answer. System testing is needed to examine identity, access, tool execution and approval behaviour. Audit records support reconstruction after the event. Monitoring helps determine whether controls continue to operate once the system changes or enters production.

For an action-taking agent, assurance therefore extends beyond the model. The unit being evaluated is the complete workflow through which the agent receives authority and affects another system.

What should a meaningful evaluation preserve?

A credible evaluation should leave enough evidence for an independent reviewer to reconstruct every material step.

At minimum, the record should answer six questions.

1. What was tested?

The evidence should identify the agent, model, policy set, tools, permissions and scenario. A result from one configuration should not be presented as proof for another.

2. What action was proposed?

The proposed operation should be recorded in structured form, including the selected tool, parameters, affected resource and intended outcome.

“Send an email” is insufficient when the evidence cannot identify the recipient, message operation or authority under which it was requested.

3. Which control made the decision?

The record should show whether the operation was allowed, denied or held for approval, together with the applicable policy and its version.

This establishes that a control produced the outcome, rather than the model merely generating the expected response.

4. Did approval affect execution?

Where approval is required, the evidence should show that the action remained blocked until an authorized reviewer approved, rejected or modified it.

The record should connect the reviewer’s decision to the exact operation that was subsequently released or refused.

5. What was executed?

The final record should connect the authorized operation to the result returned by the tool or external service. This prevents an approval for one action from being reused for another recipient, record or purpose.

6. Can the result withstand verification?

The evaluation process should detect evidence that is missing, altered, duplicated or attached to the wrong run. Otherwise, a seemingly complete record may combine unrelated events or conceal the point at which a control failed.

Why a successful demonstration can create false confidence

A screenshot can show that an approval interface exists. It says little about whether the underlying service enforced the decision.

A successful demonstration proves that one workflow completed under particular conditions. It may not reveal whether:

• prohibited routes remained unavailable;
• an approval could be replayed;
• evidence survived a dependency failure;
• incomplete records caused the evaluation to fail; or
• a change to the model or policy invalidated the earlier result.

The strength of evidence should reflect the agent’s authority and the consequences of failure. A summarization assistant may justify a comparatively light evaluation. An agent that can modify customer records, send external communications or initiate financial activity requires deeper scrutiny.

From expected answers to retained runtime evidence

ZeroH applies ASSERT (Adaptive Spec-driven Scoring for Evaluation and Regression Testing), an open-source evaluation framework, to translate defined behavioural requirements into executable evaluation scenarios and assess the resulting agent behaviour and execution evidence.

Defined scenarios run through a dedicated evaluation environment. Completion depends on retained runtime policy decisions, stored audit acknowledgements and evaluation output. The evidence is bound to the relevant tenant, caller, target, policy set, publication and evaluation run.

Verification checks whether that evidence is complete, unique, unaltered and associated with the correct evaluation. Synthetic approval and denial scenarios can pass through the ordinary interaction path while consequential external actions, such as sending an email, remain suppressed. This allows the control path to be exercised without triggering the external action.

The resulting evidence shows how the system processed a decision and whether the relevant controls operated as intended under the tested conditions. Those conditions are a dedicated evaluation environment; live deployments, their stored evidence and their authenticated user flows are qualified separately. Assurance claims should remain confined to those conditions and be reassessed when material elements of the system or its operating environment change.

A passing evaluation should make a bounded claim

“Safe” and “compliant” are broad conclusions. Their meaning depends on the use case, operating environment, applicable law and consequences.

A stronger evaluation result makes a narrower statement:

Under this identified configuration, these scenarios were executed, these controls produced these decisions, and these retained records support the result.

That claim can be examined, challenged and repeated. It can also expire when the model, permissions, tools, policies or data sources change.

Prompts describe the behaviour an organization expects. Assurance depends on evidence showing how the system behaved.

Before accepting a passing result, organizations should ask:

If the agent’s explanation disappeared, would the retained evidence still prove what the system allowed, prevented and executed?

If the answer is no, the evaluation has demonstrated a convincing response. The operation of the control remains unproven.

Learn how ZeroH approaches runtime governance and verifiable evidence at zeroh.io.