Is Human Approval Real If It Exists Only in a Prompt?

The World Economic Forum’s AI Agents in Action: Foundations for Evaluation and Governance report makes a crucial point: the appropriate level of human oversight should reflect an AI agent’s autonomy, authority and operational context.

That distinction matters because not every AI action carries the same consequence. Summarizing a document is not equivalent to sending an email. Finding a customer record is not equivalent to changing it. Drafting a payment instruction is not equivalent to executing the transfer.

As AI systems move from generating content to taking action, oversight must move with them.

In high-risk or less predictable settings, a human-in-the-loop (HITL) configuration ensures that agents can suggest or prepare actions, but final decisions remain subject to explicit human approval.
In more stable or clearly defined environments, a human-on-the-loop (HOTL) configuration allows agents to act within defined boundaries, while humans monitor behaviour, receive alerts and retain the ability to intervene or override when necessary.

Integrating these oversight models into governance structures helps maintain accountability and human judgment as agents operate with greater independence and scale. - (Quoted from AI Agents in Action: Foundations for Evaluation and Governance, WEF Whitepaper November 2025)

A prompt can guide behaviour. It cannot enforce authority.

The instinctive response to agent risk is often to strengthen the system prompt:

Ask the user before taking an important action.

That instruction may improve behaviour. But it is not an approval control.

A language model interprets instructions probabilistically. Its behaviour can vary with context, competing instructions, model updates and information retrieved from external systems. In a tool-enabled agent, that external information may itself contain instructions capable of redirecting the model.

The AgentDojo benchmark illustrates the problem. Researchers tested agents operating across workplace, banking, travel and communication environments. Instructions hidden in content encountered by the agents could induce unintended tool use. Attack success varied substantially across tasks, but reached particularly high levels in some configurations.

This is the architectural weakness: the model processes trusted instructions and untrusted content through the same reasoning mechanism. It should not also be the final authority deciding whether a consequential action is permitted to proceed.

A system prompt can tell an agent to seek approval. Only the surrounding application can make approval unavoidable.

Proposal and authorization are different functions

A useful governance boundary separates two decisions:

1. What action does the agent propose?
2. Is that action authorized to execute?

The model may perform the first function. It should not control the second when the action creates a material external effect.

Consider an agent preparing an email. The model can determine that sending a message would help complete the user’s task. It can draft the recipients, subject and body. But the actual send operation can remain technically suspended until the user sees those details and approves them.

The same principle applies when an agent proposes to:

• create or modify a customer record;
• publish content;
• disclose sensitive information;
• change permissions;
• delete data;
• initiate a financial transaction;
• make or communicate a decision affecting another person.

At the approval boundary, the model’s output becomes a proposal—not an instruction to the system.

This distinction is consistent with OWASP’s guidance on prompt injection, which recommends enforcing least privilege in code and requiring human approval for high-risk operations. The control belongs in the application layer, where permissions and execution can be constrained independently of the model.

Effective oversight requires more than an “Approve” button

A poorly designed approval step can be almost as weak as the prompt it replaces.

If the person cannot see what will happen, approval is not meaningfully informed. If any user can approve, it is not an authorization control. If the agent can change the action after approval, the approval does not bind the execution. If a timeout or technical failure allows the action to continue, the system does not fail safely.

A credible approval control should therefore answer five questions:

• What triggers approval? The system, not the model, classifies which operations require intervention.
• Who may approve? The approver’s identity and authority are verified.
• What can the approver see? The material parameters of the proposed action are presented clearly.
• What exactly is approved? The approval is bound to the displayed recipient, record, amount, fields or payload.
• What evidence remains? The proposal, decision, approver, time and resulting execution are recorded.

This is the difference between nominal human involvement and effective human oversight.

The distinction also appears in law and risk-management practice. Article 14 of the EU AI Act requires high-risk AI systems to be designed so that natural persons can effectively oversee them, including the ability to disregard, override, reverse or interrupt outputs where appropriate. The requirement applies specifically to high-risk systems, but its broader design lesson is valuable: oversight must create a real ability to intervene.

Similarly, the NIST AI Risk Management Framework calls for human-oversight processes to be defined, assessed and documented. An organization should be able to demonstrate how intervention works, not merely state that a human remains responsible.

Oversight should be proportional, not universal

None of this means that a person should approve every agent action.

Blanket approval creates friction and, eventually, approval fatigue. When people repeatedly encounter low-value confirmation requests, they learn to click through them. Human involvement then becomes ceremonial rather than protective.

The stronger approach is graduated autonomy:

• low-risk, reversible actions may proceed automatically;
• bounded actions may proceed within predefined permissions;
• consequential or externally facing actions require approval;
• prohibited actions remain unavailable regardless of approval.

This makes autonomy a governed property of the system. It can vary according to the action, data, user, environment and potential impact.

Human approval must also sit alongside other safeguards: least-privilege access, tool and data allowlists, trusted identity, parameter validation, rate limits, monitoring and audit trails. Approval is a critical control, but it is not a substitute for a secure architecture.

Building the boundary into ZeroH

This is the approach we are developing in ZeroH.

When the ZeroH sales agent proposes to send an email or create or modify a CRM record, the action pauses before execution. The person in the conversation receives an approval card showing the action’s material details. Approval or denial is returned to the suspended workflow; denial, expiration or inability to complete the approval means the action does not proceed.

The significance is not the presence of a button. It is where the boundary sits.

The model may propose the action, but it cannot approve its own proposal. The application holds the execution, verifies the responding user and resumes the workflow only after an affirmative decision. The control operates even if the model’s instructions are misunderstood or manipulated.

As agents acquire greater access to enterprise systems, this distinction will become foundational. Organizations will need to govern not only what models say, but what agents are permitted to do and who must authorize the moment a recommendation becomes an action.

A sentence in a prompt expresses an intention. A control determines what the system can actually do.