When AI Checks AI: Where Does Human Oversight Begin?

Image Source : Air Canada found liable for chatbot's bad advice on plane tickets

AI can verify evidence, assess whether it supports a claim, and help identify mistakes. But these are different kinds of checks, and each has its limits. Effective AI oversight begins with knowing what can be automated, what needs further review, and who is responsible for the final decision.

In February 2024, a Canadian tribunal ruled against Air Canada in a dispute involving incorrect advice from its chatbot.

The chatbot had told a customer that they could apply for bereavement fares after completing their travel. It even provided a link to Air Canada's official bereavement policy.

There was one problem: the policy explicitly excluded requests made after travel had been completed.

The chatbot had linked to a genuine source, but its answer contradicted that source.

In Moffatt v. Air Canada, the British Columbia Civil Resolution Tribunal found that the airline had failed to take reasonable care to ensure its chatbot was accurate.

The case illustrates an important distinction in AI assurance: verifying that evidence exists is not the same as verifying that the evidence supports a conclusion.

As organizations increasingly use AI to interpret documents, answer questions and support decisions, this distinction becomes critical.

The challenge is not simply deciding whether humans or AI should perform the checks. It is determining which checks can be automated reliably, where AI evaluation is sufficient, and when a person needs to make the decision.

What fixed rules can verify

Some aspects of an AI-generated answer can be checked without asking another AI model to evaluate it.

Software following fixed rules can verify whether a cited document exists, whether a quotation appears in the recorded source, whether the document matches a stored version, and whether an approval has been recorded.

These checks can be repeated automatically across large numbers of outputs.

For example, a system could confirm that an AI-generated statement contains an exact quotation from a particular version of a regulatory document.

But that confirmation has limits.

The quotation might be accurate while the interpretation is wrong. An approval might be recorded without being valid. A source might match a stored copy without being the latest applicable version.

Fixed rules can establish specific facts against the records available to the system. They cannot, by themselves, guarantee that those records are complete, current or correct.

In the Air Canada case, a check confirming that the linked policy page existed would have passed. It would not have established that the chatbot's advice was accurate.

That requires a different kind of evaluation.

When AI can evaluate the evidence

Determining whether a source supports a statement often requires understanding its meaning.

AI models can help perform this work.

The paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, found that strong AI judges achieved over 80% agreement with human preferences when evaluating chatbot answers. The researchers also identified weaknesses, including biases towards the position and length of responses, as well as limitations in reasoning. The finding demonstrates both the potential and the limits of using AI to evaluate other AI systems.

An AI judge may be capable of assessing whether evidence supports a claim without requiring a person to examine every answer. But its reliability cannot be assumed simply because it produces a confident response.

OpenAI's SimpleQA research offers a useful illustration. The benchmark was designed around factual questions with clear answers, yet researchers found that models often expressed more confidence than their accuracy justified.

The study also found that answers repeated more consistently tended to be more accurate. But consistency alone does not establish correctness: an AI model can repeat the same mistake.

For organizations, the practical lesson is that an AI judge should be tested on the actual work it will perform before being trusted to make decisions independently.

For example, a system assessing whether regulatory quotations support compliance statements should be tested against representative examples reviewed by qualified people. Where testing shows that the system performs reliably, and the consequences of an error are limited, it may be reasonable to allow it to make certain judgments automatically.

Its performance must continue to be checked, particularly when the model, instructions or types of cases change.

Where human judgment becomes necessary

Not every decision can be reduced to matching records or assessing the meaning of a passage.

Sources may conflict. A legal provision may have several possible interpretations. The evidence may be incomplete. An incorrect decision may expose an organization or its customers to serious consequences.

Research from Anthropic on automated alignment auditing agents illustrates the limits of automated evaluation. Its agents performed better when identifying clear, recognizable behaviours but encountered difficulties with more subjective problems. The researchers also identified overconfidence and used human review in parts of their auditing process.

These findings do not mean that people always make better decisions than AI. They show why organizations need to understand what their automated checks can reliably establish and where further judgment is required.

The greater the uncertainty or potential harm, the stronger the case for human involvement.

That involvement must also be meaningful.

For high-risk AI systems, the EU AI Act requires deployers to assign human oversight to people with the necessary competence, training and authority. Under the amended application timetable in Regulation (EU) 2026/1744, the relevant requirements are scheduled to apply from December 2027 for Annex III high-risk systems and August 2028 for Annex I high-risk systems.

The broader governance principle is straightforward: involving a person adds little value if that person cannot understand, question or change the AI system's decision.

Human decisions must also be protected

There is another part of AI oversight that receives less attention: what happens after a person has reviewed and approved something?

Consider an AI system that generates a compliance statement from an organization's policy documents. An automated check confirms that the quotation matches the source. An AI judge assesses whether the quotation supports the statement. A qualified reviewer then approves the interpretation.

A later automated update processes the same document and rewrites the approved statement. If the system can silently replace the reviewed version, the organization may lose track of what was approved, by whom and on what evidence.

Human oversight should not end when someone clicks approve.

A reviewed decision should be protected against unauthorized changes. If the underlying source changes, the system should identify the affected decision and make it available for reassessment.

Protecting a decision does not mean treating it as permanently correct. New evidence, amended regulations or changed circumstances may require it to be revised.

The distinction matters: automation should not silently undo human judgment, but neither should an earlier approval prevent necessary corrections.

This is a governance approach organizations can adopt when designing their AI workflows, rather than a specific requirement prescribed by the regulations discussed above.

What these checks cannot guarantee

Even when these safeguards work as intended, they do not eliminate the possibility of error.

A quotation may match an unreliable source. An AI judge may misinterpret a passage. Human reviewers may overlook important details or reach incorrect conclusions.

OpenAI's SimpleQA research illustrates that difficulty even in a carefully constructed benchmark: an independent review found disagreements caused both by mistakes in verification and by ambiguous questions or conflicting sources.

Automated checks, AI evaluation and human review therefore serve different purposes. None should be presented as a complete guarantee of accuracy. The goal is to make clear what each check establishes, where its limits lie, and how mistakes can be identified and corrected.

Where ZeroH fits

At Blade Labs, our evidence-bound knowledge tooling provides an example of how these distinctions can be applied. Generated statements are linked to exact quotations from their recorded sources, alongside source revisions and judgments about whether those quotations support the statements.

Checks that do not use an AI model verify quotation matches and recorded source information. Updates involving missing or mismatched quotations, or withdrawn sources, can be held rather than published.

Assessing whether a quotation supports a statement remains a separate task involving AI evaluation and owner review. Reviewed content is also protected from being silently rewritten by automated updates.

ZeroH addresses a related challenge when AI agents move from generating information to taking action. Policies and approval controls govern what agents are permitted to do, and consequential actions can be held for human approval.

Together, these approaches illustrate two important aspects of AI governance: checking the evidence behind an answer and controlling the actions an AI system is authorized to take.

They are related capabilities; their end-to-end integration must be established through implementation and testing.

Deciding where proof ends

For organizations deploying AI, the starting point is not whether every output should be reviewed by a person.

It is asking three questions:

What can we verify automatically? Use fixed rules for checks that have clearly defined answers and reliable records.
What can AI evaluate reliably? Test AI judges against real examples and allow independent decisions only where their performance and the consequences justify it.
What requires a person to decide? Bring in qualified reviewers when evidence is uncertain, interpretations are disputed or mistakes could have serious consequences.

Then ensure those decisions are recorded, protected against unauthorized changes and reconsidered when the evidence changes.

Effective AI oversight requires organizations to demonstrate what was checked, what each check established, and who was accountable for the final decision.

Explore how ZeroH governs AI agent actions through policies, permissions and approval controls.

Frequently Asked Questions

1. What is the difference between verifying an AI-generated answer and checking whether it is correct?

Verification can establish specific facts, such as whether a cited document exists or a quotation matches its source. Determining whether the source actually supports the answer requires evaluating its meaning. A genuine citation does not necessarily make an AI-generated conclusion correct.

2. Can AI evaluate the accuracy of another AI system?

Yes. AI models can assess whether statements are supported by evidence and identify possible errors. However, their reliability must be tested against representative examples reviewed by qualified people. An AI model's confidence in its answer is not, by itself, proof of accuracy.

3. When is human oversight necessary in AI systems?

Human review becomes particularly important when evidence is incomplete, sources conflict, interpretations are disputed, or an incorrect decision could have serious consequences. The appropriate level of oversight depends on the task and the risks involved.

4. Why should human-reviewed AI decisions be protected?

Without safeguards, later automated updates could overwrite previously reviewed decisions without recording what changed or why. Protecting those decisions helps preserve accountability. However, they must remain open to reassessment when new evidence or changed circumstances make that necessary.

5. How does ZeroH support AI governance and oversight?

ZeroH provides policy and approval controls around AI agent actions, helping organizations govern what agents are permitted to do and when human approval is required. Blade Labs' related evidence-bound knowledge tooling supports source verification, review and protection of reviewed content. These capabilities address different parts of AI governance, and their end-to-end integration requires implementation and testing.

References

Civil Resolution Tribunal of British Columbia. (2024). Moffatt v. Air Canada, 2024 BCCRT 149. Tribunal decision.
Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. Research paper.
Wei, J., et al. (2024). Introducing SimpleQA. OpenAI. Research article.
Bricken, T., et al. (2025). Building and Evaluating Alignment Auditing Agents. Anthropic. Research article.
European Parliament and Council. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act). EUR-Lex.