The Missing Governance Control: Retiring Outdated AI Knowledge

Keeping AI knowledge current requires more than adding new information. Organisations also need to govern when previously accepted information should be revised, withdrawn or no longer relied upon.

In REALTIME QA, a benchmark examining how language models answer questions about current events, researchers found that GPT-3 could often update its answers when retrieval supplied newer documents. However, when the retrieved information was insufficient, the model tended to return outdated answers.

The finding highlights a challenge for enterprise AI: access to current information does not necessarily prevent older information from influencing an answer.

Consider an AI assistant that helps employees interpret procurement policies. An organisation lowers the spending threshold requiring additional approval, but the previous policy remains accessible in its document archive.

Which threshold will the assistant apply? Can the organisation establish which version informed the answer? And what prevents the assistant from continuing to recommend an outdated approval process?

As AI systems increasingly inform decisions and execute tasks, these become questions of operational governance.

Why updating knowledge is difficult

AI systems acquire information through different mechanisms. Language models learn factual associations during training, while retrieval-augmented generation (RAG) systems supplement those models with external information.

Both present challenges when facts change.

Researchers behind FreshLLMs found that evaluated language models struggled with rapidly changing information and questions containing false premises. Access to current search information improved performance, but retrieval alone did not eliminate the underlying problem.

Knowledge editing, which attempts to modify specific facts learned by a model, presents another difficulty. The RippleEdits study found that evaluated editing methods struggled to propagate factual changes consistently to related knowledge.

For enterprises, the distinction matters. Removing an outdated document from a retrieval index does not erase information learned during model training or retract earlier answers.

However, organisations can govern the external knowledge sources and derived statements supplied to their AI systems.

This introduces a different concept: knowledge retirement.

Rather than attempting to erase every outdated fact from a model, knowledge retirement governs when information should cease to be treated as authoritative for a particular purpose.

When sources change, authority matters

A knowledge base may contain several versions of a policy, conflicting regulatory interpretations or summaries derived from superseded documents.

The challenge extends beyond identifying the latest document.

An older contractual provision may remain applicable to an existing agreement. A historical regulation may still be relevant to conduct occurring before an amendment. A revised source may leave the particular statement being relied upon unchanged.

Determining whether information remains valid therefore requires attention to its source, version, context and applicable period.

This is familiar in legal research. Establishing that a provision exists is only the beginning. A researcher must also determine whether it remains in force, has been amended or has been affected by subsequent authority.

AI knowledge management requires a comparable discipline when its outputs inform consequential decisions.

What existing frameworks tell us

Regulatory and risk-management frameworks already address aspects of information accuracy and lifecycle governance, although they do not prescribe a general mechanism for retiring outdated AI knowledge.

The GDPR's accuracy principle, Article 5(1)(d), requires personal data to be accurate and, where necessary, kept up to date. This obligation concerns personal data rather than every factual statement generated by AI.

For high-risk systems, Article 10 of the EU AI Act establishes requirements for training, validation and testing data governance, while Article 12 addresses automatic event logging and traceability. Neither provision directly mandates claim-level knowledge retirement.

The Monetary Authority of Singapore's 2024 AI Model Risk Management information paper also discusses model staleness arising from changes in data or model behaviour and identifies retraining, redevelopment and decommissioning among possible responses.The NIST AI Risk Management Framework provides a broader structure for governing, identifying, assessing and managing AI risks throughout the system lifecycle.

Together, these sources reinforce the importance of managing information and AI-system reliability over time.

Making knowledge retirement a governance control

A practical knowledge-retirement process needs to establish five elements:

1. Provenance: Which source and passage support the statement?
2. Version: Which revision of the source was used?
3. Validity: Under what conditions or during which period does the statement apply?
4. Change signals: What would trigger reassessment, such as amendment, withdrawal or conflicting evidence?
5. Retirement decision: Should the statement be retained, revised, suspended or withdrawn, who authorised that decision, and when?

These elements form a proposed governance approach rather than a prescribed regulatory checklist.

For the control to operate, the system must detect relevant changes, evaluate their effect on dependent claims and enforce the resulting decision.

A source amendment should trigger reassessment of affected statements rather than automatically invalidating everything derived from the earlier version. Conflicting evidence may require escalation to a policy owner, legal specialist or another authorised decision-maker.

Returning to the procurement example, the organisation should be able to establish which approval threshold is currently authoritative and prevent a superseded threshold from governing new recommendations or actions.

Knowledge validity becomes part of the conditions under which an AI system should be allowed to act.

From knowledge to action: the ZeroH connection

Blade Labs is exploring these principles through evidence-bound knowledge tooling that connects generated statements to supporting quotations and source revisions, evaluates evidentiary support, and provides mechanisms for handling unsupported claims and withdrawn sources.

For governed AI agents, the accountability question extends further.

Organisations also need to understand which information informed an agent's proposed action, what authorization or approval governed it, and what occurred during execution.

This is where ZeroH's approach to runtime governance becomes relevant. Its work on authorization, human approval and traceability addresses the execution side of accountability, complementing the governance of the information on which decisions depend.

These are related capabilities, and their end-to-end integration must be established through implementation and testing.

Knowledge retirement also has limits. Provenance cannot establish that a source is substantively correct, and removing information from a knowledge base does not guarantee that a trained model has forgotten it. Human judgment remains essential where authority, interpretation or competing evidence is disputed.

Knowing when knowledge no longer holds

As AI systems become embedded in enterprise workflows, retrieving current information is only part of maintaining reliable knowledge.

Organisations also need to identify when information loses its authority, determine which statements and decisions are affected, and enforce an appropriate response.

The critical question is whether an organisation can establish when its AI system should stop relying on information it previously treated as valid—and what happened when that information changed.

Explore ZeroH and its approach to governed AI execution and traceability.

Frequently asked questions

What does it mean for an AI system to forget information?

The term can refer to different processes. Machine unlearning seeks to reduce or remove the influence of particular training data on a model. Knowledge editing attempts to change learned factual associations. Knowledge retirement governs whether external sources and derived statements should remain available or authoritative for future use.

Does deleting a document from a vector database make an AI model forget it?

No. Removing a document from a retrieval index can prevent that document from being retrieved through that index. It does not change what the model previously learned during training or retract historical outputs. Other copies, caches or indexes may also need to be addressed.

Does the EU AI Act require AI systems to delete outdated knowledge?

The EU AI Act does not establish a general requirement to delete outdated factual knowledge from model parameters. It does impose data-governance, documentation and logging obligations on certain high-risk AI systems, subject to the scope and conditions of the relevant provisions.

How should an organisation respond when two knowledge sources conflict?

The response should depend on source authority, applicable context, the nature of the conflict and the consequences of relying on incorrect information. A governed process should identify the conflict, determine who can resolve it, record the decision and restrict reliance on disputed information where the risk warrants it.

References

1. Kasai, J., et al. (2022). REALTIME QA: What's the Answer Right Now? arXiv. https://arxiv.org/abs/2207.13332
2. Vu, T., et al. (2024). FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. Findings of the Association for Computational Linguistics: ACL 2024. https://aclanthology.org/2024.findings-acl.813/
3. Cohen, R., et al. (2023). RippleEdits: A Benchmark for Understanding the Ripple Effects of Knowledge Editing. arXiv. https://arxiv.org/abs/2307.12976
4. European Parliament and Council of the European Union. (2016). Regulation (EU) 2016/679 (General Data Protection Regulation), Article 5(1)(d). EUR-Lex. https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng
5. European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 10 and 12. EUR-Lex. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
6. Monetary Authority of Singapore. (2024). Artificial Intelligence Model Risk Management. Information Paper. https://www.mas.gov.sg/-/media/mas-media-library/publications/monographs-or-information-paper/imd/2024/information-paper-on-ai-risk-management-final.pdf
7. National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. https://doi.org/10.6028/NIST.AI.100-1