Your AI needs playback—not a blockchain

Ask the owner of a production AI system a simple question: if one of your AI-generated answers is challenged tomorrow, what can you actually prove?

Many organisations can retrieve the answer shown to the user. Far fewer can recover the source passages, document versions, retrieval ranking, prompt, model configuration and controls that shaped it. This is the gap between recording an output and preserving an evidence trail.

Retrieval-augmented generation can give a model access to relevant organisational knowledge. It does not automatically make the resulting answer traceable. The system may retrieve the wrong evidence, misinterpret the right evidence, or simply retain too little information to determine which failure occurred.

That third category—evidence failure—is often the most expensive. A technical defect that might have been corrected quickly becomes an investigation. Teams search through logs, attempt to recreate expired configurations and compare documents that may have changed since the answer was delivered. Verification becomes archaeological.

The useful mental model is answer playback: return to a recorded interaction, inspect it step by step, and separate what happened then from what a new run produces now.

Return to the moment of the answer

Imagine selecting a disputed response and opening the exact interaction that produced it. The interface first shows the answer exactly as the user received it. Material claims are highlighted. Selecting one reveals the supporting passage, its document and version, its retrieval rank, and whether the claim was directly supported or generated as a synthesis.

A timeline exposes the execution step by step: the question, the corpus searched, the candidates retrieved and reranked, the prompt and model configuration, the controls applied, and the final answer delivered. Each event can be opened and inspected.

A historical view shows the state of the knowledge base at that time. Perhaps policy version 3.2 was active when the answer was generated, while version 3.3 is active today. The investigator can recover the original version, see which passages were eligible for retrieval and determine whether the answer was reasonable given the information available then.

This is the central promise of answer playback: not perfect repeatability, but inspectable history. An AI flight recorder is the instrumentation underneath; playback is the experience that makes the record useful to investigators, operators and decision-makers.

Evidence is not the same as replay

The distinction between an original record and a later replay matters. A question can be rerun using a historical corpus, a current model or a reconstructed configuration. This can reveal how changes to documents, ranking or controls affect the result. It can also help teams test a proposed improvement.

But a replay is a new execution. Generative systems are not necessarily deterministic, and reconstructed conditions may be incomplete. A later run cannot prove what happened originally unless the relevant evidence was recorded at the time.

• Original evidence: records captured during the production interaction.
• Historical state: recoverable versions of documents and configurations that existed at the time.
• Replay: a controlled new execution used for comparison and diagnosis.

Conflating the three creates false confidence. Separating them makes an investigation defensible.

Show the target state using the client’s own knowledge

An audit report can describe this capability, but a working demonstration makes it tangible. A focused evidence lab can load a controlled subset of the client’s corpus—perhaps 20 to 100 representative documents, including changed, withdrawn and conflicting versions—and run real or representative questions through an instrumented reference system.

The lab should demonstrate a correct answer with a complete evidence trail, a wrong answer caused by poor retrieval or ranking, and a plausible answer for which the original system retained too little evidence to determine what went wrong. The third scenario shows that answer quality and accountability are different properties.

The demonstration should not become another AI platform that the client must adopt. Its purpose is to make the required evidence model visible, establish a reference point and help the organisation map those capabilities onto its existing architecture.

The focused capability set

• Versioned documents and source passages
• Recorded retrieval, filtering and reranking events
• Relationships between material claims and supporting evidence
• Versioned prompts, models and execution configuration
• Correlated control and tool events
• A tamper-evident interaction history

The valuable intellectual property is not a generic chatbot. It is the evidence model, the instrumentation pattern and the investigation experience.

The useful blockchain analogy

There is a genuine analogy to blockchain, but it should be used carefully. Blockchain systems preserve an ordered history whose later alteration can be detected. An AI evidence trail can use similar cryptographic ideas: each execution event can include a hash of its payload and the preceding event; document versions can be identified by content hashes; periodic checkpoints can be digitally signed or written to immutable storage.

These mechanisms can help demonstrate that an answer record has not been silently changed, a passage belongs to the recorded document version, events remain in their original sequence, and a collection of records existed before a signed checkpoint.

But the most important blockchain feature—distributed consensus among parties that do not trust one another—is usually unnecessary inside one accountable organisation. Append-only storage, hash-linked events, signed checkpoints, immutable retention and independent replication can deliver most of the practical value with less complexity.

Borrow blockchain’s tamper-evidence patterns only where they strengthen the evidence trail. Do not make blockchain the product story.

A shared ledger may become relevant when several independent organisations must rely on the same record—for example, a model provider, an application operator, a regulated deployer and an external auditor. Even then, a shared transparency log or independently signed checkpoints may be sufficient.

Integrity cannot create missing evidence

Cryptography can demonstrate that a record has not changed. It cannot prove that the record was complete, the source was true or the answer was correct.

A perfectly preserved evidence chain can still contain the wrong source, omit discarded candidates, record a misleading answer or fail to show that a required control never ran. Strong integrity therefore depends on trusted instrumentation and clear completeness requirements. The system must first capture the right events. Cryptographic integrity can then protect them.

From assessment to evidence lab

This can be structured as three fixed stages.

1. Traceability test

Trace a sample of real production answers as far as the available evidence allows. Identify where records, identifiers, versions or correlations are missing.

2. Evidence lab

Use a controlled subset of the client’s corpus to demonstrate what complete answer traceability looks like. Let stakeholders inspect claims, historical sources, retrieval decisions, controls and replay comparisons.

3. Production blueprint

Map the demonstrated capabilities onto the client’s actual architecture. Separate configuration and instrumentation fixes from data-model changes, pipeline redesign and genuine architectural rebuilds.

The result is not a legal certification, a promise of perfect model accuracy or an attempt to reproduce every nondeterministic output. It answers a more concrete operational question: when an AI answer is challenged, can we show what the system knew, explain what it did and establish whether the evidence has remained intact?

That is what moves AI governance from policy language into system design.

Is answer playback the same as rerunning a prompt?

No. Playback inspects records captured during the original production interaction. A rerun is a new execution and is useful for comparison, but it cannot replace missing original evidence.

Does a RAG system already provide this evidence trail?

Not automatically. Retrieval can ground an answer while still failing to retain the candidate set, ranking, document versions, claim-to-source links, prompt configuration and control events needed for an investigation.

Does tamper evidence require blockchain?

Usually not. Append-only storage, hash-linked events, signed checkpoints, immutable retention and independent replication cover most needs inside one accountable organisation.

What is the smallest useful first step?

Take a small sample of real production answers and attempt to reconstruct each one. The first point where the trail goes cold identifies the missing instrumentation or state that deserves attention.