How can an auditor verify an AI agent's transcripts haven't been changed?

A step-by-step check for auditors and evaluators: compare an AI agent's transcript with a fingerprint recorded when it was written, fetch the record from storage the lab cannot change, and check the notary signature or on-chain anchor. What each step proves, and what it does not.

Datafund 8 min read

An auditor can verify that an AI agent’s transcript has not been changed by comparing the copy they are given with a fingerprint (a SHA-256 hash) recorded when the transcript was written, and by fetching the stored record from storage the lab running the agent cannot change. If the fingerprint carries a notary signature or is anchored on a blockchain, checking that also shows when the fingerprint existed. None of this proves the transcript was complete or true when it was stored; it proves it has not changed since.

Why this question matters now

In July 2026, during an internal OpenAI evaluation of cyber capabilities, models in a sandboxed testing environment exploited a zero-day vulnerability to gain internet access and reached Hugging Face’s servers (OpenAI’s incident report). The independent review of the agents’ behaviour by METR and Redwood Research found that agents “tried to retroactively edit transcripts”. They “modified some accessible action logs, but these weren’t the source of transcripts”, and the reviewers wrote: “We did not see them successfully edit the source of truth in our transcripts (scoped to Jul 7-13).”

OpenAI’s later guidance, “Towards safety cases for frontier AI training”, recommends: “Save agent transcripts immutably for a reasonable period of time (for example, in write-once storage accounts)”, and says “Auditors should be provided with sufficient access to verify that the claims of the safety case are valid and sound, and to raise gaps if found.”

This page is about the second half: how the people entitled to check (auditors, evaluators, the client) can verify a transcript themselves.

The one rule: the fingerprint must come from before

A fingerprint handed over together with the transcript proves nothing: whoever edited the transcript could compute a fresh one. The check only works if the fingerprint was fixed at write time and has been out of the lab’s reach since. Three common ways to get that:

  • The fingerprint (or the record’s storage address) was sent to you, or to a party you trust, when the transcript was written.
  • A notary signed the fingerprint with a timestamp at write time.
  • The fingerprint was anchored, meaning written on a blockchain, at write time.

Ask which applies. If none does, you can still compare the stored copy with yours, but you are trusting that the stored copy is the original.

Step by step

1. Collect what you need

The transcript copy you are given, the record’s storage address, the fingerprint recorded at write time, and any notary signature or anchor reference.

2. Fetch the record yourself, from storage the lab cannot change

Do not treat a copy the lab sends you as the reference. Fetch the stored record yourself, from storage where nobody, the agent and the lab included, can change it in place.

On Swarm, the decentralised storage network (ethswarm.org), a record’s address is computed from its content. Swarm’s documentation puts it plainly: “Because the address is derived from the content, a content-addressed chunk cannot be changed in place. Different content yields a different address.” An edited transcript would get a different address; the original stays until its stated expiry. The Datafund Provenance Toolkit downloads through Datafund’s gateway; the hash check in the next step is what removes the need to trust the gateway.

A write-once storage account is the other common choice. In Amazon S3 Object Lock’s compliance mode, “a protected object version can’t be overwritten or deleted by any user, including the root user in your AWS account”. An auditor then relies on the provider and the account setup, so pair it with a fingerprint kept somewhere the lab does not control (write-once storage compared).

What this proves: the bytes you fetched are the bytes stored at that address (or in that locked object). What it does not: that this is the only record, or that nothing was left out.

3. Recompute the fingerprint and compare

Compute the SHA-256 hash of the record you fetched and of the copy you were given. Both must match the fingerprint recorded at write time. If the copy you were given differs, something changed between storage and hand-over; ask why.

What this proves: the transcript has not changed since the fingerprint was recorded. What it does not: when that was, unless step 4 settles it.

4. Check the notary signature or the anchor

If a notary was used, it signed the fingerprint and a timestamp with its own key. Verify the signature. In the toolkit, during testing, the notary is Datafund’s public gateway; the gateway is open source, so any party you trust can run it. A notary timestamp rests on trusting whoever runs the gateway.

If the fingerprint was anchored, check that it was written on the blockchain and when. Then the time no longer rests on the gateway alone; it inherits the chain’s trust model. In the toolkit, anchoring runs on a test network (Base Sepolia) today; mainnet anchoring is not live.

What this proves: the fingerprint existed at that time (vouched for by the notary, or recorded on chain). What it does not: anything about the content beyond the fingerprint.

5. Follow the chain, if there is one

Provenance differs from a log by linking each output to the inputs and actions that produced it (What is agent provenance?). If records are linked (the toolkit’s transformation and merge links), repeat steps 2 to 4 for each linked record, to check what the transcript says it came from.

What the whole check does not prove

  • That the transcript was complete or true when it was stored. When an agent uploads its own record, it can write anything into it first. The METR and Redwood review describes “tool call spoofing”, where agents could “pretend to run one command while actually running another”. A faithful record of a spoofed output is still a spoofed output.
  • That everything was recorded. A check cannot show what was never stored.
  • That the work was correct, lawful or authorised, or who uploaded the record.

Having the software that runs the agent write the record, rather than the agent, would narrow the gap between what happened and what was recorded. In the toolkit that is a proposal, not something it does today.

Trying it

The Datafund Provenance Toolkit is pre-release, meant for testing, on a test network (Base Sepolia); AI agent provenance tools compared lists what it does today. The free tier is limited and may fail under heavy load; retry later. Free-tier records are kept for about a day; for evidence, use a stamp funded for the retention period you need.

The quickest test is dataprovenance.app: record a file, get a receipt, then play the auditor and verify it. How to give an AI agent a record it can’t rewrite shows the check in code, and If your agent can rewrite its own record, you don’t have a record covers why it matters.

Questions people ask

How can an auditor verify that an AI agent’s transcripts have not been changed? By comparing the transcript with a fingerprint recorded when it was written, and fetching the stored record from storage the lab cannot change. If the fingerprint carries a notary signature or an on-chain anchor, checking it also shows when the fingerprint existed.

Is a hash supplied with the transcript enough? No. Anyone who edited the transcript could compute a new hash for the edited version. The hash has to have been fixed at write time and kept out of the lab’s reach since: sent to the auditor then, signed by a notary, or anchored on a blockchain.

Does a matching fingerprint prove the agent behaved well? No. It proves the record has not changed since it was stored. An agent that writes its own record can put anything in it before storing it, so the check says nothing about whether the record was complete or true.

Sources