Track legal-AI changes before reusing old test results.

Make the comparison visible

A successful test belongs to a particular setup and task. Before carrying its conclusion forward, put the earlier setup beside the current one. A changed prompt, model, source collection or document reader can affect the work even when the product name stays the same. Keep the earlier result, describe the change, and decide which conclusions need fresh evidence.

This guide offers a proposed process, not a certification method. NIST describes its AI Risk Management Framework as voluntary guidance for considering trustworthiness across AI design, use and evaluation. The Playbook’s MANAGE 4.3 suggests keeping records of system changes, their reasons, testing and deployment, together with version history. The worksheet below is our practical interpretation for a small evaluation; it is not a NIST form or required procedure. Primary sources.

Give each change a small record

Use one dated entry with four fields. Make it understandable to the next person who must explain a result.

1. Before and after. Record the model label and any available version identifier, the exact prompt, source-set date, retrieval settings and relevant software version. Mark a provider-controlled detail “unknown” when it cannot be recovered.
2. Intended difference. Describe the benefit in a sentence: for example, “Keep document dates beside each extracted event.” This gives the comparison a specific question.
3. Evidence. Save the test inputs, expected observations, actual outputs and reviewer’s notes. Reference the old result without overwriting it. Keep access appropriate to the material; a public change log is not a place for client documents.
4. Decision. Name the person deciding whether to proceed, the allowed task, unresolved limitations and the event that will trigger another review.

Compare the task, not just the score

Choose a few examples that expose the reason for the change. If a document reader changed, include a scanned page and an awkward table. If source selection changed, include a document that should be excluded. Where feasible, run the earlier and newer setups on the same retained inputs and keep the outputs side by side.

Write down what counts as an unacceptable result before looking at the new output. A higher average score can coexist with a newly missing qualification or an unsupported citation. Note differences individually. If the earlier setup is no longer available, label the comparison incomplete instead of treating a fresh run as proof of equivalence.

State the limits of the decision

A useful closing entry might read: “Use this version for this document type, with source-by-source human review; reassess after the next model or reader change.” If returning to the earlier setup is possible, record how. If it is not, identify the manual alternative. Keep the boundary beside the result so that an old pass does not become an unlimited approval.