RAG Acceptance Checklist: A Test Sheet for Document Answers

A document assistant can find the right policy and still give the wrong answer. It can also produce a plausible answer without finding the policy at all. Before accepting a retrieval-augmented generation (RAG) system, keep evidence for both steps.

Use the sheet below with the person supplying the assistant. Agree which documents are authoritative, what each question should produce, and what blocks acceptance. A polished demonstration is not a completed test sheet.

Start with one testable acceptance question

Replace “Does the assistant work?” with a bounded question: “Can an authorised support operator answer the agreed policy questions from the current document set, and handle questions that the set cannot answer?”

Freeze a test packet before the run:

• The document versions and effective dates, including which conflicting versions are retired.
• The operator role and the documents that role may access.
• Questions with expected passages and an expected answer, clarification or refusal.
• The system version, retrieval settings and date of the run.
• Mandatory checks, acceptance rules and the person authorised to sign off.

Have the business owner approve the expected behaviour. Have someone other than the implementer review the recorded evidence where practical. An automated judge can assist review; it does not supply missing source evidence.

A worked example: finding the passage is only one check

This is a fictional teaching example, not a reported client test.

Document P-01, version 2: “Support requests submitted before 16:00 on a business day are acknowledged that day. Requests received later are acknowledged on the next business day.”

Question: “A request arrived at 16:30 on Tuesday, a business day. When should it be acknowledged?”

Expected: the next business day, using P-01 version 2. Assume no holiday in this example.

Suppose the retrieved context contains that passage, but the answer says “Tuesday.” Record retrieval PASS and answer FAIL. The citation can be genuine while the conclusion is wrong. Keep the question, retrieved passage and observed answer together so the reviewer can reproduce the decision.

If no execution trace is available, retrieval is PENDING, even if the answer happens to be correct. If the assistant used a retired document, retrieval fails against the agreed packet.

Copy this acceptance record

Create one record per question in a shared document or spreadsheet. Leave observations blank until the run. Use PASS, FAIL or PENDING for applicable checks; use N/A only with a recorded reason.

Case ID:
Question and operator role:
Corpus / document versions / effective date:
System version / retrieval settings / run date:
Expected source passage or reason none should exist:
Expected behaviour: ANSWER / CLARIFY / REFUSE
Expected answer and required conditions:
Mandatory critical case: YES / NO

Observed retrieved source IDs and passage evidence:
Observed answer and citations:
Run / evidence reference:
Retrieval: PASS / FAIL / PENDING / N/A + reason
Answer: PASS / FAIL / PENDING / N/A + reason
Required refusal: PASS / FAIL / PENDING / N/A + reason
Access boundary: PASS / FAIL / PENDING / N/A + reason
Reviewer / review date:
Failure owner / next action:
Retest version / evidence / result:
Acceptance decision / decision owner / date:

For the worked example, the expected source is P-01 version 2 and the behaviour is ANSWER. The observed “Tuesday” response fails the answer check even though retrieval passed. Refusal is N/A because refusal was not required. Other unperformed checks remain PENDING or are excluded with an agreed reason.

Evidence references should point to access-controlled records. Do not paste private documents, credentials or personal data into a public worksheet or a support conversation.

Build a small test set that can fail

Include questions outside the happy path. These are proposed test designs; they are not measurements of a particular system. On a narrow screen, scroll the table horizontally.

Use synthetic documents and disposable test roles for access checks. A prompt saying “do not disclose” is not evidence that the retrieval backend enforces access. Record this as a separate mandatory check where restricted material exists.

Count retrieval, answers and refusals separately

A single accuracy percentage hides which step failed. Microsoft’s RAG evaluator documentation distinguishes retrieval quality from response groundedness, relevance and completeness. The acceptance method here uses that separation; it is not a vendor certification or a universal scoring standard.

Use these worksheet definitions:

• Retrieval coverage: cases where the expected passage was found / all cases requiring an expected passage.
• Answer acceptance: passing answers or clarifications / all cases requiring an answer or clarification.
• Required refusal: correct refusals / all cases explicitly requiring refusal.

Keep PENDING cases in the denominator and show their count beside the result. FAIL and PENDING are different states, although neither adds to the numerator. Do not silently remove a difficult case by marking it N/A after the run.

For example, a fictional answer group with six cases might have four PASS, one FAIL and one PENDING: report 4/6, with one pending, rather than 4/5. This arithmetic illustrates the sheet; it is not a measured success rate. If a group has no cases, show “not assessed,” not 100%.

Access-boundary checks sit outside these quality fractions: report each applicable critical check explicitly. For absence questions with no expected passage, retrieval can be N/A with that reason; the required refusal still needs evidence.

Agree thresholds and critical cases before testing. A missing mandatory case or unresolved critical failure blocks sign-off under this proposed method, even if an aggregate percentage looks good. Business owners must choose acceptable rules for their actual task; there is no percentage on this page that makes every RAG system safe.

Turn a failure into a retest

Classify the failed boundary before changing the system:

• Expected passage missing: inspect document availability, permissions, indexing and retrieval evidence.
• Correct passage present, unsupported conclusion: inspect how the answer uses that context.
• Relevant answer missing an exception: review completeness against the expected response.
• Unsupported question answered confidently: review the agreed refusal path and its enforcement.

Assign an owner and preserve the failed row. Record the new corpus or system version, repeat that case, and rerun the agreed regression set. A fix that helps one question may change another answer; do not replace the original evidence with the successful screenshot.

Sign-off should name the packet, tested version, reviewed cases, exceptions and decision owner. An accepted sample supports a decision about that sample and configuration. It does not establish production accuracy, eliminate hallucinations or cover future document updates.

When a workflow audit is useful

A single outdated document may need an ordinary content correction. An audit is useful when the team cannot identify authoritative sources, retrieve evidence for failed answers, or agree who can accept the result.

For recurring gaps, start with a written brief for an AI agent audit: describe who asks questions, which documents apply, one failed or unresolved example, and the evidence already available. Keep confidential source documents out of the initial brief.