How to verify an LLM answer: a verification layer that catches hallucinations

To verify an LLM answer, identify the claims that matter and inspect the evidence for each one. Fluent wording, a citation marker and a second model’s approval are different signals; none alone establishes that the answer is true.

For a business assistant, start with the approved source set and the decision the reply could trigger. A wrong opening hour and an unauthorised contractual promise need different controls. This guide proposes a review structure; it does not claim a measured hallucination reduction.

Know what reached the generator

Keep the retrieved source passages, their versions and the draft together for review. An assistant can answer incorrectly because retrieval missed the relevant rule, the rule was stale or the generator misread a correct passage. Diagnose those boundaries separately.

A prompt can require the assistant to refuse when evidence is missing. That is a behaviour to test, not an enforced guarantee. The RAG acceptance sheet separates retrieval, answer and refusal evidence in more detail.

Sampling settings can affect output variability. Choose and evaluate settings supported by the model and workflow; no universal temperature value makes factual claims reliable. Even a repeatable answer can repeat a false statement.

Five checks for a proposed reply

1. Ground material claims in approved sources

Define which sources may support business policy, catalogue information and service terms. Require a clear unresolved state when the approved packet does not answer the question. Do not let the assistant silently replace a missing business rule with a plausible generalisation.

2. Check attribution and support

A backend can check whether a quoted passage is present in the supplied source. Presence is not the same as support: the passage may describe an exception, another product or an expired policy. Review the claim’s conditions, scope and source version as well as exact text.

A citation to an inaccessible document leaves the claim unverified. Do not treat a well-formed URL as evidence that the document exists or supports the answer. Preserve source identifiers without exposing private documents to unauthorised readers.

3. Use a separate review context where helpful

A reviewer can compare the draft against the source packet and acceptance criteria without inheriting the generator’s full conversation. Include enough authorised task context to judge the intended answer. Isolation can reduce one source of influence; it does not guarantee independence or correctness.

A model judge can share blind spots with the generator or follow misleading source text. Test it with known unsupported claims and retain human escalation for unresolved decisions. Check what instructions and tools the review environment actually loads.

4. Validate calculations and tool results

Use appropriate deterministic code or authoritative application queries for totals, dates and inventory rather than relying on generated arithmetic. Validate extracted inputs, units, timezone and the returned result. A correct function called with the wrong quantity can still produce an incorrect business answer.

If a tool fails or returns an ambiguous outcome, preserve that state. Do not invent a result to finish the sentence. Before retrying a consequential write, reconcile prior effects or use the application’s duplicate protection.

5. Assign the consequential decision

Define which replies can be sent automatically and which require a responsible reviewer. Medical, legal, contractual or substantial financial decisions need controls appropriate to the actual use and authority. A human approval step only helps if the reviewer has the evidence, time and authority to decide.

Keep a claim-level evidence record

Question / source packet version:
Draft claim / why it matters:
Source passage / conditions / date:
Quote present: YES / NO / UNKNOWN
Claim supported: YES / NO / UNKNOWN
Calculation or tool evidence:
Reviewer decision / unresolved issue:
Outgoing reply or escalation / receipt:

This separates a proposed answer from an approved commitment. It also lets you distinguish a retrieval failure from a judge that approved an unsupported interpretation.

Stress-test the boundary

Use harmless fixtures to test an invented product, an unauthorised discount, a false premise, a missing source and conflicting source versions. State the expected response before running the fixture. Test the reviewer as well as the generator: an unsupported draft should not pass merely because it contains a real quote.

Record every result, including unknown and not-tested cases. A passed set establishes the observed fixtures, not a guarantee for all future customer questions. Start with a written brief when the same unsupported claim recurs: source packet, draft, expected decision and where the present check failed.