AI Agent Says "Done" But Nothing Changed: Verify the Outcome

An agent reports completion, but the required file, deployment or notification is missing. Start by checking the promised outcome independently. The completion message is one claim to test; it is not the acceptance evidence.

This article gives a diagnostic procedure, not an estimated failure rate for agents. Measure a dated sample of your own completion claims before drawing conclusions about the workflow.

Define what the claim commits to

Write down the observable result before judging the report. For a file edit, inspect the target version and bytes. For a database operation, check the intended record and transaction outcome. For a deployment, fetch the public URL and confirm the expected revision. For a notification, distinguish service acceptance from confirmed delivery when that distinction matters.

A build can pass while publication never happens. A notification request can time out after the receiver accepted it. Both require evidence beyond the agent’s summary, but they have different recovery implications. Repeating an ambiguous action can create duplicates.

Measure a bounded sample

Select a dated set of completion claims with a stated inclusion rule. A small sample can expose a failure mode; it cannot establish a universal rate or a reliable trend. Keep inaccessible evidence separate from a failed result.

Claim ID / time / task version:
Promised result:
Independent check / expected evidence:
Observed artifact / check time:
Verdict: VERIFIED / FAILED / UNKNOWN
Failure boundary / recovery decision:

verified + failed + unknown = all claims reviewed
observed failure share = failed / all claims reviewed
unknown share = unknown / all claims reviewed

Report the counts beside the fractions. Explain what the check could observe and whether the sample was representative. Do not silently count unknown results as successful. There is no general10 percent threshold below which a gate becomes honest: one missed safety-critical effect can matter more than several cosmetic errors.

Use an isolated review checkout or another appropriate read-only inspection path. Avoid moving a colleague’s active checkout or replaying production side effects merely to verify a claim. Compare existing evidence first; reruns need a safe fixture and explicit repeat protection where effects can escape the test.

Confidence is not completion evidence

Guo and colleagues’ 2017 calibration study examines classification networks and the relationship between predicted probability and correctness. It does not establish that an LLM claiming95 percent confidence completes agent tasks70 percent of the time.

For a particular agent workflow, confidence quality requires its own task definitions, labelled outcomes and evaluation method. Confident prose alone cannot establish a file change or a delivered message. A better prompt may improve behaviour; it still does not replace checking the result that matters.

Choose checks at the task boundary

Static checks

Parsing, type checks and linting can detect structural problems. They do not demonstrate a successful user journey or an external effect.

Runtime checks

Exercise representative inputs and inspect the outputs. Include expected refusal or failure behaviour where applicable. A passing fixture proves that observed case under that setup, not every future input.

End-to-end checks

Verify the boundary the task actually promises: a published page, a persisted record or a confirmed downstream result. Use a controlled fixture when a live action would be unsafe. If the production outcome remains inaccessible, report that limit explicitly.

These levels are a practical organising aid, not an exhaustive taxonomy. Security, accessibility, data quality and recovery may need additional checks and human review. Select checks for the actual risk rather than adding every possible gate.

Repair a demonstrated blind spot

For each failed claim, identify which boundary the existing check missed. Add an independent observation there, then exercise both a passing case and a fixture that should fail. Verify that the final reporting path uses the new verdict; a detector can be correct while the completion summary ignores it.

A stronger gate can catch the failures it tests. It cannot guarantee a permanent zero failure rate, and changing the model may still be appropriate when evidence points to a capability limit. Keep model behaviour, missing observations and deployment failures separate.

Start with a written brief for a recurring verification gap: sampled claims, dated artifacts, unknown outcomes and the boundary that the current gate does not inspect.