The Verification Gate Your AI Agent Is Skipping

Give the agent one reproducible entry point for the checks its task requires. A local runner can call lint, types, tests and a smoke scenario, then return a non-zero status when a configured check fails. The important boundary is what those checks observe, not whether they run locally or in a hosted service.

A passing exit code means the configured checks passed. It does not prove complete task coverage or establish an external effect the runner never inspected.

Ask two acceptance questions

Verification: does the implementation meet the stated requirements under the checks used? Static analysis and representative unit tests can contribute evidence.

Validation: does the assembled system support the intended use? An integration or end-to-end fixture can exercise the relevant path and inspect its result.

These questions overlap. They are an organising aid, not a claim that all failures fit exactly two categories. Security, accessibility, data quality and recovery may need additional checks or review.

Choose a runner for the actual environment

Local checks can shorten feedback when the dependencies and fixtures are available. Hosted runners can provide a clean environment, shared evidence or access to controlled integration fixtures. Either can be incomplete or misconfigured. Do not assume that a cloud product is required for a fleet or unsuitable for one developer.

Record the runtime and commands needed to reproduce the result. Keep credentials out of output. When a prerequisite is missing, report the blocked check instead of pretending an unrun test passed.

Order checks to make failures useful

1. Static analysis: parsing, linting and types can catch structural defects without exercising every runtime path.
2. Runtime tests: exercise representative inputs, failures and expected outputs.
3. System confirmation: inspect the critical assembled path and the outcome the task promises.

These are complementary observations, not exclusive failure classes. A smoke test does not replace broader tests, and a build does not establish that the live deployment changed. Select the checks for the current change and its risk.

During an incident, keep unrelated polishing out of a repair when it hides the failing behaviour or delays diagnosis. Refactoring may still be needed to fix or isolate the defect. Make that dependency explicit rather than banning every structural change until a smoke test is green.

An illustrative check.py

Replace the tools below with the repository’s established commands. The sample stops at the first configured failure; it has not tested your application or created the referenced smoke fixture.

#!/usr/bin/env python3
# check.py - illustrative runner for the configured checks.
import subprocess, sys

# (label, command) — in ladder order: static → runtime → system
GATES = [
    ("lint",   ["ruff", "check", "."]),
    ("types",  ["mypy", "."]),
    ("tests",  ["pytest", "-q"]),
    ("smoke",  ["python", "-m", "tests.smoke_e2e"]),
]

def main():
    for label, cmd in GATES:
        print(f"→ {label}: {' '.join(cmd)}")
        result = subprocess.run(cmd)
        if result.returncode != 0:
            print(f"\n✗ FAILED at {label} (exit {result.returncode})")
            print("  Configured checks failed. Inspect the result before continuing.")
            sys.exit(1)
    print("\n✓ OK - configured checks passed; verify any remaining task outcomes.")
    sys.exit(0)

if __name__ == "__main__":
    main()

Keep the runner outside inappropriate worker write access when it serves as independent acceptance authority. Tie its output to the artifact revision and preserve enough logs to inspect a failure. A worker-authored green status is not a substitute for running the protected check.

Confirm that the gate can reject a failure

Use an isolated fixture that should fail and verify that the runner returns a failing result. Then check the final completion or release path: does it actually stop or report the failed verdict? Testing only the detector can miss a reporting path that ignores it.

Before declaring completion, compare the configured coverage with the task’s promised result. If the task includes publication, persistence or delivery, obtain the appropriate independent evidence or report it unverified. Replaying a production side effect is not a safe default test.

Task / artifact revision:
Configured checks / environment:
Exit result / evidence path / time:
Deliberate failing fixture rejected:
Promised outcome outside these checks:
Independent outcome evidence / UNKNOWN:
Remaining blocker / permitted next action:

Wire the repository’s existing instructions to its authoritative check command rather than creating another parallel checklist. Start with a written brief for a recurring false completion: the claim, actual artifact, current runner and the missing observation.