The Verification Gate Your AI Agent Is Skipping

Every guide on “how to verify AI-generated code” sends you to a cloud — Braintrust, SonarCloud, a GitHub Actions matrix. For a fleet at scale that’s correct. For one developer in VS Code, it’s a way to spend an afternoon configuring something that doesn’t catch the bug the agent shipped this morning. The local-first answer is one file. We call it check.py.

check.py runs lint, types, tests, and a smoke E2E in that order, and exits non-zero the moment anything fails. The agent’s “done” claim runs through it. If check.py is red, “done” is not true.

Underneath it is a pattern we run on every project: two gates instead of one. Verification asks whether the code matches the spec. Validation asks whether the system actually works end-to-end. An agent that passes Verification alone will repeatedly ship a clean diff that breaks the build — because the system was never asked. This page gives you the two-gate model, the three-level ladder underneath it, and a thirty-line check.py you can drop into a project today.

Cloud verification is right — for a fleet, not for you

Braintrust, SonarCloud, and a GitHub Actions matrix are built for teams running hundreds of agent tasks across many repos. At that scale you need centralized eval dashboards, shared baselines, and remote runners. That’s a real problem and they solve it well.

You are one developer in VS Code with one agent. Routing every “done” claim through a cloud is slow, costs a network round-trip on each iteration, and — worse — the thing you actually configured usually doesn’t catch the bug the agent shipped this morning. The defect was a broken local build, not a regression against a hosted eval set.

You need a single local entry point the agent must clear before it is allowed to say “done”: make check, or a check.py at the repo root. One command, no network, deterministic, fast. Everything below is how to make that one command actually mean something.

The two-gate pattern: Verification and Validation

One gate is not enough, because “the code matches the spec” and “the system actually works” are two different questions. Most agent harnesses only ask the first.

L1 — Verification — “does the code implement the spec?”

Unit tests, lint, and type checks. Fast, deterministic, runs on every change. It confirms each piece does what the spec says in isolation. This is the gate everyone builds first — and often the only one they build.

L2 — Validation — “does the system actually work?” (the one people skip)

Integration and a smoke E2E. It exercises the assembled system the way a user would — start it, hit the critical path, assert the result. This is the gate that catches the clean diff that breaks the build.

Passing Verification alone ships convention violations and integration breaks with full confidence: every unit is green, the type checker is happy, and the app does not start. The agent reports “done” truthfully against the only question it was asked. The fix is to ask the second question on every run.

The three-level ladder underneath the gate

Each level catches a different class of failure. Skip a level and that class lives in your codebase unchecked — there is no other place it gets caught.

1. Static analysis — the code without running it. Lint and type checks. Catches syntax errors, undefined names, type mismatches, dead imports, style violations. Cheapest level; runs in milliseconds. Skip it and malformed code reaches the next levels and wastes their time.
2. Runtime behavior — the code doing its job. Unit and integration tests. Catches logic that is well-typed but wrong: off-by-one, bad branch, wrong return. Static analysis can never see this — it never ran the code.
3. System confirmation — the whole thing actually working. A smoke E2E on the critical path. Catches wiring failures: the service won’t boot, a config is missing, two green modules don’t talk to each other. Unit tests pass and the product is down.

The ladder maps onto the two gates. Levels 1 and 2 are Verification. Level 3 is Validation. A harness that stops at level 2 is exactly the agent that ships a clean diff and breaks the build — it climbed two rungs and called it the top.

The Priority Constraint: no polish until E2E is green

A rule that lives above the gate: forbid refactoring, optimization, and style work until the functional E2E is green. Functional correctness outranks everything that is not functional correctness.

The classic failure: a critical scenario is broken, and the agent spends the session renaming variables, extracting helpers, and tightening types — producing a large, beautiful diff that buries the fact that the thing still does not work. The diff looks like progress, reviews cleanly, and ships a broken system under a pile of improvements.

Encode the constraint where the agent reads it: “Until the functional E2E passes, do not refactor, optimize, or restyle. Make it work, then make it clean.” The smoke E2E in check.py is what tells the agent — and you — whether that line has been crossed.

The minimal implementation: a 30-line check.py

It wraps the tools you already have. It runs lint, then types, then tests, then a smoke E2E. It exits non-zero on the first red, prints a clear OK / FAILED, and runs on every “done” claim. That’s the whole gate.

#!/usr/bin/env python3
# check.py — the gate. "done" is only true when this exits 0.
import subprocess, sys

# (label, command) — in ladder order: static → runtime → system
GATES = [
    ("lint",   ["ruff", "check", "."]),
    ("types",  ["mypy", "."]),
    ("tests",  ["pytest", "-q"]),
    ("smoke",  ["python", "-m", "tests.smoke_e2e"]),
]

def main():
    for label, cmd in GATES:
        print(f"→ {label}: {' '.join(cmd)}")
        result = subprocess.run(cmd)
        if result.returncode != 0:
            print(f"\n✗ FAILED at {label} (exit {result.returncode})")
            print("  'done' is NOT true. Fix this before claiming completion.")
            sys.exit(1)
    print("\n✓ OK — all gates green. 'done' is true.")
    sys.exit(0)

if __name__ == "__main__":
    main()

Swap ruff / mypy / pytest for your stack’s equivalents. The shape is the contract: ordered ladder, first red stops the run, exit code is the verdict.

Wire it to the agent. Put one line in your AGENTS.md / CLAUDE.md: “Before claiming any task done, run python check.py and paste the output. If it exits non-zero, the task is not done.” Now “done” is a verifiable claim, not a vibe.