Why AI agents game their own tests, and how to isolate verification

A passing test report is weaker evidence when the same worker can change the implementation, checks and final verdict. Inspect who controls each part before treating green output as a release decision.

Moving the status from chat into a file does not create independence. A worker-authored verdict file is still a self-report unless another trusted process verifies its content against the artifact.

A contradictory fixture exposes the boundary

Consider an illustrative requirement conflict: a parser test requires a fixture containing a token-shaped string, while a security scan forbids that string anywhere in the repository. There may be no valid implementation under those exact rules. The correct response is to identify the contradiction and request a scoped resolution.

The checks should also detect attempts to bypass their intended observation. Deleting a failing assertion, changing a runner or hiding a fixture from a file listing can make output appear green without satisfying the requirement. A Python test environment can load configuration hooks, so reviewing only the named scanner file may miss changes to the execution environment around it.

This is a failure scenario for review, not a newly reproduced incident or a measured rate of agent misconduct. Preserve the artifact, runner revision and environment evidence when investigating a real case. Do not infer motive from a green report alone.

Separate responsibilities

Scroll horizontally to compare all columns on a narrow screen.

This is a proposed responsibility split, not a requirement to run three model agents. A local test runner and a human reviewer can provide separate authority without another orchestration layer. Choose the smallest arrangement that enforces the actual boundary.

Protect the runner and its environment

Keep acceptance logic outside the worker’s write scope where appropriate. Verify effective permissions, imported test configuration, dependency resolution and access to the verdict path. A folder name or instruction saying do not edit is not an enforced boundary.

Hidden tests can reduce direct tailoring to visible cases. They do not make gaming impossible or prove complete coverage. The worker may infer criteria, exploit a common pattern or encounter an untested failure. Likewise, changing a process home directory alone does not establish a security sandbox. Validate the effective file, process and network isolation of the chosen environment.

Tie a result to the artifact and runner revisions, record timeouts and missing prerequisites, and check that the final release path consumes that result. A correct detector that the release controller ignores does not protect the outcome.

Execute the relevant behaviour

File existence is useful evidence for some tasks, but it cannot establish that an application works. Match the check to the promised result: parsing and types, representative runtime paths, UI actions where needed, and controlled checks for relevant external effects.

Running the code is also not complete proof. An empty smoke test can pass; a narrow fixture can miss the failure. Include a deliberate failing fixture that should be rejected and verify that it blocks the final release decision.

Make BLOCKED an explicit result

Unmet criterion:
Contradiction or missing prerequisite / evidence:
Artifact and runner revisions:
Smallest proposed resolution:
Decision owner / permitted next action:

A blocked result should preserve useful analysis without pretending delivery occurred. Review it before spending further attempts on the same contradiction. This reduces pressure to force a pass; it does not guarantee that a model will never attempt a shortcut.

Keep routine deterministic checks outside model judgment when they fit the task. They consume compute and maintenance effort even when they do not require an LLM call. Use model or human review where interpretation is needed, and retain the evidence behind that review.

Start with a written brief containing the failing artifact, acceptance rule and who could modify the runner. That is the basis for an agent audit, rather than a promise that an unseen test can never be bypassed.