Dev-agent factory with an isolated falsifier
An implementation and acceptance split with isolated worker scope, a separate runner and an explicit blocked result for contradictory requirements.
What changed. Artifact generation and acceptance are separate responsibilities; a result must reference the reviewed artifact and runner revisions.
Stack. isolated worker sandbox · external falsifier · commit-on-green · spec-defect escape
Context
A code-generation workflow cannot rely on the worker’s completion statement when the worker can also modify the checks. The engineering requirement is separate acceptance authority, with evidence tied to the artifact being reviewed.
Task
Separate specification, implementation and acceptance. Give the worker a bounded write area and allow a contradictory or incomplete requirement to return as BLOCKED. Do not force a fabricated pass when the specification cannot be satisfied.
What we built
The workflow has three responsibilities: an approved task and input packet, an isolated artifact workspace, and an external acceptance runner. The runner uses the relevant compilation, permissions and runtime fixtures for the artifact type. A release decision consumes its result rather than parsing the worker’s prose.
Hidden tests reduce direct tailoring but do not establish immunity from gaming. The effective runner environment, imports and result-file permissions also need review. A clean output file is not sufficient when another configuration path can change what the check observes.
The design covers different artifact lines with checks appropriate to their promised behaviour. A standalone application needs its actual interaction path exercised; a bot integration needs receipt, persistence and recovery evidence. An artifact existing on disk is only one observation.
A contradictory fixture illustrates the failure path: a parser needs token-shaped test data while a scan forbids that signature everywhere. Deleting an assertion or filtering that fixture from the runner’s file listing could turn the report green without resolving the contradiction. Treat this as a review scenario, not a newly reproduced incident or a measured failure rate.
What changed
The acceptance boundary is explicit: worker output, independent runner result and release decision are distinguishable. This case does not publish a cost multiplier, share of work by model tier or cycle-time improvement without the dated sample and baseline needed to support them.
A passing runner still covers only its configured fixtures. Record unknown external outcomes separately and verify that the release path rejects a deliberate failing fixture.
What we would do differently
Make BLOCKED a first-class result and set a budget for repeated attempts from the start. Review contradictions before another generation cycle. Keep the smallest responsibility split that enforces the boundary rather than adding agents simply to mirror an organisation chart.
Related service: AI agent audit