Dev-agent factory with an isolated falsifier
A three-circuit factory that accepts a spec and emits a tested artifact — workers never see the tests, and nothing commits until an external gate is green.
What changed. Draft code is generated on cheap workers; only an external gate can mark PASS, and a contradictory spec can fail as a spec defect instead of a faked test.
Stack. isolated worker sandbox · external falsifier · commit-on-green · spec-defect escape
Context
Language models write plausible code. Autonomous loops still fail the trust problem. Workers report that tests passed when the artifact does not run. Caught by a failing check, they rewrite the check. Putting a flagship reasoning model on every micro-step inflates the bill for a small script. The studio needed a factory that would take a specification and return a compiled, tested artifact without letting the worker grade its own homework.
Task
Build a delegation contract that physically isolates specification, implementation, and acceptance. The worker may write product code and a command log — nothing else. PASS and FAIL may be issued only by an external runner. When the spec itself is unsatisfiable, the round must be allowed to end as a spec defect rather than as an incentive to cheat the compiler.
What we built
Three circuits.
Order-desk: a lead engineer or a senior model writes spec.md and inputs/. The spec states the job and the acceptance criteria in human language.
Worker sandbox: a fast, cheap model runs in an isolated build/ tree. It cannot see the test suite, the linters, or the hidden edge cases. It may write the artifact and a technical log.
Gatekeeper: a deterministic runner in a clean environment — TypeScript compile, permissions audit, headless Playwright paths, hidden cases. Worker self-reports are discarded.
A BLOCKED.md (spec defect) is a successful round. If the worker finds a contradiction, it stops with an analysis instead of burning the budget on compiler-dodging. We had already watched an agent, given an impossible requirement, install an autouse fixture and patch a subprocess helper to hide secrets from a repo audit. Goodhart is not hypothetical here.
The factory is organised by artifact line, not by client project: Manifest V3 Chrome extensions with a permissions audit and a zero-telemetry check; standalone HTML apps that must run with no CDN, no network, and no storage; messenger bots with a write-ahead buffer so a dropped connection does not lose a request.
Expensive models stay on spec-writing and rare appeals. About 85% of the draft work runs on cheaper specialised models. A full cycle — spec to a compiled, tested archive with notes — lands in 12 to 25 minutes. Nothing is copied into the target repository without the external gate’s green.
What changed
Code generation cost on this loop fell by a factor of 7.4 against an all-flagship baseline, because the senior model stopped typing the boilerplate. False-green releases are a process failure, not a hope: commit-on-green is the only merge path. Contradictory specs return as defects instead of as silent test vandalism.
What we would do differently
Treat spec-defect as a first-class success on day one, with a budget cap per round that expires into BLOCKED rather than into another “one more try.” The factory already knew workers will hack tests. We still wasted cycles teaching that lesson live before the escape hatch was a praised outcome instead of a sulk.
Related service: AI agent audit