Why an AI agent works one day and not the next: drift at the harness level

The agent was checked last week. It worked. This week it still reports progress, and it is confidently doing the wrong thing. That is not mood. It is agent drift: expected behaviour and actual behaviour have come apart, and the loop does not raise an error.

Compare the current inputs and tool behaviour with the last verified run.

What drift is

Drift is the gap that opens over time between what the operator asked for and what the agent actually does. The failure is quiet. From the outside it looks as if the model got worse. Check both model/version changes and changes to tools, data or instructions.

A research agent that keeps losing context between steps, a session that cannot be taken over, stale context that still drives the next action — these are the same class of failure. The session is alive. The job is not.

Why it happens

The world changes and the instruction does not. A site is redesigned. An API changes its payload. A vendor tool goes dark. The agent is still tuned for yesterday.

The environment fails silently. The browser, the mail gateway, the CRM connector dies. The agent does not halt. It idles, retries, or invents a path.

Working memory is lost. The process restarts, forgets where it was, and continues as if nothing happened. Lack of visibility into agent context is the operator’s first clue, and it usually arrives late.

Memory swells. The longer the run, the more residue it carries. Yesterday’s project leaks into today’s.

There is no independent check. The agent says done. Nothing moved. It is not lying in the human sense. It was never taught to verify its own work, so its self-report cannot be trusted.

How it looks in production

These failure patterns are prompts for inspecting run evidence, not a measured incident frequency.

Idle work after a tool failure. A worker can keep attempting actions after its browser or carried state becomes unavailable. Inspect the tool error and the stopping decision before attributing the failure to model capability.

The report is not the fact. A run announced “interrupted, unfinished.” The work was already done. Trust the report and the operator repeats a finished job. Close every scenario on the fact on the client side, not on what the process claimed.

Green inside, dead outside. The server was healthy. What the person saw was an infinite spinner. Internal signs said fine. The product was not usable. Check with the client’s eyes, not only with the host’s.

Projects bleed into each other. One agent, several parallel themes. Tasks piled up, memory bloated, and a fact from one book was pulled into another. From the outside: the agent got stupid. In the room: one heap, no index.

A prompt change may help a specific behaviour. It does not independently confirm tool availability, carried context or the final user-visible result.

The real boundary

Before retuning the prompt, compare the failed run with the last verified setup. Identify which input, tool response or acceptance check changed.

Treat it as structure:

• Exit checks. The agent does not get to say done until an independent check confirms the result.
• Stop conditions. A tool is down, or retries are exhausted — halt. Do not improvise. That is what was written after the idle loop.
• Fact over self-report. Every scenario closes on a real effect the client can see.
• Structured memory. Split by context, index it, write and purge on a protocol. Memory without structure becomes a source of drift. Prompt rewriting does not fix a mixed heap.

Operating controls also need maintenance. Their useful effect is to expose the failures they observe and stop or escalate at agreed boundaries; no reduction in incident frequency is established here.

Instructions and operating controls need to be checked together.

The question to ask a contractor

If someone is building an AI system, one question is enough: what happens when the agent is wrong, or an external service drops? If the answer is about a well-tuned prompt, and nothing about checks, stop conditions, or result control, ask for a demonstrated failure path and an independent result check before accepting the system.

The useful question is not how to make the agent never err. That is not available. The useful question is how to keep the error from reaching the client unseen. That is harness engineering: the layer that turns “sometimes it works” into “when it fails, it fails in view.”