AI Agent Drift: The Five Failure Layers, Diagnosed

When an agent solves the wrong task, forgets a decision or reports an outcome that never happened, start with the observed failure. These symptoms can involve different parts of its working environment. The five-layer map below helps choose the next check; it is not an exhaustive taxonomy or proof that the model cannot be the problem.

Five places to look

L1: task intent

Compare the requested outcome with the acceptance rule the agent actually used. A clean implementation can still solve the wrong problem. State the intended user result, required evidence and exclusions before another attempt.

L2: project conventions

Check whether the relevant rule was discoverable and applicable to the files changed. An unwritten architectural constraint can be missed even when tests pass. Link to the authoritative rule rather than copying contradictory versions into several instruction files.

L3: execution environment

Compare the runtime, dependencies, fixtures and commands used by the agent with the environment that rejected the result. A local success does not establish reproducibility in CI or production. Record a missing dependency as a blocker rather than substituting an imagined test result.

L4: verification

Inspect the actual artifact or external outcome. A completion statement is a claim, and a passing check covers only what it observes. Verify that the final reporting path consumes the gate verdict. See completion-claim checks.

L5: carried context

Check what the new session actually loaded: current instructions, relevant decisions and outstanding blockers. A stored note is useful only if it is found, read and reconciled with the current source. See cross-session handoffs.

One incident may cross several layers. An ambiguous task can combine with missing context and an incomplete check. Record a primary hypothesis and contributing boundaries; do not force every failure into one exclusive label.

Review a bounded set of runs

Choose a dated sample with an inclusion rule. Recent failed runs can expose mechanisms but will not estimate the overall failure rate unless the sampling method supports that conclusion.

Run / task version / date:
Expected outcome and acceptance rule:
Observed artifact / failure:
Evidence available / missing:
Primary hypothesis / contributing layers:
Proposed change / retest fixture:
Observed retest result / remaining uncertainty:

Group the findings to locate repeated boundaries. Frequency alone does not decide priority: severity, recovery cost and evidence strength also matter. A rare unsafe effect can take precedence over several harmless convention errors.

Compare a proposed remedy on representative tasks while recording model and environment versions. Where practical, change one relevant factor at a time. Do not attribute an improvement to the harness if the model, tasks and scoring rule also changed. No comparative success percentages are asserted here.

Route to the relevant guide

• Repeated attempts without evidence of progress: retry loops and stop rules.
• Completion claims without the result: independent verification.
• Instructions competing with task context: instruction size and routing.
• Decisions lost at a session boundary: handoff evidence.
• Missing reproducible checks: a local verification gate.

These are starting points. Tool permissions, data quality, model capability and external service failures may need separate investigation. A model change can be appropriate after the evidence identifies a capability limit; a larger model does not automatically repair an unobserved deployment result.

What to request in an audit

Start with a written brief: the affected task, dated runs, current acceptance checks and the outcome that remains uncertain. The useful deliverable is a finding anchored to evidence, a bounded repair and a retest method. Avoid accepting a layer score or a tool recommendation without seeing the underlying observation.