Agent Decision Trace Schema: Record Evidence and Replay Limits

An agent decision trace should let a reviewer connect the input available at a decision boundary to the action authorised and the result observed. A completion message alone cannot do that. Neither can an approval card without its execution outcome.

In a production agent team, our exported build records contain approved requests that passed and approved requests that failed. That is a useful distinction: permission to execute and successful execution are separate events.

The case below concerns operational decisions around an agent’s requested action. It does not reconstruct private model reasoning.

A build record can separate permission from outcome

On 29 October 2026, a developer agent requested builds through approval cards. Each card described a change snapshot with a revision reference. An orchestrator approved the request, and a host runner executed the project’s fixed command list.

The available extract contains these records, renamed for publication:

The original extract covers A through D. Subsequent approval cards and runner messages supply E and F. These are selected records, not a census from which to infer a production failure rate. The export does not define how reported lines were counted. That column is not a measure of agent effort or code quality. Its separate code-change flag was blank, which leaves that field unknown.

For C and D, the operator attributed the failures to browser environment problems: browser discovery and a missing application-data directory, respectively. The extract does not include the error output needed to independently confirm those causes. A useful trace preserves that difference between a recorded failure and a reported diagnosis.

D, E and F used the same requested revision. E also failed. The operator then reported that the panel server had continued using the old worker code because it had not been restarted after the repair. After the server restart, F passed with the same requested revision. The later runner outcome establishes that this request passed on that attempt; the explanation of why the earlier attempt failed remains the operator’s attribution. This sequence does not isolate every possible runtime difference.

The runner outcomes are evidence beyond the requesting agent’s own account. They still do not prove that every output was correct, or that the surrounding approval process was independently audited.

Preserve the boundaries that a reviewer needs

The operator described a pre-run revision check: if the files changed after approval, the run would not proceed. The extract contains no demonstrated rejection for that reason. Treat it as a documented control, not a tested outcome of this case.

A trace for this workflow needs an input reference, the approval decision, evidence of the revision check, and the runner outcome. The diagnostic explanation should retain its own source. Otherwise, a later reader may mistake the operator’s plausible explanation for an observed error.

This separation also helps when investigating agent drift: a changed result, a changed input and a changed execution environment are different possibilities to check.

A proposed schema for the reader’s own records

The JSON Schema below is a proposed schema, not an export of our system’s internal format. It is intended to keep missing evidence visible and to separate inspection from a deliberate replay experiment.

A null value means UNKNOWN for the nullable fields. The literal value “unknown” serves that purpose in status fields. A populated field remains a claim until its referenced evidence is checked.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Proposed operational agent decision record",
  "type": "object",
  "required": [
    "trace_ref",
    "input_snapshot",
    "approval",
    "revision_check",
    "execution",
    "diagnosis",
    "replay"
  ],
  "properties": {
    "trace_ref": {
      "type": "string",
      "description": "An opaque reference; use a public alias in shared extracts."
    },
    "input_snapshot": {
      "type": "object",
      "required": [
        "reference",
        "revision",
        "tool_version"
      ],
      "properties": {
        "reference": {
          "type": [
            "string",
            "null"
          ]
        },
        "revision": {
          "type": [
            "string",
            "null"
          ]
        },
        "tool_version": {
          "type": [
            "string",
            "null"
          ]
        }
      },
      "additionalProperties": false
    },
    "approval": {
      "type": "object",
      "required": [
        "actor_role",
        "decision",
        "at"
      ],
      "properties": {
        "actor_role": {
          "type": [
            "string",
            "null"
          ]
        },
        "decision": {
          "enum": [
            "approved",
            "rejected",
            "unknown"
          ]
        },
        "at": {
          "type": [
            "string",
            "null"
          ],
          "description": "Timestamp with timezone when known; null means unknown."
        }
      },
      "additionalProperties": false
    },
    "revision_check": {
      "enum": [
        "matched",
        "mismatched",
        "unknown"
      ]
    },
    "execution": {
      "type": "object",
      "required": [
        "runner_version",
        "started_at",
        "finished_at",
        "outcome",
        "exit_codes",
        "evidence_ref"
      ],
      "properties": {
        "runner_version": {
          "type": [
            "string",
            "null"
          ]
        },
        "started_at": {
          "type": [
            "string",
            "null"
          ]
        },
        "finished_at": {
          "type": [
            "string",
            "null"
          ]
        },
        "outcome": {
          "enum": [
            "passed",
            "failed",
            "not_run",
            "unknown"
          ]
        },
        "exit_codes": {
          "type": [
            "array",
            "null"
          ],
          "items": {
            "type": "integer"
          }
        },
        "evidence_ref": {
          "type": [
            "string",
            "null"
          ]
        }
      },
      "additionalProperties": false
    },
    "diagnosis": {
      "type": "object",
      "required": [
        "statement",
        "basis"
      ],
      "properties": {
        "statement": {
          "type": [
            "string",
            "null"
          ]
        },
        "basis": {
          "enum": [
            "operator_account",
            "execution_evidence",
            "unknown"
          ]
        }
      },
      "additionalProperties": false
    },
    "replay": {
      "type": "object",
      "required": [
        "mode",
        "unknowns"
      ],
      "properties": {
        "mode": {
          "enum": [
            "inspect_only",
            "isolated_test",
            "unknown"
          ]
        },
        "unknowns": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      },
      "additionalProperties": false
    }
  },
  "additionalProperties": false
}

Use references to approved, appropriately redacted artifacts rather than copying private payloads into every record. A public version can replace internal identifiers with local aliases, as the build table does.

The schema deliberately keeps the approval timestamp separate from execution start and finish. The original extract lacks those times. The later channel records add request and result events, but they still do not expose exact approval, processing-start and processing-finish timestamps. They cannot fully separate approval delay from runner duration.

An exit-code array preserves what the runner reported. Without a command-to-code mapping, it does not identify every check that ran. Schema validation can check the record’s structure; it cannot prove the evidence exists or that its interpretation is correct.

What a replayable agent audit can actually repeat

A trace supports different levels of review.

Inspection follows saved references and checks whether the stated conclusion matches the recorded inputs and outputs. It can detect an unsupported success claim without running the original action again.

An isolated test deliberately recreates a bounded part of the workflow with controlled inputs. It needs a stated environment and an explicit account of what the test excludes.

Repeating the original external action may create a new side effect. A trace alone does not authorise that action. Inspecting a record should not silently become another deployment, notification or write.

The combined evidence still lacks runner and browser versions, complete execution logs and the complete inputs required for a faithful replay. It records outcomes and supports a bounded operational review. It does not establish deterministic reproduction.

For a replayable agent audit, define the review boundary in the brief: which decision must be examined, what evidence is available, and whether the work is limited to inspecting records or includes an isolated test.

Close the record with what remains unresolved

A reviewer can conclude from this case that approval and execution outcome differ. The record also preserves a reported explanation of the failures.

The stronger causal conclusion remains unresolved without execution evidence. The later records show a pass after the restart for the same requested revision. They do not establish a general reliability improvement, a complete account of the runtime, or the correctness of every built artifact.

That is still a useful trace. It tells the next reviewer where the evidence ends, which claim needs checking, and which action has yet to be authorised.