🔧 What a Harness Is Made Of

5 subsystems—anatomy of an agent's engineering infrastructure

In Module 1 we dissected 5 failure layers—the places where agents break. Now let's break down what a well-built harness is filled with so those failures don't happen. Key definition:

🍳 Kitchen Metaphor: 5 Subsystems

Imagine a professional kitchen. The cook (agent) is brilliant, but without the right infrastructure even the best chef produces chaos.

⚡ Order of Attack: Feedback First

When building a harness from scratch—don't start with the most obvious. Start with what gives maximum ROI at minimum cost:

🔗 Linking Two Lenses: Failure Layers → Subsystems

In Module 1 you learned to diagnose by mapping a failure to one of 5 layers. Now—fix: each fix lands in one of 5 subsystems.

📊 Interactive 1: Harness Effect on a Real Project

Case study: TypeScript + React, ~20,000 lines, GPT-4o model. The model did not change. Harness subsystems were added—and results grew.

🧩 Interactive 2: Failure Layer → Which Subsystem Fixes It?

Three scenarios—determine which harness subsystem needs investment.

🔬 Interactive 3: Ablation Calculator

Isometric model control — experimental method: hold the model constant, disable (ablate) one subsystem at a time, measure the drop in success rate. This tells you which subsystem matters most for your specific project.

🗄️ The Repository as a System of Record

Module 3 · Harness Engineering—only what lands in the repo exists for the agent

Single Source of Truth for the Agent

The agent operates in an isolated information bubble: it sees the system prompt, the task text, repository files, and tool output. Everything else—Slack threads, Jira tickets, Confluence pages, verbal agreements—does not exist for it.

This means: the repo must become a system of record—an authoritative source for decisions, constraints, current project state, and verification standards. Not a secondary archive, but the single place where the agent (and any new team member) looks for the truth.

Cold-Start Test—Five Questions

The best way to check a repository is to imagine a fresh agent session walking into it, remembering nothing from past conversations. Can the repo answer all five questions?

🧮 Interactive 1 — Cold-Start Scorer

Score your repository for each question (0 = no info, 1 = exhaustive answer). The calculator will show the total score and approximate KVG (Knowledge Visibility Gap).

Three Repository Health Metrics

KVG — Knowledge Visibility Gap

The share of project-critical decisions and constraints living outside the repository: in people's heads, Slack, email.

Discovery Cost — Cost of Discovery

How many tokens of the agent's context budget are spent to find what it needs, even if it exists in the repo. Information may exist but be buried in a giant file or scattered across dozens of folders.

Knowledge Decay Rate — Speed of Staleness

The share of documents drifting from the code over time. Code evolves— documentation often stays in place.

Solution: place docs next to the code they describe (src/payments/ARCHITECTURE.md, not docs/legacy/payments.md). Then a PR changing code inevitably passes the documentation—and the reviewer notices the mismatch.

ACID for Agent State Management

ACID principles from databases apply to the git repository as a store of agent work state.

git add -p          # only needed changes
git commit -m "feat: add retry logic"
git stash           # if you need to interrupt

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: run-tests
        entry: pytest -q

git worktree add ../agent-b feature/b
# agent A: main
# agent B: ../agent-b (isolated)

PROGRESS.md      ✓ survives session
DECISIONS.md     ✓ survives session
chat history     ✗ disappears

Knowledge Near Code

One mega-document in the root is an antipattern. As the repo grows, the agent spends its entire budget searching for the needed paragraph. Rule:

• Every module / service / library—its own ARCHITECTURE.md and CONSTRAINTS.md
• Root CLAUDE.md — navigation and run/check commands only
• Root PROGRESS.md — live state: what's in progress, what's blocked, what decisions were made today

project/
├── CLAUDE.md              # navigation + commands
├── PROGRESS.md            # current state
├── src/
│   ├── payments/
│   │   ├── ARCHITECTURE.md   ← agent reads immediately on entering module
│   │   ├── CONSTRAINTS.md    ← constraints next to code
│   │   └── *.py
│   └── auth/
│       ├── ARCHITECTURE.md
│       └── *.py
└── tests/
    └── AGENTS.md          # test acceptance criteria

🧠 Quiz

M4 📄 Why One Giant Instruction File Fails

Instruction Bloat, Lost-in-the-Middle, and the Progressive Disclosure Principle

In M1–M3 we established that the presence of a rule in AGENTS.md is critical. Now let's look at the flip side: if you keep piling everything into one file, it starts to hurt. A file of ~300+ lines slows the agent, reduces accuracy, and blurs priorities.

🗻 The Instruction Bloat Phenomenon

Instruction Bloat — when the instruction file grows so large that it takes up a significant share of the context window. Empirical rule: an instruction file > ~10–15% of the context window starts pushing out working task information.

📍 Lost in the Middle (Liu et al., 2023)

Research showed that LLMs better use information at the beginning and end of long context. Information in the middle "sinks." Practical conclusion: critical rules cannot be buried in the middle of a long file.

📊 Signal-to-Noise Ratio of Instructions

Instruction SNR — the share of file items relevant to the current task. If the agent is adding a new notification channel, and AGENTS.md has 37 items of which only 30 relate to this task, SNR = 81%. If it's running tests, only 18–19 items are relevant—SNR = 50%.

Low SNR is noise: the agent spends attention on irrelevant rules and may apply them inappropriately ("webhook rule unexpectedly applies to refactoring").

🔢 Interactive 1 — SNR Calculator

Enter the number of relevant items and total items in the file— get the SNR. In Lesson 4, items "always relevant" across all 5 task types were 9 out of 37—this is the skeleton of the future routing file (skeleton SNR = 100%).

📊 Interactive 2 — SNR by Task Type

The same AGENTS.md yields different SNR depending on task type. Average across 5 types ≈ 48%—almost half the instructions in any task are noise.

Lesson 4 data: T1 — add channel, T2 — debug errors, T3 — write tests, T4 — refactoring, T5 — update dependencies. Average SNR ≈ 48%.

🗺️ Routing File: Table Instead of Prose

Routing File — a short file (50–200 lines) that does not contain details but points to thematic documents. Essentially a table: condition → document.

## Conditions → documentation

| Condition                        | Document                     |
|----------------------------------|------------------------------|
| Adding a new channel             | docs/channels.md             |
| Changing DB schema               | docs/db-migrations.md        |
| Writing or changing tests        | docs/testing-guide.md        |
| Deploy / CI                      | docs/deploy.md               |
| Dependency changes               | docs/deps-policy.md          |
| Any task (safety rules)          | docs/safety-critical.md      |

🔭 Progressive Disclosure

Progressive Disclosure — the principle of "overview now, details on demand." The routing file gives quick orientation; detailed docs load only when needed. The agent doesn't "read the whole textbook" before every task—it looks for the needed chapter.

📖 Case Study: SaaS Team, 50 → 600 Lines → Refactoring

The team started with a 50-line AGENTS.md. Over a year, adding rules one by one, the file grew to 600 lines. Here's what happened:

• Task success: 45% (agent applied rules to wrong contexts)
• Safety rule compliance: 60% (critical rules ended up in the middle)
• Team complaints: "agent ignores our deploy requirements"

After refactoring: routing file (80 lines) + 6 thematic documents:

• Task success: 45% → 72%
• Safety rule compliance: 60% → 95% (critical rules moved to a separate doc that's always loaded)

🧪 Interactive 3 — Check Your Understanding

Next—the session lifecycle: initialization and context continuity between agent runs.

Session Lifecycle

Initialization as a separate phase · Context continuity between sessions

In previous modules we covered how an agent picks tasks and builds a plan. Now—about what happens inside a single session and at the boundary between sessions: initialization and context handoff.

A. Initialization as a Separate Phase

Typical mistake: the agent immediately writes business code without laying the foundation. Tools aren't configured, tests don't run, paths aren't created—and the first import breaks everything. This is mixing Initialization and Implementation.

Bootstrap Contract — 4 Mandatory Conditions

Initialization is considered complete when all four items are satisfied:

1. ✅ Can run — environment starts without errors
2. 🧪 Can test — at least one test passes green
3. 📊 Progress is visible — there's a measurable artifact (file, log, DB row)
4. 🔗 Can pick up — the next agent (or next session) understands what's done and what's next

TTFV — Time To First Verification

Key metric of initialization efficiency. The faster the agent achieves the first green test, the lower the risk it will spend context on an incorrect foundation.

🧩 Interactive: Bootstrap Contract

Check your understanding of the four Bootstrap Contract conditions.

B. Context Between Sessions

Every new session starts with a clean context. Without structured handoff the agent reads state from scratch and risks making decisions already made earlier—differently.

Context Anxiety

Anthropic finding: Sonnet 4.5 near the context limit exhibits premature convergence—choosing simpler solutions, skipping verification, rushing to close the task. This is not a behavior bug; it's a rational response to resource pressure.

Continuity Artifacts

Honest Takeaway: When Is PROGRESS.md Actually Needed?

🧩 Interactive: Is PROGRESS.md Needed Here?

Three scenarios from real worklogs. For each—decide if PROGRESS.md is needed.

📊 Cost of Splitting: Marathon vs Multiple Sessions

Data from lesson 5: the same 3-channel task was solved three ways. Metric—total tokens.

Overreach and Under-finish: Why WIP=1

Agents activate too many tasks per session and complete too few. Let's break the mechanics and the antidote.

Overreach: Too Many Tasks in Flight

The agent starts a session with an ambitious plan: "I'll implement 5 features at once." Each task activates—a branch is created, first lines of code written. But by the end of the session none have passed verification. This is overreach: scattered partial implementations instead of one completed feature.

Under-finish and VCR

Under-finish — the share of activated tasks that failed verification despite generated code. Overreach directly causes under-finish: the more tasks in flight, the higher the chance each won't be brought to executable proof.

Proof of Completion: Executable Evidence

"By eye" doesn't count. A task is complete only when there is executable condition that can be run and passes.

Little's Law: The Math of WIP

Little's Law—a fundamental result of queueing theory. For a stable system:

If λ = 1 feature/day and L = 5 (five tasks simultaneously), then W = 5 days per task. Keep L = 1 → W = 1 day → probability of error accumulation is minimal.

🧮 Interactive 1 — Cycle Time Calculator (Little's Law)

📊 Interactive 2 — The Price of WIP=1 Discipline

Data from a real experiment (lesson 7, worklog): ~700 lines, 6 features, Sonnet. WIP=1 is more expensive in tool calls—but catches what the gate misses.

⚠ Honest Empirical Takeaway

# No limits—shipped silently:
class DispatchNotFound(Exception): ...   # ← WRONG

# WIP=1 — caught during careful walkthrough:
class DispatchNotFound(NotifyError): ... # ← CORRECT

# Callers with "except NotifyError" miss
# DispatchNotFound — regex linter doesn't see inheritance.

# retry() recreates Dispatcher.from_config() with fresh
# defaults — config overrides from the original send are lost.
# Test passed because it tested happy-path without overrides.

WIP=1 in Practice: Activation Rules

🧠 Quiz

📋 Feature Lists as Data Structures + Multi-Level Validation

Module 7 · AI Agents and Harness Engineering

In modules 1–6 we talked about what to check and how to build gates. Here—about where to store scope and why agents declare victory too early. Both questions are connected: poor feature data structure = blind gate = false victory.

A. Feature List — Data Structure, Not a Planner

Typical mistake: the feature list lives in chat, Trello, or the manager's head. In the harness approach the feature list is a repo artifact that automation operates on. It has a strict schema, is versioned, and serves as input for the verification pipeline.

🔷 Triplet — Mandatory Record

Each line in the feature list carries exactly three fields. Missing even one—the record is incomplete and cannot enter the pipeline:

{
  "description": "POST /api/send returns 200 and writes message to DB",
  "verify_cmd":  "pytest tests/test_send.py::test_send_200 -x -q",
  "state":       "passing"
}

🔄 Feature State Machine

Each feature passes through 4 states. Transition → passing happens only on successful verify_cmd result. State passing is irreversible: regression means a new bug, not a rollback.

  not_started ──→ active ──→ passing  (irreversible)
                    ↕
                 blocked

📌 SSoT — Single Source of Truth

All scope information comes from one feature list. It's a chain: source → derived → executable. Each link can drift—if PROGRESS.md diverges from code, the harness loses orientation.

📊 State Pressure — Readiness Metric

The number of features not in passing is State Pressure. Zero = project complete. This is a quantitative, objective readiness metric—unlike subjective "feels like it's done."

🧮 State Pressure Calculator

B. Why Agents Declare Victory Too Early

Agents judge by "the code I wrote looks correct," not "the system E2E satisfies the spec." This is a systematic error, reinforced by math.

📐 Calibration Bias

Guo et al. (ICML 2017) showed: modern neural networks are systematically overconfident—stated confidence exceeds actual accuracy. The model says "95% sure it's done" where real accuracy is 70%.

📊 Worklog data: weak vs strict checking

Lesson 8 — 11 deliberately planted bugs. Two checking modes:

📊 Worklog data: false-done and gate blindness

Lesson 9 — 21 runs, calibration bias = 24% (5/21 false-done). Key covariate: gate coverage:

🔒 Dual Verification-Validation Gateway

For victory to be honest, you need two independent gates:

Both are mandatory. L1 without L2 = checked details, missed system. L2 without L1 = catching symptoms, not causes.

🏗️ 3-Level Validation

⚡ Completion Priority Constraint

Strict priority order:

🎯 Quiz

Module 8: E2E Testing, Runtime Observability, and Clean Session Exit

Course synthesis: calibrate harness by pressure, not by fashion

A. Only E2E Testing Changes the Result

Unit tests check each component in isolation. But an agent is a system where components interact. Inter-component failures are invisible to unit tests:

• Interface mismatch — module A expects a string, B returns a dict: both tests are green, system is broken.
• State propagation — an error in step 2 silently changes step 4's result; each step is tested independently and passes.
• Environment dependencies — test mocks the filesystem, but in real environment the path is different.

Anthropic Case: Retro Game Editor

Anthropic compared two approaches to the same task (identical prompt—develop a retro game editor):

30× in cost—and it's a justified investment: the cheap run produces a broken artifact, the expensive one—a working one. The evaluator agent in the harness performed E2E checks after each iteration.

📊 Cost Comparison: Bare Agent vs Harness

B. Make Agent Runtime Observable

An agent without observability operates as a "black box": it reports its own state, and its self-assessment is systematically biased toward completion. As in M3 we talked about ACID: you can't trust the agent that a transaction is complete—you need an external observer.

Practically: add to your check.py (from M2) not just unit tests, but a minimal E2E smoke: start the app, walk the critical path, ensure output matches expected. This is runtime observability in minimal form.

C. Every Session Must Leave a Clean State

As in ACID-Durability (M3): a transaction is either fully applied or rolled back. An agent session is a transaction on the codebase. You cannot leave half-done state, a red gate, or undocumented decisions.

A bad session exit is technical debt paid at the next start: the new session spends time understanding what was done, what wasn't done, and why this or that decision was made. This is context loss, which we discussed in M5.

✅ Session Exit Checklist

Check each item before closing the session:

D. Course Synthesis — Meta-Takeaway

🧪 Final Quiz: Course Synthesis

Done 🎉

8 modules completed. Here's what's in your toolkit now:

• See that the bottleneck is not the model but the harness; map every failure to one of 5 defensive layers.
• Distinguish 5 harness subsystems (Instructions / Tools / Environment / State / Feedback) and fix Feedback first.
• Make the repository a system of record: Cold-Start Test, KVG, ACID state.
• Split bloated AGENTS.md into routing file + topical docs (SNR, lost-in-the-middle).
• Maintain context between sessions: init phase, continuity artifacts, handoff.
• Introduce WIP=1 and executable proof of completion; calculate VCR and Little's Law.
• Build feature lists as data structures and multi-level validation (weak vs strict).
• Require E2E, make runtime observable, and leave clean session state.
• Calibrate methodology by scale—distinguish claims that reproduce from those that work only under real harness pressure.

What to Read Next

• Walking Labs — "Learn Harness Engineering." Course primary source: walkinglabs.github.io/learn-harness-engineering
• Liu et al. (2023) — "Lost in the Middle." Empirics on LLM attention degradation to the middle of long context.
• Guo et al. (ICML 2017) — "On Calibration of Modern Neural Networks." Why neural networks systematically overestimate their confidence.
• Anthropic — engineering notes on agents and context. Context anxiety, multi-agent harness, init phase.

▶ Retake Course