Product vs. commodity box

Module 2 of 9 — Positioning and the integrator's business model

In M1 we covered why grounding equals trust: an agent that knows the real catalog doesn't make things up. Now the next question: what exactly are you selling the client, and why is it not the same thing as a commodity AI chatbot?

🎯 What the store owner is actually buying

The integrator doesn't sell a "chatbot." They sell a solved problem.

⚖️ Four differences from commodity AI chatbots

💰 Business model: setup + subscription

The right structure for an integrator has two components:

• SETUP: covers the cost of implementation — don't underprice it, this is engineering work
• Subscription: provides predictable cash flow and aligns incentives (see below)
• Not pay-per-lead: on principle — it isn't just a different price tag, it's a different incentive mechanic

🔬 Deep dive: the economics of incentives (misalignment)

Why pay-per-lead is a structural problem, not just "expensive":

🧮 Calculator: pay-per-lead vs. subscription

See what a quality lead actually costs under each model. Plug in your own numbers:

🏷️ Price anchoring: how to position next to the box

The market has commodity solutions with a clear monthly price — the client already knows that range of figures. That's an anchor. Don't fight it — use it.

You position yourself not cheaper than the box, but above the anchor — done-for-you, grounded, with data ownership. It's a different product, not a pricier equivalent.

❓ Quiz: check your understanding

Harness architecture: thin kernel + instances

Module 3 of 9 — you'll understand why "build once, sell to many" isn't a metaphor but an engineering decision.

Pipeline: from feed to brief

As we saw in M2, reusability is production capacity. Let's start with the stages that make up the pipeline we'll be reusing.

┌─────────────────────────────────────────────────────────────────┐
│                        HARNESS PIPELINE                         │
└─────────────────────────────────────────────────────────────────┘

  Catalog feed            catalog.json              Brief / escalation
  (any source)                │                    to a human
       │                      │                         ▲
       ▼                      ▼                         │
  ┌─────────┐         ┌──────────────┐         ┌───────────────┐
  │  INGEST │──────►  │  RETRIEVAL   │──────►  │    CONSULT    │
  └─────────┘         └──────────────┘         │   (LLM call)  │
                                               └───────┬───────┘
                                                       │
                                               ┌───────▼───────┐
                                               │     GUARD     │
                                               └───────────────┘

Stage         What it does
─────────────────────────────────────────────────────────────────
ingest        Normalizes the raw feed → catalog.json (unified format)
retrieval     Deterministically selects candidates for the query (→ M5)
consult       LLM call: grounded answer based only on candidates
guard         Checks the answer for hallucinations and constraints (→ M6)
brief         Structured output or handoff to an operator
─────────────────────────────────────────────────────────────────

Thin kernel and swappable instances

The key idea: the pipeline works the same way for any store. Only the content changes — the catalog and the consultant's personality. So we split the system into two parts.

  Client A ──► [ Instance A ] ──┐
  Client B ──► [ Instance B ] ──┤──► [ KERNEL ] ──► LLM API
  Client C ──► [ Instance C ] ──┘

  New client = add an instance.  Don't touch the kernel.
  

The kernel↔persona boundary: an interface contract

The kernel works with an instance through a strict contract: a set of "slots" the persona is required to fill. The kernel doesn't know a slot's implementation — only its shape.

  instance/
  ├── catalog.json          # normalized list of items
  ├── persona/
  │   ├── profile.json      # tone, name, constraints
  │   ├── calc.js           # SLOT: function calcPower(area, region) → kW
  │   ├── faq.json          # question→answer pairs
  │   ├── promos.json       # current promotions
  │   └── routing.json      # escalation rules (→ operator)
  └── widget/               # front-end widget

  The kernel calls: instance.persona.calc.calcPower(area, region)
  The kernel doesn't know the formula — only the call signature.
  

Where the API key lives: a threat model

The front-end widget runs in the buyer's public browser. This is the single most important security constraint in this architecture.

// widget.js — PUBLIC file
const LLM_KEY = "sk-proj-..."; // it will be stolen!

fetch("https://api.llm.example/...", {
  headers: { "Authorization": "Bearer " + LLM_KEY }
})
      

// widget.js — PUBLIC
fetch("/api/consult", {
  method: "POST",
  body: JSON.stringify({ question: q })
  // no key here
})

// server.js — PRIVATE
const LLM_KEY = process.env.LLM_API_KEY;
// rate-limit + origin-check → LLM API
      

  Buyer's browser
       │
       │  POST /api/consult  {"question": "..."}
       ▼
  ┌─────────────────────────────────┐
  │   Server bridge (your server)   │
  │   • rate-limit                  │
  │   • origin-check (CORS)         │
  │   • LLM_KEY = env.LLM_API_KEY   │
  └──────────────┬──────────────────┘
                 │  Authorization: Bearer sk-proj-...
                 ▼
           LLM API endpoint
  

Classifier: kernel or instance?

Test yourself — click each component and find out where it belongs.

Test yourself

Modular prompt composition

Module 4 of 9 · Anti-soup: why a prompt is a system of blocks, not a monolith

In M3 we dissected the kernel/persona core — where the base/ and persona/ blocks come from. Now the key maintenance question: how to assemble these blocks correctly, why order matters, and why structured output beats free-form text.

Monolith vs. blocks: what goes wrong

Picture the system prompt for a heating-equipment online store: one wall of text, 1,500 lines — role, rules, FAQ, power-rating table, promotions, return policy, response format, escalation. The moment the store adds a new promotion, an engineer hunts for the right spot in that wall. Over time:

• Instruction interference: two directives in different places start contradicting each other. The model "averages" them unpredictably.
• Lost-in-the-middle: in a long context, important instructions in the middle sink — the model "sees" the beginning and the end more.
• Unmaintainability: for N clients you need N diverging copies, and edits never stay in sync.

Before / After: the same content

Below is a fragment of a monolith and the same content split into blocks. The information is identical, but the structure differs.

You are an AI consultant for a heating store.
Reply politely, in English.
Never make up prices. If you don't
know — say "check with a manager".
For power calculation: 1 kW per 10 m²
at a ceiling height of 2.7 m.
Recommend a boiler with a +20% margin.
Current promotion: 10% off installation
for orders placed before month-end.
If a customer asks for a return — hand off
to a live manager. Give the answer in
this format: first the calculation, then the
conclusion, then a question to the customer. Don't forget...
(another 1400 lines in the same vein)

## [base/role]
You are an AI consultant for a heating store.

## [base/guardrails]
Never make up prices.
If you don't know — "check with a manager".

## [base/output_contract]
Format: { calc, recommendation, question }

## [persona/calc]
Power: 1 kW / 10 m² (ceiling 2.7 m).
Boiler margin: +20%.

## [persona/promos]
10% off installation before month-end.

## [persona/routing]
Return request → live manager.

Block architecture

Block order = priority

Order is not random. The model "sees" the beginning and the end more strongly than the middle. We use this deliberately:

• Guardrails — right at the start, hard. Earliest of all, otherwise they can be "overridden" by later instructions.
• Role — the first line: it sets the frame for everything else.
• Output_contract — closer to the end, right before generation — fresh in the model's "memory" at output time.
• Resolve conflicts explicitly inside the block: "if X contradicts guardrails — guardrails take priority".

Output_contract: structure instead of free-form text

A monolithic prompt often ends with something like: "First give the calculation, then the conclusion, then ask a question". That's an instruction for a human. For a parser it's a disaster:

• The model rephrases slightly → the regex silently breaks → data lost
• Golden tests are green — they check the old format the model no longer generates
• A bug in production, invisible in CI (we'll cover this in detail in M8)

Output_contract = a contract that the provider helps enforce via structured outputs / tool-call schema. Parsing is trivial: JSON.parse() instead of regexes.

Calculation: area 50 m², power 5 kW.
Recommendation: 7 kW boiler.
Question: which fuel type do you prefer?

{
  "calc":   { "area": 50, "power_kw": 5 },
  "rec":    "7 kW boiler",
  "question": "Which fuel type?"
}

🔧 Prompt constructor

Toggle blocks on and off — watch how the assembled prompt, the token estimate, and the warnings change.

Token budget: don't bloat the blocks

Every block costs tokens. In the constructor above you can see the budget grow. Recommendations:

• guardrails — concise and hard, not an encyclopedia
• faq — the fattest block; keep only the genuinely frequent questions
• calc — formulas, not examples; the customer provides examples in the dialogue
• Total system prompt: target ≤ 800 tokens, up to 1,200 acceptable, beyond that — review needed

Check yourself

Module 5: Grounding via Retrieval

How a catalog becomes the model's vocabulary — and why deterministic selection is more precise than a neural net

Connection to M1: grounding = narrowing the generation space

In Module 1 we defined grounding as a principle: instead of letting the model "hallucinate" arbitrary answers, we narrow the space of allowable generations down to what actually exists in our system.

Retrieval is the concrete mechanism for this narrowing. Before calling the LLM we pull relevant candidates from the catalog and pass the model only those. The bot advises exclusively on the products it was given — it cannot offer anything outside the set.

Ingest: from feed to catalog.json

Before you can retrieve candidates, you need to bring the data into a single schema. A feed can come from any source — an XML export, a CSV from a supplier, a REST response from an ERP. Normalization creates one contract, regardless of source.

{
  "id":      "BOIL-042",
  "name":    "Gas wall boiler 24 kW",
  "price":   530,
  "url":     "/catalog/boil-042",
  "attrs": {
    "power_kw":    24,
    "fuel":        "gas",
    "area_m2_max": 240,
    "type":        "wall",
    "circuit":     "double"
  }
}

Ingest runs in batches, politely (rate-limit, retry with backoff), producing catalog.json, refreshed on a schedule or on an event. The LLM sees only this normalized file, not the "raw" feed.

Deterministic candidate selection BEFORE the LLM

The key architectural choice: candidates are selected by ordinary code, before the model is called. For a heating e-commerce store it looks like this:

def select_candidates(area_m2: float, fuel: str, catalog: list, n: int = 10):
    # Step 1: compute required power (rule of thumb: 1 kW per 10 m²)
    required_kw = area_m2 / 10.0

    # Step 2: deterministic attribute filter
    filtered = [
        p for p in catalog
        if p["attrs"]["fuel"] == fuel
        and p["attrs"]["power_kw"] >= required_kw * 0.85   # -15% tolerance
        and p["attrs"]["power_kw"] <= required_kw * 1.30   # +30% headroom
    ]

    # Step 3: sort by price (or by score)
    filtered.sort(key=lambda p: p["price"])

    return filtered[:n]   # ← this is the "candidate set"

Why deterministic selection rather than a neural net?

• Predictable — the same query always yields the same candidate set.
• Cheap — attribute filtering runs in O(n), with no GPU and no API calls.
• Debuggable — you can say exactly why a product made it into the set or not.
• Guarantees reality — candidates exist in the catalog; they have a price and availability.

The candidate set = the LLM's effective vocabulary

After filtering, we pass the selected products into the prompt. From that moment on the model can only name what is in the set. If a product didn't make it into retrieval — for the bot, it doesn't exist.

You are a consultant for a heating e-commerce store.
The customer wants to heat {area_m2} m² with a gas boiler.
Answer ONLY using products from the list below. Do not invent other models.

PRODUCT LIST:
{json.dumps(candidates, ensure_ascii=False, indent=2)}

Customer question: {user_question}

This is exactly why the candidate set is called the model's "effective vocabulary": it sets the bounds of what it can recommend. Set size is the main control lever for consultation quality.

📊 The recall / precision tradeoff: interactive plot

Too few candidates — the right product may not make it into the set (a miss). Too many — the context is bloated, the model "drowns" in irrelevant items, focus drops. Let's find the balance.

Illustrative curves — to demonstrate the principle:

Deterministic vs. semantic (embeddings)

Recommended hybrid: first a hard deterministic filter on attributes (cuts off everything numerically irrelevant), then, if needed, a semantic rerank inside the already-filtered set (sorts by the "spirit" of the query, e.g. "quiet" or "economical").

✅ Check yourself

Module 6. Runtime guard and graceful fallback

Defense-in-depth: prompt → retrieval → assert → fallback. When the bot gets it wrong — we degrade into value, not into embarrassment.

Context: what we already have and what's missing

In M1 we drew the soft/hard grounding boundary — the hard layer requires machine verification, not trust in the prompt. In M5 retrieval feeds the bot only real candidates from catalog.json. Seems like that's enough?

Defense-in-depth: four layers

Layer 1  PROMPT GUARD (soft)
        "Use only the products from the catalog below"
        → Reduces hallucination frequency. No guarantee.
        → The model can still "fill in" something nonexistent.

Layer 2  RETRIEVAL CONSTRAINT (M5)
        Feed only real candidates into the context.
        → The model sees the truth, not the whole internet.
        → Still probabilistic: it can mix up SKUs/prices.

Layer 3  RUNTIME ASSERT (hard) ← the key layer of this module
        After generation: every product/price the bot names
        is checked VERBATIM against catalog.json.
        Didn't pass → block the answer until retry/fallback.
        → The only layer that gives a machine guarantee.

Layer 4  FALLBACK (graceful degradation)
        Retry didn't help → DON'T show a broken answer.
        DON'T show a stack trace. Politely → capture the lead.
        → We degrade into conversion, not into an error.

Connection to the grounding spectrum from M1: layer 1 is L1 (soft / prompt), layer 2 is L2 (RAG / real candidates), layer 3 is L3 (hard / assert). And L0 (no grounding at all) is the baseline level that we don't ship to production. Defense-in-depth is exactly "climb up the spectrum and don't throw away the lower rungs — stack them."

Runtime assert: what it looks like in code

After receiving the answer from the LLM, the harness parses the mentioned products and prices, then checks them against the source of truth:

def assert_response(answer: str, catalog: list[dict]) -> dict:
    """
    Simplified runtime-assert example.
    catalog = [{"sku": "KTL-42", "name": "Alpha Boiler 24kW", "price": 950}, ...]
    """
    catalog_names = {item["name"].lower() for item in catalog}
    catalog_prices = {item["price"] for item in catalog}

    violations = []
    for item in catalog:
        # If the name is mentioned — the price must match exactly
        if item["name"].lower() in answer.lower():
            # Find the price near the product name (simplified)
            for price in extract_prices(answer):
                if price not in catalog_prices:
                    violations.append(f"Price {price} is not from the catalog")

    return {"ok": len(violations) == 0, "violations": violations}

State machine: the path of every request

The harness works as a deterministic state machine. Every request has exactly one outcome:

REQUEST
  │
  ▼
[consult] ──────────────────────────────────────────────────────────
  │  LLM generates an answer with products/prices
  ▼
[guard.check] ─── PASS ──▶ [show answer] ✅
  │
  FAIL (invented product or price)
  │
  ▼
[retry] ← narrow the candidates, strengthen the instruction, 1 extra LLM call
  │
  ▼
[guard.check] ─── PASS ──▶ [show answer after retry] ✅
  │
  FAIL again
  │
  ▼
[graceful fallback]
  DON'T show a broken answer
  DON'T show a stack trace
  "Let me clarify the details and pass you to a manager"
  → lead capture: name + phone + brief for the manager ✅

The economics of retry

Context: 2000 tokens
Answer:   300 tokens
Cost per request: ~$0.003

No retry:     1 call  = $0.003
With 1 retry: 2 calls = $0.006 (worst case)
With 3 retries: 4 calls = $0.012

→ 1 retry doubles the worst case.
  More than that inflates both latency and the bill.

Graceful degradation: why it's a win, not a failure

Intuition says: "the bot couldn't do it — that's bad." Harness engineering flips this around:

An honest caveat: estimate vs. final answer

Even after passing the assert, a boiler power calculation or a system cost is a preliminary estimate. The real parameters depend on the site survey, heat losses, ceiling height, and radiator type.

🧪 Guard simulator

Pick a scenario — the simulator will show the path through the state machine step by step.

Check yourself

🛡️ Shield, not replacement: brief and handoff

Module 7 of 9 — how the bot brings a lead to a hot state and hands it off to a human

Philosophy: the bot is the manager's shield

The core idea is simple: the consultant bot removes the routine — initial questions, capacity sizing, candidate selection — but does not close the deal. Its job is to bring the lead to a "hot" state and hand it off to a live manager.

This is not a shortcoming but an architectural choice. A bot without a shield is a bot that tries to impersonate an expert in the final negotiation. On a complex, expensive product (a heating boiler is $500–1,200 and an engineering decision that lasts for years) the buyer wants a live specialist. A bot trying to "push the close" will only breed distrust.

Depth: handoff = a conversion event

Which metric matters for a shield bot? Not conversation length. Not the share of conversations "resolved without a human." The main metric is the number of quality briefs handed off to a manager.

The handoff is the conversion. The entire system was built for this moment: persona-routing (M3), output_contract (M4), fallback guards (M6). The bot gathers data, qualifies intent, builds a structured card — and hands it off. The manager receives not "some customer showed up," but a hot lead with parameters, budget, and contact.

Depth: qualification > closing for trust

For an expensive/complex product there is a rule: honest qualification + handoff to an expert converts better than impersonating a manager.

• A buyer of a heating boiler is making a decision for 10–20 years. They want to talk to an engineer, not a chatbot.
• A bot that says "Great choice! Shall we place the order?" raises anxiety. A bot that says "I've selected three options, clarified the budget — I'm now handing you to our specialist, who will call you back within the hour" builds trust.
• Rule: don't pretend to be human when closing. Disclosing the bot's limits at the right moment is not weakness but honesty that works.

Depth: the brief as structured extraction (link to M4)

In M4 we studied the output_contract — the discipline of structured extraction. The brief for the manager is exactly the same discipline: from the conversation the bot extracts a card with fields, not a free-form retelling.

The manager receives:

{
  "area_m2":       150,
  "fuel_type":     "gas",
  "budget":        "$500-700",
  "candidates":    ["K-102 Gas boiler 32 kW"],
  "contact":       "+1 9XX XXX-XX-XX (Telegram)",
  "temperature":   "hot — asks for a callback today"
}

The real product ids are not "something boiler-like," but the specific K-102 from the catalog. The manager opens the card and already knows what to offer.

When to hand off: handoff triggers (link to M3)

In M3 we built routing rules. The handoff is one of the routes, with clear triggers:

How not to lose the lead

1. Capture the contact BEFORE handoff — name and phone/messenger are needed before the session is closed.
2. The brief goes out immediately — not after the manager's call, but the moment the button/trigger is hit. CRM / Telegram notification / email — in real time.
3. Manager offline — log the request ("I've noted you down, we'll call you back by 6:00 PM"), don't vanish into silence. A promise to call back = retaining the lead.

🔧 Interactive: brief builder

Go through the 5 steps of the mini-scenario — the bot asks questions and fills in the structured lead card. Collect all the fields, then click "Hand off to a manager."

Area:       —
Fuel:       —
Budget:     —
Candidate:  —
Contact:    —
Temp:       —

Check yourself

Verification Gate: golden ≠ adversarial

Module 8 of 9 — why "I checked it by hand" is not verification, and how to test a non-deterministic system

Verification Gap: the divide between "looks like it works" and "works"

In M6 we built a runtime assert that catches a contract violation right inside the bot. Today we'll cover how to make sure the bot is reliable before production, — and why that is fundamentally harder than it seems.

Why the happy path gives false confidence

Golden tests are fixed reference dialogues that the bot should "be able to" pass. These are regression tests: they verify that something that worked before didn't break after a prompt or model change.

Golden vs Adversarial: what's the difference

What goes into an adversarial set for a heating online-store bot:

How to test a non-deterministic system

An LLM gives a slightly different answer every time. Pinning down the exact text is impossible — it changes. And this is the same stochasticity as in M1: temperature / top-p > 0 make the output probabilistic (at temperature=0 the spread is smaller, but that adds no anchoring to the truth). Since the wording isn't fixed, asserting it verbatim is pointless. This sounds like a problem, but the solution is simple:

Invariants for a heating online-store bot:

🔍 Find the hallucination

Invariant inv_1 + inv_2: the product must be in the catalog, and the price must match. Below is a real catalog and 4 bot answers. Spot the violation.

K-101  "Gas boiler 24 kW"                  — $460
K-102  "Gas boiler 32 kW"                  — $590
R-210  "Aluminum radiator, 10 sections"   — $74
B-050  "Indirect water heater, 100 L"     — $315

How the automated gate works

The gate is a script that runs the golden + adversarial cases, asserts the invariants and emits an objective pass/fail. DoD: the gate is green.

# verify_bot.py — gate structure (pseudocode)
import json, re

CATALOG = json.load(open("catalog.json"))

def run_suite(cases):
    results = []
    for case in cases:
        answer = call_bot(case["input"])          # call the bot
        results.append({
            "case": case["id"],
            "inv1_no_phantom": check_inv1(answer),  # SKUs ∈ catalog
            "inv2_price_ok":   check_inv2(answer),  # prices match
            "inv3_role_ok":    check_inv3(answer, case.get("adversarial")),
            "inv4_non_empty":  len(answer.strip()) > 0 and "Traceback" not in answer,
        })
    return results

def check_inv1(answer):
    found = re.findall(r'[A-Z]-\d+', answer)
    return all(code in CATALOG for code in found)

def check_inv2(answer):
    for code in re.findall(r'[A-Z]-\d+', answer):
        price_match = re.search(r'\$\s*(\d[\d,]*)', answer)
        if price_match:
            stated = int(price_match.group(1).replace(' ', ''))
            if stated != CATALOG[code]["price"]:
                return False
    return True

GOLDEN  = json.load(open("cases_golden.json"))
ADVERSARIAL = json.load(open("cases_adversarial.json"))

all_results = run_suite(GOLDEN) + run_suite(ADVERSARIAL)
failed = [r for r in all_results if not all(r[k] for k in r if k != "case")]
if failed:
    print("FAIL:", [r["case"] for r in failed])
    raise SystemExit(1)
print("PASS — all invariants hold")

The gate's lifecycle

Check yourself

Deployment, Economics, Support

Module 9 of 9 — the finale. How to ship a grounded consultant to production, account for its cost, and keep quality up over time.

🚀 Deployment: two independent layers

A grounded AI consultant for an online store consists of two parts that deploy separately from the client's main site — that way we never touch their CMS or site builder.

<script src="https://api.bot.example/widget.js"
  data-origin="https://shop.example">
</script>

✅ Production smoke test (checklist)

After every deploy, run six checks. None of them requires touching code — just a browser and DevTools:

💰 Token economics: where the cost of a dialog comes from

Every dialog with an LLM costs money. The formula is simple:

Why modularity (M4) and candidate-set size (M5) hit your wallet: the system prompt and the candidates make up the bulk of the input context. With 20 candidates instead of 5, the input context can grow 2–3×. That is exactly why M5 taught us to select the minimal sufficient set.

⚙️ Model choice: the quality / latency / price trade-off

Strategies:

• One model for the worst case — simpler to maintain, slightly more expensive. A good fit for low traffic.
• Routing — classify the request (simple/complex) and pick the model. Saves up to 60–80% on simple questions. More complex infrastructure; you need a gate (M8) for both paths.

🧮 Project economics calculator

Plug in your own numbers and see what the LLM actually costs relative to the retainer. All prices are illustrative.

* Prices are illustrative; real rates depend on the provider and the model.

📉 Drift in production — the main threat to grounding

A grounded consultant is exactly as strong as its catalog is fresh. In production, everything drifts:

🔄 Support as a product: the link to M2

In M2 we talked about the "setup + retainer" model. Now you can see why it is built that way:

The calculator above shows it: LLM tokens are a small share of the cost. The retainer pays for the freelancer's work, not for reselling tokens. The client pays for a working, up-to-date service, not for API requests.

❓ Check yourself

Done 🎉

All 9 modules complete. You now have the full playbook: from the mechanics of hallucination to deploying and maintaining a grounded consultant.

• You understand WHY an LLM lies and what it means to "ground" an answer in a real catalog
• You can sell a done-for-you consultant, not a box (setup + subscription)
• You build a thin kernel + instances: a new client = swapping the persona
• You compose the prompt from blocks (no monolith), not as one lump
• You run deterministic retrieval of candidates BEFORE the LLM
• You set up defense-in-depth: guard + retry + graceful fallback into lead capture
• You design the handoff as a conversion event (shield, not replacement)
• You tell golden from adversarial tests — closing the verification gap
• You compute dialog economics and keep production drift under control

What to read next

• Harness-engineering methodology — the verification gap and modularity as a discipline of its own
• Your LLM provider's structured output / tool-use documentation
• Guides on prompt injection and LLM security (OWASP LLM Top-10)
• RAG patterns: hybrid retrieval, rerankers, retrieval-quality evaluation

▶ Restart the course