The agent stack for SEO

Four layers: from data sources to a finished output

Stack architecture

Any SEO agent has four layers. Each layer owns a job — the boundaries matter when you pick tools.

A minimal Python agent

Here is the skeleton of a simple SEO agent. In a real project each function grows; the structure stays the same:

import anthropic
import sqlite3
from datetime import datetime

# --- 1. Data Sources ---
def fetch_gsc_data(site_url: str, days: int = 28) -> list[dict]:
    """
    Returns top pages from the Google Search Console API.
    Requires: google-auth, google-api-python-client
    Scope: googleapis.com/auth/webmasters.readonly
    """
    # ... initialize credentials ...
    # ... call searchanalytics.query(...) ...
    return [
        {"page": "/blog/seo-guide", "clicks": 320, "position": 4.2},
        # ...
    ]

def fetch_serp(keyword: str) -> dict:
    """
    Live SERP via SerpAPI / DataForSEO.
    """
    # import serpapi; top-10 results
    return {"organic_results": [...]}

# --- 2. Processing ---
def normalize_pages(raw_data: list[dict]) -> list[dict]:
    """Deduplicate URLs, normalize positions."""
    seen = set()
    result = []
    for row in raw_data:
        url = row["page"].rstrip("/")
        if url not in seen:
            seen.add(url)
            result.append({**row, "page": url})
    return result

def save_to_db(conn: sqlite3.Connection, rows: list[dict]) -> None:
    conn.execute("""
        CREATE TABLE IF NOT EXISTS pages
        (page TEXT, clicks INT, position REAL, date TEXT)
    """)
    for row in rows:
        conn.execute(
            "INSERT INTO pages VALUES (?,?,?,?)",
            (row["page"], row["clicks"], row["position"],
             datetime.today().isoformat())
        )
    conn.commit()

# --- 3. Agent Logic ---
client = anthropic.Anthropic()

def analyze_pages(pages: list[dict]) -> str:
    """LLM analyzes the data and writes recommendations."""
    data_str = "\n".join(
        f"- {p['page']}: position {p['position']:.1f}, clicks {p['clicks']}"
        for p in pages[:20]  # top 20
    )
    message = client.messages.create(
        model="claude-sonnet-4-6"  # current ID: check docs.anthropic.com,
        max_tokens=1024,
        system=(
            "You are an SEO analyst. Analyze page positions, "
            "find growth opportunities, give concrete priorities."
        ),
        messages=[{
            "role": "user",
            "content": f"GSC data for 28 days:\n{data_str}\n\n"
                       "Return the top 5 recommendations in Markdown."
        }]
    )
    return message.content[0].text

# --- 4. Output ---
def save_report(text: str, path: str = "report.md") -> None:
    with open(path, "w", encoding="utf-8") as f:
        f.write(f"# SEO report {datetime.today().date()}\n\n{text}")
    print(f"Report saved: {path}")

# --- Main ---
if __name__ == "__main__":
    conn = sqlite3.connect("seo.db")
    raw = fetch_gsc_data("example.com")
    pages = normalize_pages(raw)
    save_to_db(conn, pages)
    report = analyze_pages(pages)
    save_report(report)

Choosing tools

SERP API: price and capability

LLM: picking a model for an SEO agent

Agent stack cost calculator

At 10,000 queries/month the full stack is about $30/month — less than one hour of specialist time. Run your scenario:

Check yourself

Technical site audit

Audit agent: find, prioritize, report

Technical SEO checklist

The audit agent checks each of these items as it crawls the site:

• robots.txt and sitemap.xml — files reachable, syntax valid, sitemap registered in GSC
• Index status — no unwanted noindex; canonical points at the right URLs
• Broken links (4xx / 5xx) — every internal link returns 2xx
• Duplicate content — identical or near-identical pages without canonical cause cannibalization
• Core Web Vitals — LCP ≤ 2.5s, CLS ≤ 0.1, INP ≤ 200ms (direct Google ranking factors)
• HTTPS and mixed content — all content loads over HTTPS, no HTTP assets on an HTTPS page
• Mobile layout — viewport meta present, content is not clipped on screens < 375 px
• Structured Data / Schema.org — markup is valid and covers the key entity types
• Image alt text — every meaningful image has a descriptive alt
• Meta tags — title 50–60 characters, description 120–158 characters, unique on every page
• Click depth — important pages are reachable in no more than 3 clicks from the homepage
• Orphan pages — no indexable page lacks inbound internal links

Audit agent architecture

crawl(url)
  → fetch_page(url)          # download HTML, status, headers
  → check_all(page): [findings]  # run all 12+ checks
  → prioritize(findings):    # priority = severity × traffic_impact
  → generate_report(findings)    # JSON / Markdown / HTML report
  → notify(report)           # Slack / email / Jira ticket

Crawler tools

Prioritizing findings

A raw list of hundreds of findings is useless. The audit agent applies:

• Severity (1–5): 5 = a direct Google ranking factor, 1 = cosmetic fix
• Traffic Impact (1–5): 5 = site-wide, 1 = a single page
• Final range: 1–25

Priority examples

Finding priority calculator

Check yourself

Keyword Intelligence

Clustering, intent mapping, and gap opportunities

Keyword classification

Before the agent works the keyword set, it needs a keyword taxonomy — one classification by length and by intent. Without it clustering is noise: the agent does not know which pages to assign the terms to.

By query length

By intent (Search Intent)

Search intent is the goal behind the query. An agent that mislabels intent assigns the keyword to the wrong page type, and the content never reaches the top — even if the technical quality is high.

Semantic clustering

The agent collects thousands of keywords; without grouping they are noise. Clustering turns a flat list into a map of future site structure: each cluster is one page (or section).

Agent output after clustering — a cluster map with a site-structure proposal: 'Cluster "buy iPhone" (42 keywords, avg KD 35) → create /shop/iphone/ with 1 category page and 4 product pages'.

Keyword Gap Analysis

Gap analysis finds keywords where a competitor already ranks and you do not. Demand is proven, relevance is proven by the competitor; the job is to ship a stronger page.

The gap-analysis sweet spot — keywords with high volume + low KD + matching intent for pages you already have or can create cheaply. The agent can flag this quadrant automatically.

Example Python pseudocode:

# gap_analysis.py — agent gap-analysis pseudocode

import requests

AHREFS_TOKEN = "..."

def fetch_domain_keywords(domain: str) -> set[str]:
    """Return the domain keyword set via Ahrefs API."""
    # GET api.ahrefs.com/v3/site-explorer/organic-keywords
    resp = requests.get(
        AHREFS_TOKEN,  # endpoint lives in config
        params={"target": domain, "limit": 10000},
        headers={"Authorization": f"Bearer {AHREFS_TOKEN}"}
    )
    resp.raise_for_status()
    return {row["keyword"] for row in resp.json()["keywords"]}

def fetch_kw_metrics(keywords: list[str]) -> dict:
    """Fetch Volume and KD for a keyword list."""
    # POST api.ahrefs.com/v3/keywords-explorer/overview
    resp = requests.post(
        AHREFS_TOKEN,  # endpoint lives in config
        json={"keywords": keywords},
        headers={"Authorization": f"Bearer {AHREFS_TOKEN}"}
    )
    resp.raise_for_status()
    return {r["keyword"]: r for r in resp.json()["keywords"]}

def gap_analysis(your_domain: str, competitor_domain: str, top_n: int = 50):
    your_kws     = fetch_domain_keywords(your_domain)
    competitor_kws = fetch_domain_keywords(competitor_domain)

    gap_kws = list(competitor_kws - your_kws)          # set difference
    metrics  = fetch_kw_metrics(gap_kws)

    # Sort: high volume, low difficulty first
    ranked = sorted(
        gap_kws,
        key=lambda kw: metrics.get(kw, {}).get("volume", 0)
                       / (metrics.get(kw, {}).get("kd", 100) + 1),
        reverse=True
    )

    return [
        {
            "keyword": kw,
            "volume":  metrics[kw]["volume"],
            "kd":      metrics[kw]["kd"],
            "intent":  metrics[kw].get("intent", "unknown"),
            "score":   metrics[kw]["volume"] / (metrics[kw]["kd"] + 1),
        }
        for kw in ranked[:top_n]
        if kw in metrics
    ]

if __name__ == "__main__":
    results = gap_analysis("yoursite.com", "competitor.com", top_n=50)
    for r in results:
        print(f"{r['score']:6.1f}  Vol={r['volume']:5d}  KD={r['kd']:2d}  "
              f"[{r['intent'][:4]}]  {r['keyword']}")

Keyword map

A Volume vs Difficulty scatter is the main visual for prioritization. Each point is a keyword; dashed lines split the space into quadrants.

Check yourself

The content agent

From a keyword to a finished SEO article in one pipeline

A 6-stage pipeline

The content agent does not write an article in one call — it runs six sequential stages, each narrowing context and raising output quality.

System prompt: what goes in

Content-agent quality is mostly the system prompt. Required pieces:

• Target keyword and LSI terms — as an explicit list so the agent knows which variants to use naturally.
• Search intent — what the user wants: learn (informational), buy (transactional), or find a site (navigational).
• Required structure — H1 with the keyword, 4–5 H2s, a required close/CTA.
• Hard limits — do NOT invent statistics without sources, do NOT use the keyword above 2% of the text.
• Target length — 1,000–1,500 words for informational, 500–800 for transactional.

System prompt template (pseudocode):

You are an SEO copywriter. Write the article strictly to spec.

TARGET QUERY: {target_keyword}
LSI TERMS: {lsi_list}
INTENT: {intent_type}  # informational | transactional

STRUCTURE:
  H1: must contain {target_keyword}
  H2 x 4-5: {h2_list}
  Close: short summary + call to action

CONSTRAINTS:
  - Do NOT invent statistics without a source
  - Keyword density: 1-2% of the text
  - Length: {min_words}-{max_words} words

META:
  title: 50-60 characters, contains {target_keyword}
  description: 120-158 characters, reason to click

AUDIENCE: {audience_description}
TONE: {tone}  # formal | conversational | expert

When the agent, when the editor

Content pipeline cost

Calculate the real cost of producing content at your scale.

Check yourself

SERP monitoring and link profile

The agent watches ranks and backlinks — and raises the alarm first

SERP monitoring

Search positions move every day. Tracking them by hand means you always learn late. A monitor agent makes this systematic.

What to track

Anomaly thresholds

• Traffic drop > 20% in a day — diagnose immediately: algo update or a technical issue
• Rank drop > 10 places in a day on one keyword — check competitors and fresh content on the page
• Drop on all keywords at once — likely robots.txt / noindex or a Google algo update

Market-wide volatility is tracked by MozCast and SERPoscope — if those indexes spike, it is not just your site in the storm.

Monitor agent: pseudocode

# Rank-monitor agent (cron: every 24h)

def serp_monitor_agent(keywords: list, threshold_drop: int = 5):
    state = load_state("serp_positions.json")   # previous positions
    alerts = []

    for keyword in keywords:
        current_pos = fetch_position(keyword)   # GSC API / DataForSEO
        prev_pos    = state.get(keyword, current_pos)

        drop = current_pos - prev_pos           # positive = worse

        if drop > threshold_drop:
            alerts.append({
                "keyword":  keyword,
                "prev":     prev_pos,
                "current":  current_pos,
                "drop":     drop,
                "severity": "HIGH" if drop > 15 else "MEDIUM"
            })

        state[keyword] = current_pos            # update state

    save_state("serp_positions.json", state)

    if alerts:
        # Extra context: check MozCast
        volatility = fetch_mozcast_score()
        context = "algo-update likely" if volatility > 80 else "site-specific issue"

        send_alert(alerts, context=context)

    return alerts

Rank movement over 30 days

On the chart below: the lower the line, the better the rank (position 1 beats position 25).

Link profile

Backlinks have been a core Google ranking factor since PageRank. Links pass "authority" (link equity) from one domain to another.

Link quality metrics

Toxic links and risk

Toxic-link tells:

• Domains with DR < 5 and irrelevant topics
• The same anchor text on hundreds of links (over-optimized anchors)
• Links from forum signatures, comments, low-quality directories
• A sudden burst of hundreds of links in a short window

Link Gap Analysis

Link gap — domains that link to competitors but not to you. A ready prospect list.

# Agent: link-profile analysis + link gap

def link_profile_agent(my_domain: str, competitors: list):

    # 1. Fetch backlinks via API (Ahrefs / Semrush)
    my_backlinks  = fetch_backlinks(my_domain)
    comp_backlinks = {c: fetch_backlinks(c) for c in competitors}

    # 2. Score referring domains
    scored = []
    for link in my_backlinks:
        score = compute_link_score(
            dr=link["domain_rating"],
            tf=link["trust_flow"],
            relevance=link["topical_relevance"]   # 0..1
        )
        scored.append({**link, "score": score})

    # 3. Filter toxic (score < 0.2)
    toxic    = [l for l in scored if l["score"] < 0.2]
    healthy  = [l for l in scored if l["score"] >= 0.2]

    if toxic:
        build_disavow_file(toxic)   # -> disavow.txt for Google

    # 4. Link gap: competitor domains you do not have
    my_domains   = {l["domain"] for l in healthy}
    gap_domains  = set()
    for comp, links in comp_backlinks.items():
        for link in links:
            if link["domain"] not in my_domains:
                gap_domains.add(link["domain"])

    # 5. Prioritize prospects by DR
    prospects = sorted(gap_domains,
                       key=lambda d: get_dr(d), reverse=True)[:50]

    return {
        "healthy_links": len(healthy),
        "toxic_links":   len(toxic),
        "prospects":     prospects
    }
  

Check yourself

Orchestrating SEO pipelines

How to assemble every worker into one system with reliable triggers

Orchestrator-worker pattern

In an agent SEO system the orchestrator is the central coordinator. It does not do the work itself: it takes a task, splits it, hands pieces to workers, and collects results. Workers are specialized agents; each owns a domain.

Running workers in parallel cuts wall-clock time: while KeywordWorker pulls data, AuditorWorker crawls the site.

class SEOOrchestrator:
    def route_task(self, task_type, complexity):
        """Pick a model by task type (wired to M10)."""
        if complexity == 'high' or task_type in ('strategy', 'eeat', 'competitive'):
            return 'claude-opus-4'       # frontier — deep analysis
        elif task_type in ('content', 'audit', 'monitor'):
            return 'claude-sonnet-4-6'   # workhorse — content and audit
        else:
            return 'claude-haiku-4-5'    # bulk — high-volume processing

    async def run(self, task: SEOTask):
        # Decompose the task
        subtasks = self.decompose(task)

        # Run workers in parallel
        results = await asyncio.gather(
            self.auditor.run(subtasks.audit),
            self.keyword.run(subtasks.keywords),
            self.monitor.run(subtasks.serp),
        )

        # Aggregate and decide
        plan = self.aggregate(results)

        # HITL check before irreversible actions
        if plan.requires_human_approval:
            await self.request_approval(plan)
        else:
            await self.execute(plan)

Trigger types

The pipeline starts three ways — each fits a different scenario:

Interactive pipeline

Click any block to see its role in the system.

Human-in-the-loop: when it is mandatory

Automation does not mean no humans. HITL is mandatory when an action is irreversible, reputationally risky, or can take the site down.

Error handling and state

Fault tolerance

delay = base * (2 ** attempt)
# 1s → 2s → 4s → 8s
# for API 429 / 503

What to persist between runs (State)

The pipeline must remember previous runs — otherwise anomalies and trends are invisible.

Check yourself

Evaluation and iteration

KPIs, anti-patterns, and the path from this course to a first working agent

KPIs for SEO agents

Metrics sit on two levels: operational (the system works correctly) and business (the system produces value).

Operational KPIs

Business KPIs

Content A/B tests

Anti-patterns — what kills the system

SEO automation ROI

Calculate how much an agent system saves versus manual SEO.

5 steps to a first working agent

Check yourself

AI search and GEO

How to change SEO strategy when search moves into AI answers

What is happening to search

From 2024–2026 the SERP shifted structurally: an AI-generated answer sits above organic results, and users reach the blue links less often.

GEO is still SEO

GEO (Generative Engine Optimization) is a label for optimizing not for a user click but for citation by an AI engine. The goal: become a source that Perplexity, Google AI Overview, or ChatGPT Search includes in the answer.

How AI actually picks sources (RAG + Query Fan-Out)

Google describes the mechanism directly — not a separate "AI ranking", but a layer on ordinary search:

1. Retrieval (RAG). The same core ranking systems pull relevant fresh pages from the index. You enter an AI answer only if the page is indexed and snippet-eligible — you cannot skip ordinary SEO.
2. Query Fan-Out. For one query the model spawns several related sub-queries (e.g. "how to get rid of weeds" → a set of follow-ups) and gathers results for each. Cover the topic and adjacent sub-questions, not one exact phrase.
3. Generation. From retrieved pages the model takes concrete facts and forms an answer with clickable source links.

Signals that actually work are SEO fundamentals, not "AI tricks": E-E-A-T and expertise, domain authority and citability, crawler access, semantic HTML, solid page experience. Structured data is not a requirement for AI features (see the note below).

Content formats AI prefers

AI Overview probability by query type

Strategy by query type

Not every query is equally exposed to AI Overviews. The tactic depends on intent:

AI Overview monitoring agent

The agent's job is to watch whether an AI Overview appeared for your keywords and to change content strategy on the alert.

Tools

• DataForSEO SERP API — supports ai_overview in the response object (field items_types[]).
• SerpAPI / Bright Data — alternatives with similar AIO detection.
• State store — SQLite/Redis: record the date AIO first appeared and the movement after.

Agent pseudocode

# AIO Monitor Agent — pseudocode
# Run: cron every 24h (or a webhook on rank change)

import serp_api, db, alerts

def run_aio_monitor(keywords: list[str]):
    for kw in keywords:
        result = serp_api.fetch(kw, features=["ai_overview", "featured_snippet"])

        aio_present = result.has("ai_overview")
        prev_state  = db.get_aio_state(kw)          # True/False/None

        if aio_present and not prev_state:
            # AIO appeared for the first time
            db.set_aio_state(kw, True, date=today())
            alerts.send(
                channel="slack",
                msg=f"[AIO ALERT] AI Overview appeared for '{kw}'. "
                    f"Current URL in position 1: {result.top_organic_url}. "
                    f"Recommendation: raise quality and the answer-first structure of the page."
            )
            content_pipeline.schedule_quality_review(kw)

        elif not aio_present and prev_state:
            # AIO gone — traditional SEO is the priority again
            db.set_aio_state(kw, False, date=today())
            alerts.send(
                channel="slack",
                msg=f"[AIO GONE] AI Overview disappeared for '{kw}'. "
                    f"Traditional SEO is the priority again."
            )

        # Log organic rank regardless of AIO
        db.log_rank(kw, result.organic_position, result.url)

Updating the content pipeline for GEO

Bridge to module M5: add a GEO layer on top of the existing content-generation pipeline.

1. Changes in the content-agent system prompt

2. Structured data — optional, not an "AI lever"

# Generate FAQ schema (add to the SEO Check pipeline step)

def generate_faq_schema(faq_items: list[dict]) -> str:
    """
    faq_items: [{"question": "...", "answer": "..."}, ...]
    Returns a JSON-LD string to insert in <head>.
    """
    schema = {
        "@context": "schema.org",
        "@type": "FAQPage",
        "mainEntity": [
            {
                "@type": "Question",
                "name": item["question"],
                "acceptedAnswer": {
                    "@type": "Answer",
                    "text": item["answer"]
                }
            }
            for item in faq_items
        ]
    }
    return f'<script type="application/ld+json">{json.dumps(schema)}</script>'

# Likewise: HowTo schema for step-by-step instructions,
# Article schema with author.name + author.url for E-E-A-T

3. Intent detection as an SEO Check step

# Add to the SEO Check Agent before publish:

def check_aio_risk(keyword: str, content: str) -> dict:
    intent = classify_intent(keyword)          # 'informational'|'commercial'|...
    aio_prob = AIO_RISK_MAP.get(intent, 0.5)   # from the monitoring database

    if aio_prob > 0.4:
        geo_score = score_geo_readiness(content)
        # geo_score: 0..1 (FAQ, definitions, structured data, authorship present)
        if geo_score < 0.6:
            return {
                "status": "warn",
                "msg": f"AIO risk {aio_prob:.0%}, GEO readiness {geo_score:.0%}. "
                       "Recommended: add FAQ, structured data, improve the definition."
            }
    return {"status": "ok", "aio_prob": aio_prob}

Quiz: check your understanding