Indexation Coverage Auditor
AGENT ROLE: Autonomous indexation-reconciliation agent. Parse the sitemap, gather coverage states, join a crawl and traffic, build one URL ledger, classify each URL deterministically, cluster the gaps by cause, and emit the JSON in references/output.schema.json.
OBJECTIVE
Turn four disconnected views of a site (sitemap, GSC coverage, crawl, analytics) into a single reconciled ledger that says, for every URL, whether it is indexed and earning, and — when it is not — which specific indexation failure it belongs to and how to fix that cluster.
INPUTS
sitemap (REQUIRED): sitemap URL or local file; sitemap-index and .gz are followed/decompressed (bounded by max_sitemaps, max_urls).
site (OPTIONAL): GSC property, required only for --inspect and --live-crawl.
coverage (OPTIONAL): local JSON map url -> {coverageState, verdict, indexingState} (a GSC export) — avoids the API.
crawl (OPTIONAL): crawl as a url -> {status, indexable, internal_links} map or a list of such records.
traffic (OPTIONAL): local JSON map url -> clicks|sessions.
inspect (OPTIONAL flag): call the URL Inspection API for sitemap URLs (bounded by max_inspect, default 200 — the API has a low daily quota).
live_crawl (OPTIONAL flag): bounded BFS crawl from site when no --crawl file is given (max_crawl default 2000, max_depth default 6).
AUTHENTICATION (Search Console URL Inspection API — only with --inspect)
- No credentials are needed for sitemap parsing or crawling.
- IF
--inspect THEN REQUIRE env GSC_OAUTH_TOKEN (OAuth 2.0 access token, scope https://www.googleapis.com/auth/webmasters.readonly). IF unset THEN STOP error.code="AUTH_MISSING_CREDENTIALS".
- Endpoint:
POST https://searchconsole.googleapis.com/v1/urlInspection/index:inspect with {inspectionUrl, siteUrl}. IF 403 THEN STOP error.code="AUTH_NO_SITE_ACCESS".
EXPECTED TOOL CALLS
- Run
scripts/indexation_audit.py --sitemap {url} --site {property} [--inspect --max-inspect 200] [--coverage coverage.json] [--crawl crawl.json] [--traffic traffic.json].
- The script parses the sitemap (recursing indexes), optionally inspects a bounded URL set, optionally crawls, then joins everything on the normalised URL.
PROCEDURE (deterministic)
STEP 1 — SITEMAP: parse <loc> from urlset/sitemapindex (recurse, gunzip). IF zero URLs THEN STOP error.code="EMPTY_SITEMAP".
STEP 2 — COVERAGE: from --coverage and/or --inspect, map each URL to (in_index, cause) via coverageState/verdict (Submitted-and-indexed → indexed; Discovered → discovered_not_indexed; Crawled-not-indexed; noindex → excluded_noindex; duplicate/canonical → excluded_canonical; redirect; soft-404).
STEP 3 — JOIN: universe = union(sitemap, crawl, traffic, coverage). Per URL record in_sitemap, crawl_status, internal_links, indexable, in_index, coverage_state, clicks.
STEP 4 — CLASSIFY each URL into exactly one class: indexed_earning, indexed_no_traffic, submitted_not_indexed, discovered_not_indexed, crawled_not_indexed, excluded_noindex, excluded_canonical, excluded_redirect, orphaned_earning (clicks>0, no inlinks, not in sitemap), crawl_error, not_in_sitemap_indexable, coverage_unknown.
STEP 5 — CLUSTER the non-healthy URLs by cause; attach the canonical likely_fix; sort clusters by size. EMIT summary + gaps + ledger.
RATE LIMITS & ERROR HANDLING
- URL Inspection:
429 → backoff 2^attempt (max 5); persistent quota → stop inspecting, set inspect_quota_capped=true, and classify the rest as coverage_unknown (never fabricate a state). 403 → AUTH_NO_SITE_ACCESS.
- Sitemap/crawl fetch
429/5xx → backoff 2^attempt (max 4). Crawl paces 150ms/request; concurrency 1 (sequential BFS).
- Cost guards:
max_inspect, max_crawl, max_sitemaps, max_urls.
MISSING / INSUFFICIENT DATA
- No coverage at all (
coverage_source="none") → status="insufficient": the sitemap↔crawl↔traffic ledger is still produced (orphans, crawl errors, sitemap gaps), but index-state classes are reported as coverage_unknown rather than guessed.
- No traffic file →
indexed_earning collapses into indexed_no_traffic; orphan detection needs both traffic and crawl inlinks and is skipped otherwise. NEVER invent index state or clicks.
OUTPUT
One JSON object per references/output.schema.json. No prose.
FILES
scripts/indexation_audit.py — sitemap parse, URL Inspection, crawl, ledger join, classification, gap clustering.
references/output.schema.json — output contract.