seoskills.sh
Catalog/SEO Audit/Indexation Coverage Auditor

Indexation Coverage Auditor

Reconciles your sitemap, Search Console coverage, a fresh crawl and analytics into one ledger, then groups unindexed pages by your likely cause.

Listed ·Updated

Built by seoskills.sh

SKILL.md

Indexation Coverage Auditor

AGENT ROLE: Autonomous indexation-reconciliation agent. Parse the sitemap, gather coverage states, join a crawl and traffic, build one URL ledger, classify each URL deterministically, cluster the gaps by cause, and emit the JSON in references/output.schema.json.

OBJECTIVE

Turn four disconnected views of a site (sitemap, GSC coverage, crawl, analytics) into a single reconciled ledger that says, for every URL, whether it is indexed and earning, and — when it is not — which specific indexation failure it belongs to and how to fix that cluster.

INPUTS

  • sitemap (REQUIRED): sitemap URL or local file; sitemap-index and .gz are followed/decompressed (bounded by max_sitemaps, max_urls).
  • site (OPTIONAL): GSC property, required only for --inspect and --live-crawl.
  • coverage (OPTIONAL): local JSON map url -> {coverageState, verdict, indexingState} (a GSC export) — avoids the API.
  • crawl (OPTIONAL): crawl as a url -> {status, indexable, internal_links} map or a list of such records.
  • traffic (OPTIONAL): local JSON map url -> clicks|sessions.
  • inspect (OPTIONAL flag): call the URL Inspection API for sitemap URLs (bounded by max_inspect, default 200 — the API has a low daily quota).
  • live_crawl (OPTIONAL flag): bounded BFS crawl from site when no --crawl file is given (max_crawl default 2000, max_depth default 6).

AUTHENTICATION (Search Console URL Inspection API — only with --inspect)

  1. No credentials are needed for sitemap parsing or crawling.
  2. IF --inspect THEN REQUIRE env GSC_OAUTH_TOKEN (OAuth 2.0 access token, scope https://www.googleapis.com/auth/webmasters.readonly). IF unset THEN STOP error.code="AUTH_MISSING_CREDENTIALS".
  3. Endpoint: POST https://searchconsole.googleapis.com/v1/urlInspection/index:inspect with {inspectionUrl, siteUrl}. IF 403 THEN STOP error.code="AUTH_NO_SITE_ACCESS".

EXPECTED TOOL CALLS

  • Run scripts/indexation_audit.py --sitemap {url} --site {property} [--inspect --max-inspect 200] [--coverage coverage.json] [--crawl crawl.json] [--traffic traffic.json].
  • The script parses the sitemap (recursing indexes), optionally inspects a bounded URL set, optionally crawls, then joins everything on the normalised URL.

PROCEDURE (deterministic)

STEP 1 — SITEMAP: parse <loc> from urlset/sitemapindex (recurse, gunzip). IF zero URLs THEN STOP error.code="EMPTY_SITEMAP". STEP 2 — COVERAGE: from --coverage and/or --inspect, map each URL to (in_index, cause) via coverageState/verdict (Submitted-and-indexed → indexed; Discovered → discovered_not_indexed; Crawled-not-indexed; noindex → excluded_noindex; duplicate/canonical → excluded_canonical; redirect; soft-404). STEP 3 — JOIN: universe = union(sitemap, crawl, traffic, coverage). Per URL record in_sitemap, crawl_status, internal_links, indexable, in_index, coverage_state, clicks. STEP 4 — CLASSIFY each URL into exactly one class: indexed_earning, indexed_no_traffic, submitted_not_indexed, discovered_not_indexed, crawled_not_indexed, excluded_noindex, excluded_canonical, excluded_redirect, orphaned_earning (clicks>0, no inlinks, not in sitemap), crawl_error, not_in_sitemap_indexable, coverage_unknown. STEP 5 — CLUSTER the non-healthy URLs by cause; attach the canonical likely_fix; sort clusters by size. EMIT summary + gaps + ledger.

RATE LIMITS & ERROR HANDLING

  • URL Inspection: 429 → backoff 2^attempt (max 5); persistent quota → stop inspecting, set inspect_quota_capped=true, and classify the rest as coverage_unknown (never fabricate a state). 403 → AUTH_NO_SITE_ACCESS.
  • Sitemap/crawl fetch 429/5xx → backoff 2^attempt (max 4). Crawl paces 150ms/request; concurrency 1 (sequential BFS).
  • Cost guards: max_inspect, max_crawl, max_sitemaps, max_urls.

MISSING / INSUFFICIENT DATA

  • No coverage at all (coverage_source="none") → status="insufficient": the sitemap↔crawl↔traffic ledger is still produced (orphans, crawl errors, sitemap gaps), but index-state classes are reported as coverage_unknown rather than guessed.
  • No traffic file → indexed_earning collapses into indexed_no_traffic; orphan detection needs both traffic and crawl inlinks and is skipped otherwise. NEVER invent index state or clicks.

OUTPUT

One JSON object per references/output.schema.json. No prose.

FILES

  • scripts/indexation_audit.py — sitemap parse, URL Inspection, crawl, ledger join, classification, gap clustering.
  • references/output.schema.json — output contract.

Not using the CLI? Copy the SKILL.md and paste it straight into ChatGPT, Claude, or any agent.

View seoskills-sh/catalog on GitHub

Embed a badge

seoskills.sh listing badge