Firecrawl Agent
by firecrawl / cli
Autonomously navigate websites and extract structured data across pages.
Merges Ahrefs, Search Console and optional Semrush backlink data into one referring-domain set, with gained and lost domains and where the tools disagree.
Listed ·Updated
AGENT ROLE: Autonomous link-reconciliation agent. Fetch Ahrefs referring domains, ingest the GSC top-linking-sites export, optionally add Semrush, collapse every host to its registrable domain, and reconcile into one authoritative set with source attribution. Emit the JSON in references/output.schema.json. Stateful across runs via authoritative_domains.
Produce a single deduped referring-domain list that is honest about where each domain was seen (Ahrefs, GSC, Semrush), quantify how much each tool uniquely contributes, and surface the domains each tool misses plus the run-over-run gains and losses — so link audits stop trusting any one vendor's index.
target (REQUIRED): root domain to reconcile (e.g. example.com).gsc-links-csv (OPTIONAL but recommended): the Search Console "Top linking sites" CSV export. Absent → Ahrefs is returned without reconciliation.use-semrush (OPTIONAL flag): also pull Semrush referring domains (needs SEMRUSH_API_KEY).previous (OPTIONAL): prior run's output (its authoritative_domains drives gained/lost). Absent → baseline.max-domains (OPTIONAL, default 5000): cost guard on rows requested per vendor.verify-min-dr (OPTIONAL, default 30): Ahrefs-only domains at/above this DR are flagged for manual verification.AHREFS_API_TOKEN. IF unset THEN STOP error.code="AUTH_MISSING_AHREFS_TOKEN".
GET https://api.ahrefs.com/v3/site-explorer/refdomains with header Authorization: Bearer {token}, select=domain,domain_rating,dofollow_links,linked_domains,first_seen,last_seen.401 → STOP AUTH_INVALID_AHREFS_TOKEN; 403 → STOP AHREFS_FORBIDDEN.--gsc-links-csv. Do NOT claim GSC data without the CSV.--use-semrush AND SEMRUSH_API_KEY set: GET https://api.semrush.com/analytics/v1/?type=backlinks_refdomains (returns ;-delimited CSV). Missing key → degrade with sources.semrush="skipped_no_key".scripts/backlink_reconciler.py --target example.com --gsc-links-csv top_linking_sites.csv [--use-semrush] [--previous prev.json].STEP 1 — FETCH Ahrefs referring domains (capped at max-domains), collapsing subdomains to the registrable domain and keeping the strongest DR per domain.
STEP 2 — READ the GSC export; detect the host column (first column when unlabeled) and any linking-pages count; normalize hosts to registrable domains.
STEP 3 — OPTIONALLY fetch Semrush referring domains and normalize.
STEP 4 — UNION into an authoritative set; attribute each domain to the sources that reported it and attach the best available attributes (DR, GSC linking pages, first seen).
STEP 5 — PARTITION vs GSC: both, ahrefs_only, gsc_only; compute overlap and each tool's unique share.
STEP 6 — FLAG verification candidates: gsc_only (Google sees a link Ahrefs misses) and high-DR ahrefs_only (authority domain Google's export omits — possibly nofollow/disallowed/uncounted).
STEP 7 — DIFF vs previous.authoritative_domains → gained_domains, lost_domains. First run → status="baseline".
429 → backoff 2^attempt (max 5) then STOP RATE_LIMITED; 5xx retry ≤3.429/error the run continues with sources.semrush carrying the reason (never fabricated).--gsc-links-csv → reconciliation="skipped_no_csv"; return the Ahrefs set only, do not invent GSC coverage.sources.gsc="empty_export".previous) → status="baseline"; no gained/lost computed.One JSON object per references/output.schema.json. authoritative_domains MUST be persisted and passed back as the next run's --previous.
scripts/backlink_reconciler.py — Ahrefs + GSC + Semrush fetch, registrable-domain normalization, reconciliation and diff.references/output.schema.json — output contract.Not using the CLI? Copy the SKILL.md and paste it straight into ChatGPT, Claude, or any agent.