Firecrawl Agent
by firecrawl / cli
Autonomously navigate websites and extract structured data across pages.
Fetches authority, backlink, volume and difficulty metrics across Ahrefs, Semrush, Moz and DataForSEO in one pass, within a per-vendor credit budget.
Listed ·Updated
AGENT ROLE: Autonomous multi-vendor metrics agent. Route each requested (metric, target) to the vendors that provide it and are configured, coalesce calls that share an endpoint, enforce each vendor's credit budget, and return a unified source-attributed table plus a spend report. Emit the JSON in references/output.schema.json. Never fabricate a value a vendor did not return.
Get the same metric set from every configured vendor under a hard credit budget, attribute each value to its source, compute a cross-vendor consensus, and stay useful (partial + reasons) when a vendor is missing or spent — instead of failing the whole request.
domains (OPTIONAL): comma-separated domains (targets for domain_authority, backlinks).keywords (OPTIONAL): comma-separated keywords (targets for volume, difficulty).metrics (OPTIONAL, default all four): subset of domain_authority,backlinks,volume,difficulty.budgets (OPTIONAL): JSON object of vendor→credit budget, e.g. {"ahrefs":100,"semrush":50} (default 100 each).max-requests (OPTIONAL, default 1000): global network-call guard.Configure any of, and only those are used:
AHREFS_API_TOKEN (Bearer).SEMRUSH_API_KEY (optional SEMRUSH_DB, default us).MOZ_ACCESS_ID + MOZ_SECRET_KEY (HTTP Basic).DATAFORSEO_LOGIN + DATAFORSEO_PASSWORD (HTTP Basic).
IF none is configured THEN STOP error.code="AUTH_NO_VENDOR_CONFIGURED". A vendor whose creds are absent is simply skipped with reason unconfigured (never an error).scripts/metrics_orchestrator.py --domains example.com --keywords "seo tools" --metrics domain_authority,backlinks,volume,difficulty.url_metrics and Semrush phrase_this each return multiple metrics and are coalesced to a single call per target.STEP 1 — DETECT configured vendors; STOP if none. Validate requested metrics against the route table.
STEP 2 — BUILD the request set: for each metric, for each target of its type, list the supporting vendors in preference order.
STEP 3 — For each (vendor, endpoint-group, target): IF the group was already fetched for this target THEN reuse it (dedup/coalesce — no extra credit). ELSE check the vendor budget: IF spent + cost > budget THEN mark the vendor exhausted and skip (reason budget_exhausted); ELSE call, charging cost credits, with 2^attempt backoff.
STEP 4 — EXTRACT the metric from the group result. Collect one attributed value per vendor that answered.
STEP 5 — For each (metric, target) row: IF ≥1 value THEN status="ok" with sources, per-vendor values, and a consensus mean. ELSE status="unavailable" with the per-vendor reasons.
STEP 6 — EMIT the rows, a per-vendor spend_report (requests, credits spent, remaining, status), the count of coalesced calls saved, and degraded when any vendor was skipped/exhausted/rate-limited or any row is unavailable.
429 → backoff 2^attempt (max 5); persistent → vendor status rate_limited, its pending rows degrade (not the whole run).5xx retry ≤3; DataForSEO 402 → reason payment_required. Calls are paced (sleep 0.2); effective concurrency 1.no_value; the value is never invented.volume with no --keywords).unavailable with reasons — the run still returns status="partial".One JSON object per references/output.schema.json. No prose.
scripts/metrics_orchestrator.py — vendor routing, call coalescing, credit budgeting, unified table + spend report.references/output.schema.json — output contract.Not using the CLI? Copy the SKILL.md and paste it straight into ChatGPT, Claude, or any agent.