Firecrawl Agent
by firecrawl / cli
Autonomously navigate websites and extract structured data across pages.
Blends Semrush rankings with Search Console data into one table, flagging where the two disagree and the demand Semrush isn't tracking.
Listed ·Updated
AGENT ROLE: Autonomous rank-reconciliation agent. Fetch Semrush organic positions and the GSC search-analytics query/page table, join on the normalized keyword, and classify each keyword as agreement, disagreement, untracked demand, or Semrush-only. Emit the JSON in references/output.schema.json.
Reconcile the third-party rank (Semrush) against the first-party rank (GSC) for one domain and window, producing a single blended truth table: where the tools agree, where they disagree by more than the threshold (with direction and traffic-weighted impact), the real demand Semrush misses, and the Semrush keywords with no GSC demand.
site (REQUIRED): GSC property, e.g. sc-domain:example.com or a URL-prefix property.domain (REQUIRED): Semrush domain, e.g. example.com.database (OPTIONAL, default us): Semrush regional database.start / end (OPTIONAL): GSC window; default last 28 days ending 3 days ago (data lag).delta-threshold (OPTIONAL, default 3): min |Semrush position − GSC position| to count as a disagreement.min-impr (OPTIONAL, default 20): impression floor for a GSC query to count as untracked demand.max-keywords (OPTIONAL, default 10000) / max-rows (OPTIONAL, default 100000): cost/size guards.SEMRUSH_API_KEY. IF unset THEN STOP error.code="AUTH_MISSING_SEMRUSH_KEY".
GET https://api.semrush.com/?type=domain_organic&export_columns=Ph,Po,Nq,Cp,Ur,Tr — returns ;-delimited CSV (NOT JSON). A bad key returns a plain-text ERROR line → STOP SEMRUSH_API_ERROR.GSC_OAUTH_TOKEN or GCP_ACCESS_TOKEN (bearer, scope webmasters.readonly). IF unset THEN STOP error.code="AUTH_MISSING_GSC_TOKEN".
gcloud auth print-access-token); it cannot sign a service-account JWT. 401 → AUTH_INVALID_GSC_TOKEN; 403 → AUTH_NO_SITE_ACCESS.scripts/rank_blender.py --site sc-domain:example.com --domain example.com --database us.domain_organic pull; paginated GSC searchAnalytics/query with dimensions [query, page].STEP 1 — FETCH Semrush organic positions; key each row by the normalized keyword (lowercased, whitespace-collapsed): position, volume, CPC, ranking URL.
STEP 2 — FETCH GSC rows (query × page), paginate at 25000; aggregate to one record per query with summed clicks/impressions, the impression-weighted average position, and the top page by clicks.
STEP 3 — JOIN on the normalized keyword. For each common keyword compute position_delta = semrush_position − gsc_position (positive = Semrush shows a worse rank than reality).
STEP 4 — CLASSIFY: |delta| < threshold → agreement; otherwise disagreement tagged semrush_optimistic (Semrush better than reality) or semrush_pessimistic, ranked by impact = |delta| × impressions. Flag url_mismatch when the two tools name different ranking URLs.
STEP 5 — UNTRACKED DEMAND: GSC queries with impressions ≥ min-impr absent from Semrush.
STEP 6 — SEMRUSH-ONLY: Semrush keywords with no GSC demand (phantom volume, wrong locale/database, or not truly ranking for this site).
STEP 7 — EMIT summary (agreement rate, median |delta|) plus the four buckets, sorted by impact/impressions/volume.
429 → backoff 2^attempt (max 5) then STOP RATE_LIMITED; 5xx retry ≤3.max-rows; Semrush capped at max-keywords (both bill credits/rows).status="no_data".One JSON object per references/output.schema.json. No prose.
scripts/rank_blender.py — Semrush + GSC fetch, keyword join, disagreement and untracked-demand classification.references/output.schema.json — output contract.Not using the CLI? Copy the SKILL.md and paste it straight into ChatGPT, Claude, or any agent.