Firecrawl & Jina Web Scraping

Firecrawl vs WebFetch

Prefer firecrawl scrape URL --only-main-content over the WebFetch tool—it produces cleaner markdown, handles JavaScript-heavy pages, and avoids content truncation (>80% benchmark coverage). WebFetch is acceptable as a fallback when Firecrawl is unavailable.

# Preferred approach:
firecrawl scrape https://docs.example.com/api --only-main-content

Keyless Mode (v2.11.0+)

Firecrawl's core endpoints work without an API key, rate-limited per IP per day. No signup required.

Keyless-eligible commands: Search, Scrape, Parse, Interact Requires API key: Crawl, Map, Batch Scrape, Agent, Extract, Monitor

# Quick start — no API key needed:
firecrawl scrape https://example.com --only-main-content
firecrawl search "query" --limit 10

# Python script also works keyless for eligible commands:
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape https://example.com

Rate limits: Per-IP daily cap on requests and credits (exact numbers unpublished, below 1,000 credits/day). Sign up for a free API key to get 1,000 credits and higher rate limits.

Keyless MCP endpoint: https://mcp.firecrawl.dev/v2/mcp — exposes Search, Scrape, Parse without any key. Use for MCP-native integrations.

Full setup (recommended for heavier use):

npx -y firecrawl-cli@latest init --all --browser
# Installs CLI + skill segments into coding agents + opens browser for OAuth auth

Search accuracy: 94.7% on SimpleQA — custom relevance model scores paragraphs against your query, returning the most relevant excerpts with 10x fewer tokens than full-page processing.

Token-Efficient Scraping

Inspired by Anthropic's dynamic filtering—always filter before reasoning. This reduced input tokens by ~24% and improved accuracy by ~11% in their benchmarks.

The Principle: Search → Filter → Scrape → Filter → Reason

DO:

Search (titles/URLs only) → Evaluate relevance → Scrape top hits → Filter by section → Reason

DON'T:

Search → Scrape everything → Reason over all of it

Step-by-Step Efficient Workflow

# Step 1: Search — get titles/URLs only (cheap)
firecrawl search "query" --limit 20

# Step 2: Evaluate results, pick 3-5 best URLs

# Step 3: Scrape only those, filter to relevant sections
firecrawl scrape URL1 --only-main-content | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py \
  --sections "API,Authentication" --max-chars 5000

Post-Processing with filter_web_results.py

Pipe any Firecrawl or Exa output through this script to reduce context before reasoning:

# Extract only matching sections from scraped page
firecrawl scrape URL --only-main-content | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --sections "Pricing,Plans"

# Keep only paragraphs with keywords
firecrawl search "query" --scrape --pretty | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --keywords "pricing,cost" --max-chars 5000

# Extract specific JSON fields from API output
python3 ~/.claude/skills/exa-search/scripts/exa_search.py "query" --json | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --fields "title,url,text" --max-chars 3000

# Combine filters with stats
firecrawl scrape URL --only-main-content | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --sections "API" --keywords "endpoint" --compact --stats

Full path: python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py Flags: --sections, --keywords, --max-chars, --max-lines, --fields (JSON), --strip-links, --strip-images, --compact, --stats

Other Token-Saving Patterns

• Use --only-main-content to strip navigation and footer boilerplate, reducing token consumption. Omit only when nav/footer content is specifically needed.
• Use --only-clean-content (Python API script) for aggressive cleaning—strips nav, ads, and cookie banners. Stronger than --only-main-content; use when the page is still noisy after main-content filtering.
• Use firecrawl map URL --search "topic" first to find relevant subpages before scraping
• Use --format links first to get URL list, evaluate, then scrape selectively
• Use --max-chars with exa_contents.py to cap extraction length
• Use --formats summary (Python API script) over full text when you need the gist, not raw content

Claude API Native Tools (for API Agent Builders)

Anthropic's API now offers built-in dynamic filtering tools:

web_search_20260209 / web_fetch_20260209
Header: anthropic-beta: code-execution-web-tools-2026-02-09

These have built-in dynamic filtering via code execution. Use them when building Claude API agents directly. Use Firecrawl/Exa when you need: autonomous agents, batch scraping, structured extraction, domain-specific crawling, or when not on the Claude API.

---

Available Tools

1. Official Firecrawl CLI (firecrawl) — Primary

Keyless setup: npm install -g firecrawl-cli@latest (search/scrape/parse/interact work immediately) Full setup: npm install -g firecrawl-cli@latest && firecrawl login --api-key $FIRECRAWL_API_KEY Agent setup: npx -y firecrawl-cli@latest init --all --browser (installs CLI + skills + OAuth)

Command	Purpose	Quick Example
scrape	Single page → markdown	firecrawl scrape URL --only-main-content
crawl	Entire site with progress	firecrawl crawl URL --wait --progress --limit 50
map	Discover all URLs on a site	firecrawl map URL --search "API"
search	Web search, 94.7% SimpleQA accuracy (+ optional scrape)	firecrawl search "query" --limit 10

Full CLI reference: references/cli-reference.md

2. Auto-Save Alias (fc-save) — Shell Alias

Requires shell alias setup (not bundled with this skill).

fc-save URL
# → Saves to ~/Desktop/Screencaps & Chats/Web-Scrapes/docs-example-com-api.md

3. Python API Script (firecrawl_api.py) — Advanced Features

Command: python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py <command> Requires: pip install firecrawl-py requests. FIRECRAWL_API_KEY optional for keyless commands (search, scrape, parse, interact); required for all others.

Command	Purpose	Quick Example
search	Web search with scraping	firecrawl_api.py search "query" -n 10
scrape	Single URL with page actions	firecrawl_api.py scrape URL --formats markdown summary
batch-scrape	Multiple URLs concurrently	firecrawl_api.py batch-scrape URL1 URL2 URL3
crawl	Website crawling	firecrawl_api.py crawl URL --limit 20
map	URL discovery	firecrawl_api.py map URL --search "query"
parse	Parse local documents (PDF, DOCX, XLSX)	firecrawl_api.py parse report.pdf
extract	LLM-powered structured extraction	firecrawl_api.py extract URL --prompt "Find pricing"
agent	Autonomous extraction (no URLs needed)	firecrawl_api.py agent "Find YC W24 AI startups"
parallel-agent	Bulk agent queries (v2.8.0+)	firecrawl_api.py parallel-agent "Q1" "Q2" "Q3"
interact	Post-scrape browser interaction	firecrawl_api.py interact SCRAPE_ID --prompt "Click pricing"
interact-stop	Stop an interact session	firecrawl_api.py interact-stop SCRAPE_ID

Agent models: spark-1-fast (10 credits, simple), spark-1-mini (default), spark-1-pro (thorough)

Full Python API reference: references/python-api-reference.md

4. DeepWiki — GitHub Repo Documentation

~/.claude/skills/firecrawl/scripts/deepwiki.sh <owner/repo> [section] [options]

AI-generated wiki for any public GitHub repo. No API key required.

# Overview
~/.claude/skills/firecrawl/scripts/deepwiki.sh karpathy/nanochat

# Browse sections
~/.claude/skills/firecrawl/scripts/deepwiki.sh langchain-ai/langchain --toc

# Specific section
~/.claude/skills/firecrawl/scripts/deepwiki.sh karpathy/nanochat 4.1-gpt-transformer-implementation

# Full dump for RAG
~/.claude/skills/firecrawl/scripts/deepwiki.sh openai/openai-python --all --save

5. Jina Reader (jina) — Fallback

Use when Firecrawl fails or for Twitter/X URLs (Firecrawl blocks Twitter, Jina works).

jina https://x.com/username/status/123456

---

Firecrawl vs Exa vs Native Claude Tools

Need	Best Tool	Why
Free scrape, no API key	firecrawl scrape (keyless)	Core commands work without auth
Free search, no API key	firecrawl search (keyless)	94.7% SimpleQA accuracy, keyless
Single page → markdown	firecrawl scrape --only-main-content	Cleanest output
Search + scrape in one shot	firecrawl search --scrape	Combined operation
Crawl entire site	firecrawl crawl --wait --progress	Link following + progress
Local file → markdown	firecrawl_api.py parse FILE	Direct upload, no URL needed
Autonomous data finding	firecrawl_api.py agent	No URLs needed
Semantic/neural search	Exa exa_search.py	AI-powered relevance
Find research papers	Exa --category "research paper"	Academic index
Quick research answer	Exa exa_research.py	Citations + synthesis
Find similar pages	Exa exa_similar.py	Competitive analysis
Claude API agent building	Native web_search_20260209	Built-in dynamic filtering
Twitter/X content	jina URL	Only tool that works
GitHub repo docs	deepwiki.sh owner/repo	AI-generated wiki
Anti-bot / Cloudflare bypass	scrapling stealth fetch	Local Turnstile solver
Element-level extraction	scrapling + CSS selectors	Precision targeting, adaptive tracking
No API key scraping	scrapling HTTP fetch	100% local, no credentials
Site redesign resilience	scrapling adaptive mode	SQLite similarity matching
Budget JS-rendered scrape	cf_browser.py markdown URL	CF free tier: 10 min/day, $0.09/hr paid
Free static page fetch	cf_browser.py markdown URL --no-render	FREE during beta (no JS)
Budget multi-page crawl	cf_browser.py crawl URL	5 free crawls/day, 100 pages each
Incremental re-crawl	cf_browser.py crawl --modified-since	Built-in, Firecrawl lacks this
Page screenshot/PDF	cf_browser.py screenshot/pdf URL	Built-in CF endpoints, cheaper
AI structured extraction	cf_browser.py json URL --prompt "..."	Workers AI included free

---

Common Workflows

Single Page Scraping

firecrawl scrape https://example.com/page --only-main-content
# Or auto-save: fc-save URL
# Or to file: firecrawl scrape URL --only-main-content -o page.md

Documentation Crawling

# Map first, then crawl relevant paths
firecrawl map https://docs.example.com --search "API"
firecrawl crawl https://docs.example.com --include-paths /api,/guides --wait --progress

Research Workflow

firecrawl search "machine learning best practices 2026" --scrape --scrape-formats markdown

Document Parsing (Local Files)

Parse local documents into clean Markdown. Use parse for local or non-public files; use scrape for public URLs pointing to documents—both use the same Rust-based parser.

# PDF to markdown
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse report.pdf

# Excel spreadsheet with main content only
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse data.xlsx --only-main-content

# Word doc with zero data retention, save to file
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse contract.docx --zero-data-retention -o contract.md

# Raw JSON output for programmatic use
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse invoice.pdf --json

Supported formats: PDF, DOCX, DOC, XLSX, XLS, HTML, HTM, ODT, RTF (up to 50 MB).

PDF Parsing (Fire-PDF v2.9)

Fire-PDF is now the default parsing pipeline for all PDF scrapes. Three modes:

# Auto mode (default) — detects text layer vs scanned, chooses best method
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://example.com/report.pdf"

# Fast mode — text layer only, skip OCR (use for PDFs with selectable text)
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape URL --pdf-mode fast

# OCR mode — force full OCR (use for scanned docs or image-only PDFs)
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape URL --pdf-mode ocr

# Limit pages parsed (large documents)
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse report.pdf --pdf-mode auto --pdf-max-pages 20

Flag	Values	Notes
--pdf-mode	fast, auto, ocr	Default: auto. fast = text layer only; ocr = force OCR
--pdf-max-pages	integer	Caps pages parsed; useful for budget control on large PDFs

Both flags work on scrape (for PDF URLs) and parse (for local files).

Agent-Powered Research (No URLs Needed)

python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py agent \
  "Compare pricing tiers for Firecrawl, Apify, and ScrapingBee"

Interact Workflows (Post-Scrape Browser Interaction)

Scrape a page, then take actions on it—click buttons, fill forms, extract dynamic content. Two modes: AI prompts (natural language) and code execution (Node.js/Python/Bash).

When to Use Interact vs. Actions

Need	Use	Why
Click/wait before a single scrape	--actions on scrape	Fire-and-forget, no session overhead
Multiple interactions with same page	interact	Persistent session, back-and-forth
Fill forms, log in, navigate	interact	Stateful, multi-step
Simple "wait for JS to load"	--actions with wait	Cheaper, no session

Basic Interact (AI Prompt Mode)

# Step 1: Scrape and note the Scrape ID from output
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://example.com/pricing"

# Step 2: Interact using natural language
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --prompt "Click the Enterprise pricing tab"

# Step 3: More interactions on same session
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --prompt "What is the monthly price for the Enterprise plan?"

# Step 4: Stop when done
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact-stop SCRAPE_ID

Code Execution Mode (Cheaper)

# Execute Playwright code directly (2 credits/min vs 7 for prompts)
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --code "const text = await page.locator('.pricing-table').textContent(); console.log(text);"

# Python mode
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --code "text = await page.locator('.content').text_content(); print(text)" \
  --language python

Persistent Profile (Login Sessions)

# Scrape with a named profile — browser state persists across sessions
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://app.example.com/login" \
  --profile my-app --json

# Interact to log in
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --code "await page.fill('#email', 'user@example.com'); await page.fill('#password', 'pass'); await page.click('button[type=submit]');"

# Later: scrape another page with same profile — cookies restored
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://app.example.com/dashboard" \
  --profile my-app

Important: Interact does NOT return page markdown. To get updated content after interaction, use code mode to extract specific elements, or issue a follow-up scrape.

Full interact reference: references/interact-reference.md

---

Billing Notes

• Credit/token unification (v2.9): Credits and tokens are now unified—15 tokens = 1 credit. All pricing is expressed in credits.
• Default cache TTL: Results are cached for 2 days. Use --max-age 0 (or maxAge: 0 in API) to force a fresh scrape regardless of cache.
• query format: Pass formats=["query"] (Python API) to get a direct answer (data.answer) instead of full markdown. Use for factual lookups where you don't need the full page content.
• audio format: formats=["audio"] returns an MP3 of the page read aloud. Useful for accessibility pipelines or voice interfaces.
• wikimedia engine: Pass engine="wikimedia" in search options to route queries through Wikimedia. Useful for encyclopedic lookups.

Troubleshooting

# Check status and credits
firecrawl --status && firecrawl credit-usage

# Re-authenticate
firecrawl logout && firecrawl login --api-key $FIRECRAWL_API_KEY

# Check API key
echo $FIRECRAWL_API_KEY

• Scrape fails: Try jina URL, or add --wait-for 3000 for JS-heavy sites
• Async job stuck: Check with crawl-status/batch-status, cancel with crawl-cancel/batch-cancel
• Disable telemetry: export FIRECRAWL_NO_TELEMETRY=1

---

Reference Documentation

File	Contents
references/cli-reference.md	Full CLI parameter reference (scrape, crawl, map, search, fc-save, jina, deepwiki)
references/python-api-reference.md	Full Python API script reference (all commands, SDK examples)
references/firecrawl-api.md	Firecrawl Search API reference
references/firecrawl-agent-api.md	Agent API (spark models, parallel agents, webhooks)
references/actions-reference.md	Page actions for dynamic content (click, write, wait, scroll)
references/interact-reference.md	Interact API: post-scrape browser interaction (prompt, code, profiles)
references/branding-format.md	Brand identity extraction (colors, fonts, UI)

Test Suite

python3 ~/.claude/skills/firecrawl/scripts/test_firecrawl.py --quick    # Quick validation
python3 ~/.claude/skills/firecrawl/scripts/test_firecrawl.py            # Full suite
python3 ~/.claude/skills/firecrawl/scripts/test_firecrawl.py --test scrape  # Specific test