Firecrawl Scraping

Overview

Scrape individual web pages and convert them to clean, LLM-ready markdown. Handles JavaScript rendering, anti-bot protection, and dynamic content.

Quick Decision Tree

What are you scraping? │ ├── Single page (article, blog, docs) │ └── references/single-page.md │ └── Script: scripts/firecrawl_scrape.py │ └── Entire website (multiple pages, crawling) └── references/website-crawler.md └── (Use Apify Website Content Crawler for multi-page)

Environment Setup

Required in .env

FIRECRAWL_API_KEY=fc-your-api-key-here

Get your API key: https://firecrawl.dev/app/api-keys

Common Usage

Simple Scrape

python scripts/firecrawl_scrape.py "https://example.com/article"

With Options

python scripts/firecrawl_scrape.py "https://wsj.com/article"
--proxy stealth
--format markdown summary
--timeout 60000

Proxy Modes

Mode Use Case

basic

Standard sites, fastest

stealth

Anti-bot protection, premium content (WSJ, NYT)

auto

Let Firecrawl decide (recommended)

Output Formats

• markdown

• Clean markdown content (default)

• html

• Raw HTML

• summary

• AI-generated summary

• screenshot

• Page screenshot

• links

• All links on page

Cost

~1 credit per page. Stealth proxy may use additional credits.

Security Notes

Credential Handling

• Store FIRECRAWL_API_KEY in .env file (never commit to git)

• API keys can be regenerated at https://firecrawl.dev/app/api-keys

• Never log or print API keys in script output

• Use environment variables, not hardcoded values

Data Privacy

• Only scrapes publicly accessible web pages

• Scraped content is processed by Firecrawl servers temporarily

• Markdown output stored locally in .tmp/ directory

• Screenshots (if requested) are stored locally

• No persistent data retention by Firecrawl after request

Access Scopes

• API key provides full access to scraping features

• No granular permission scopes available

• Monitor usage via Firecrawl dashboard

Compliance Considerations

• Robots.txt: Firecrawl respects robots.txt by default

• Public Content Only: Only scrape publicly accessible pages

• Terms of Service: Respect target site ToS

• Rate Limiting: Built-in rate limiting prevents abuse

• Stealth Proxy: Use stealth mode only when necessary (paywalled news, not auth bypass)

• GDPR: Scraped content may contain PII - handle accordingly

• Copyright: Respect intellectual property rights of scraped content

Troubleshooting

Common Issues

Issue: Credits exhausted

Symptoms: API returns "insufficient credits" or quota exceeded error Cause: Account credits depleted Solution:

• Check credit balance at https://firecrawl.dev/app

• Upgrade plan or purchase additional credits

• Reduce scraping frequency

• Use basic proxy mode to conserve credits

Issue: Page not rendering correctly

Symptoms: Empty content or partial HTML returned Cause: JavaScript-heavy page not fully loading Solution:

• Enable JavaScript rendering with --js-render flag

• Increase timeout with --timeout 60000 (60 seconds)

• Try stealth proxy mode for protected sites

• Wait for specific elements with --wait-for selector

Issue: 403 Forbidden error

Symptoms: Script returns 403 status code Cause: Site blocking automated access Solution:

• Enable stealth proxy mode

• Add delay between requests

• Try at different times (some sites rate limit by time)

• Check if site requires login (not supported)

Issue: Empty markdown output

Symptoms: Scrape succeeds but markdown is empty or malformed Cause: Dynamic content loaded after page load, or unusual page structure Solution:

• Increase wait time for JavaScript to execute

• Use --wait-for to wait for specific content

• Try html format to see raw content

• Check if content is in an iframe (not always supported)

Issue: Timeout errors

Symptoms: Request times out before completion Cause: Slow page load or large page content Solution:

• Increase timeout value (up to 120000ms)

• Use basic proxy for faster response

• Target specific page sections if possible

• Check if site is experiencing issues

Resources

• references/single-page.md - Single page scraping details

• references/website-crawler.md - Multi-page website crawling

Integration Patterns

Scrape and Analyze

Skills: firecrawl-scraping → parallel-research Use case: Scrape competitor pages, then analyze content strategy Flow:

• Scrape competitor website pages with Firecrawl

• Convert to clean markdown

• Use parallel-research to analyze positioning, messaging, features

Scrape and Document

Skills: firecrawl-scraping → content-generation Use case: Create summary documents from web research Flow:

• Scrape multiple article pages on a topic

• Combine markdown content

• Generate summary document via content-generation

Scrape and Enrich CRM

Skills: firecrawl-scraping → attio-crm Use case: Enrich company records with website data Flow:

• Scrape company website (about page, team page, product pages)

• Extract key information (funding, team size, products)

• Update company record in Attio CRM with enriched data