Firecrawl Observability

Overview

Monitor Firecrawl web scraping and crawling jobs for success rates, credit consumption, and extraction quality. Key signals include crawl job completion rate, pages scraped per credit, scrape latency (single page vs full crawl), credit burn velocity, and extraction success rate (did the markdown/structured data extraction return useful content).

Prerequisites

• Firecrawl account with API access

• Webhook endpoint for job status callbacks

• Metrics backend for tracking

Instructions

Step 1: Track Crawl Job Status via Webhooks

// firecrawl-webhook-handler.ts app.post('/webhooks/firecrawl', (req, res) => { const { jobId, status, pagesScraped, creditsUsed, failedUrls } = req.body; emitCounter('firecrawl_jobs_total', 1, { status }); emitGauge('firecrawl_pages_per_job', pagesScraped, { job: jobId }); emitCounter('firecrawl_credits_used', creditsUsed); if (failedUrls?.length > 0) { emitCounter('firecrawl_page_failures', failedUrls.length, { job: jobId }); } res.sendStatus(200); # HTTP 200 OK });

Step 2: Monitor Credit Consumption

set -euo pipefail

Check credit usage and remaining balance

curl https://api.firecrawl.dev/v1/usage
-H "Authorization: Bearer $FIRECRAWL_API_KEY" |
jq '{credits_used_today, credits_remaining, daily_avg: .credits_used_month / 30, days_until_empty: (.credits_remaining / (.credits_used_month / 30 + 0.01))}'

Step 3: Measure Extraction Quality

// Track whether scraped content is actually usable function evaluateExtractionQuality(result: any) { const markdown = result.markdown || ''; const metrics = { contentLength: markdown.length, hasHeadings: /^#{1,3}\s/m.test(markdown), hasContent: markdown.length > 200, # HTTP 200 OK isErrorPage: /404|not found|access denied/i.test(markdown), # HTTP 404 Not Found }; const quality = metrics.hasContent && !metrics.isErrorPage ? 'good' : 'poor'; emitCounter('firecrawl_extraction_quality', 1, { quality }); return quality; }

Step 4: Alert on Crawl Issues

groups:

• name: firecrawl rules:
  • alert: FirecrawlHighFailureRate expr: rate(firecrawl_page_failures[1h]) / rate(firecrawl_pages_per_job[1h]) > 0.2 annotations: { summary: "Firecrawl page failure rate exceeds 20%" }
  • alert: FirecrawlCreditBurnHigh expr: rate(firecrawl_credits_used[1h]) > 200 # HTTP 200 OK annotations: { summary: "Firecrawl burning >200 credits/hour" } # HTTP 200 OK
  • alert: FirecrawlCreditLow expr: firecrawl_credits_remaining < 500 # HTTP 500 Internal Server Error annotations: { summary: "Firecrawl credits below 500 -- refill soon" } # HTTP 500 Internal Server Error
  • alert: FirecrawlPoorExtraction expr: rate(firecrawl_extraction_quality{quality="poor"}[1h]) / rate(firecrawl_extraction_quality[1h]) > 0.3 annotations: { summary: "Over 30% of Firecrawl extractions returning poor quality" }

Step 5: Dashboard Panels

Track: crawl job success/failure rate, pages scraped per hour, credit consumption trend, extraction quality ratio, average scrape latency, and top failed domains (to identify sites blocking scraping).

Error Handling

Issue Cause Solution

High page failure rate Target site blocking bots Enable stealth mode or add delays between requests

Poor extraction quality Dynamic JS content not rendering Enable waitFor option to wait for page load

Credit burn spike Crawl job with no limit set Always set limit and maxDepth on crawl jobs

Webhook not firing Endpoint unreachable Verify endpoint URL and SSL certificate

Examples

Basic usage: Apply firecrawl observability to a standard project setup with default configuration options.

Advanced scenario: Customize firecrawl observability for production environments with multiple constraints and team-specific requirements.

Output

• Configuration files or code changes applied to the project

• Validation report confirming correct implementation

• Summary of changes made and their rationale

Resources

• Official monitoring documentation

• Community best practices and patterns

• Related skills in this plugin pack