Firecrawl Webhooks & Events

Overview

Handle Firecrawl webhooks for async crawl and scrape job notifications. Firecrawl supports native webhooks on crawl jobs via api.firecrawl.dev -- when a crawl completes, partially completes, or fails, Firecrawl POSTs results to your configured webhook URL.

Prerequisites

• Firecrawl API key stored in FIRECRAWL_API_KEY environment variable

• HTTPS endpoint accessible from the internet for webhook delivery

• Understanding of Firecrawl crawl vs scrape vs map endpoints

• Queue system for processing large crawl results

Webhook Event Types

Event Trigger Payload

crawl.completed

Full crawl finishes Array of page results, metadata

crawl.page

Individual page scraped Single page markdown/HTML content

crawl.failed

Crawl job errors Error message, partial results

crawl.started

Crawl begins Job ID, configuration

batch_scrape.completed

Batch scrape finishes Array of scraped pages

Instructions

Step 1: Start a Crawl with Webhook

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY!, });

const crawlResult = await firecrawl.asyncCrawlUrl("https://docs.example.com", { limit: 100, scrapeOptions: { formats: ["markdown", "html"], }, webhook: "https://api.yourapp.com/webhooks/firecrawl", });

console.log(Crawl started: ${crawlResult.id});

Step 2: Handle Webhook Events

import express from "express";

const app = express(); app.use(express.json());

app.post("/webhooks/firecrawl", async (req, res) => { const { type, id, data } = req.body;

res.status(200).json({ received: true }); # HTTP 200 OK

switch (type) { case "crawl.completed": await handleCrawlComplete(id, data); break; case "crawl.page": await handlePageScraped(id, data); break; case "crawl.failed": await handleCrawlFailed(id, data); break; } });

async function handleCrawlComplete(jobId: string, data: any) { const { pages, totalPages } = data; console.log(Crawl ${jobId} complete: ${totalPages} pages scraped);

for (const page of pages) { await indexPage({ url: page.metadata?.sourceURL, title: page.metadata?.title, markdown: page.markdown, scrapedAt: new Date(), }); } }

Step 3: Process Individual Pages

async function handlePageScraped(jobId: string, data: any) { const { markdown, metadata } = data;

// Extract structured data from scraped page await documentStore.upsert({ url: metadata.sourceURL, title: metadata.title, content: markdown, links: metadata.links || [], statusCode: metadata.statusCode, }); }

async function handleCrawlFailed(jobId: string, data: any) { console.error(Crawl ${jobId} failed: ${data.error});

await alerting.send({ severity: "high", message: Firecrawl job ${jobId} failed, error: data.error, partialResults: data.partialResults?.length || 0, }); }

Step 4: Poll as Fallback

async function pollCrawlStatus(jobId: string) { const status = await firecrawl.checkCrawlStatus(jobId);

if (status.status === "completed") { await handleCrawlComplete(jobId, { pages: status.data }); } else if (status.status === "scraping") { console.log(Progress: ${status.completed}/${status.total} pages); setTimeout(() => pollCrawlStatus(jobId), 5000); # 5000: 5 seconds in ms } }

Error Handling

Issue Cause Solution

Webhook not received URL unreachable Verify endpoint is publicly accessible

Partial results Page limit reached Increase limit or filter with includePaths

Rate limited Too many concurrent crawls Queue crawl jobs, respect rate limits

Timeout on large sites Crawl takes too long Use maxDepth and includePaths filters

Examples

Build Search Index from Crawl

async function buildSearchIndex(pages: any[]) { const documents = pages.map(page => ({ id: page.metadata.sourceURL, content: page.markdown, title: page.metadata.title, url: page.metadata.sourceURL, }));

await searchEngine.indexBatch(documents); console.log(Indexed ${documents.length} pages); }

Resources

• Firecrawl API Documentation

• Firecrawl Webhook Guide

• Firecrawl JS SDK

Next Steps

For deployment setup, see firecrawl-deploy-integration .

Output

• Configuration files or code changes applied to the project

• Validation report confirming correct implementation

• Summary of changes made and their rationale