Launch Week I / Day 7: Crawl Webhooks (v1)
Welcome to Day 7 of Firecrawl’s Launch Week! We’re excited to introduce new /crawl webhook support.
Crawl Webhook
You can now pass a webhook parameter to the /crawl endpoint. This will send a POST request to the URL you specify when the crawl is started, updated and completed.
The webhook will now trigger for every page crawled and not just the whole result at the end.
Webhook Events
There are now 4 types of events:
crawl.started- Triggered when the crawl is started.crawl.page- Triggered for every page crawled.crawl.completed- Triggered when the crawl is completed to let you know it’s done.crawl.failed- Triggered when the crawl fails.
Webhook Response
success- If the webhook was successful in crawling the page correctly.type- The type of event that occurred.id- The ID of the crawl.data- The data that was scraped (Array). This will only be non empty oncrawl.pageand will contain 1 item if the page was scraped successfully. The response is the same as the/scrapeendpoint.error- If the webhook failed, this will contain the error message.
Learn More
Learn more about the webhook in our documentation.
About the Author
Nicolas Camara is the Chief Technology Officer (CTO) at Firecrawl. He previously built and scaled Mendable, one of the pioneering "chat with your documents" apps, which had major Fortune 500 customers like Snapchat, Coinbase, and MongoDB. Prior to that, Nicolas built SideGuide, the first code-learning tool inside VS Code, and grew a community of 50,000 users. Nicolas studied Computer Science and has over 10 years of experience in building software.
More articles by Nicolas Camara
Using OpenAI's Realtime API and Firecrawl to Talk with Any Website
Build a real-time conversational agent that interacts with any website using OpenAI's Realtime API and Firecrawl.
Extract website data using LLMs
Learn how to use Firecrawl and Groq to extract structured data from a web page in a few lines of code.
Getting Started with Grok-2: Setup and Web Crawler Example
A detailed guide on setting up Grok-2 and building a web crawler using Firecrawl.
Launch Week I / Day 6: LLM Extract (v1)
Extract structured data from your web pages using the extract format in /scrape.
Launch Week I / Day 7: Crawl Webhooks (v1)
New /crawl webhook support. Send notifications to your apps during a crawl.
OpenAI Swarm Tutorial: Create Marketing Campaigns for Any Website
A guide to building a multi-agent system using OpenAI Swarm and Firecrawl for AI-driven marketing strategies
Build a 'Chat with website' using Groq Llama 3
Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.
Scrape and Analyze Airbnb Data with Firecrawl and E2B
Learn how to scrape and analyze Airbnb data using Firecrawl and E2B in a few lines of code.