Introducing Fire Engine for Firecrawl

Firecrawl handles web scraping orchestration but doesn’t do the actual scraping. It initially relied on third-party services like Fetch and Playwright for data retrieval. However, these services often failed on certain sites or were too slow, causing issues for users. To address this, we built Fire Engine, now the default backend for Firecrawl. It’s designed to be more reliable and faster, solving the core problems we and our users encountered with other scraping services.

Fire Engine is a scraping primitive designed to increase Firecrawl’s scraping capabilities.

We’re proud to say that Fire Engine outperforms leading competitors in key areas:

• Reliability: 40% more reliable than scraping leading competitors when scraping different types of websites

• Speed: Up to 33.17% faster than scraping leading competitors

And this is just the beginning, we are working closely with Firecrawl users to further improve reliability, speed, and more.

Fire Engine combines a variety of browser and non-browser based techniques to balance speed and reliability, ensuring that you get data back without compromise. To do this, Fire engine has:

• Efficient Headless Browser Management: Running browsers at scale is notoriously difficult, but Fire Engine handles this with ease.
• Persistent Browser Sessions: By keeping browsers running, Fire Engine improves efficiency when handling new requests, reducing startup times and resource usage.
• Advanced Web Interaction Techniques: Employing a sophisticated array of methods—including browser-based, browserless, and proprietary approaches
• Intelligent Request Handling: From smart proxy selection to advanced queuing, every aspect of the request process is optimized for speed and reliability.

With this technology, Fire Engine allows firecrawl to handle millions of requests daily with speed and accuracy.

Fire Engine powers Firecrawl to handle thousands of daily requests efficiently. It’s currently available exclusively through Firecraw cloud, Developers can test Fire Engine’s capabilities by signing up for Firecrawl.