Web Crawling API - Part of the Firecrawl context API

Firecrawl Crawl turns a domain into clean markdown your agent can read. Pass a URL and Crawl walks the subpages, scrapes each one in a real browser, and returns the same clean markdown for every page. One job, one webhook, one consistent output format across thousands of pages.

Teams building knowledge bases, RAG pipelines, AI agents, content migrations, and onboarding flows that turn a customer's URL into something the product can use. Over 1.25M developers and 150,000+ companies build with Firecrawl, including teams at Zapier, Botpress, Vercel, and Coinbase, and we've served more than 5 billion requests to date.

An in-house crawler is a few weeks of work and a long tail of maintenance: sitemap parsing, link discovery, JS rendering, rate limits, retries, proxies, robots.txt, and the HTML-to-markdown layer that LLMs actually need. Crawl ships all of that as one endpoint. The output is the same clean markdown across every page, ready to drop into an LLM. Customers consolidating from Zyte, Bright Data, and in-house tooling have done the math.

Use Scrape when the agent already has the URL. Use Map when you want to know what URLs exist on a site before paying to scrape them. Use Crawl when the input is a domain and the output is every page on the site — knowledge bases, full-site classification, content migrations. Search pairs well with Crawl when the agent has to discover the URL first.

Yes. Every page in the crawl renders in a real Chromium instance, so React, Vue, Next.js, and other SPA frameworks come back the same way they would in a browser tab. You get the full DOM, including content fetched after page load.

By default, Crawl reads the site's sitemap and follows links recursively from the root URL. You can switch sitemap handling to skip or only, follow links to subdomains or external sites, and ignore query parameters so the same path isn't re-scraped with different query strings.

Yes. Pass includePaths or excludePaths as regex patterns to control which paths are crawled, set maxDiscoveryDepth to cap how deep the crawler follows links, and use crawlEntireDomain or allowSubdomains when you want sibling paths or subdomains included.

Yes. Subscribe to webhook events (crawl.started, crawl.page, crawl.completed, crawl.failed) or a WebSocket to process pages the moment they're ready instead of waiting for the whole job. The SDKs also include a poll-and-wait method for simpler workflows.

Markdown by default. Pass scrapeOptions to request structured JSON via a schema, raw or clean HTML, full-page screenshots, links, and metadata for every page in the crawl. Every option that works on Scrape works on Crawl.

Yes. Production customers run Crawl at hundreds of concurrent browsers and millions of URLs per day. Jobs queue gracefully if you hit your limit, and Scale and Enterprise plans support 150 to 600+ browsers with dedicated rate limits and SLAs.

1 credit per page crawled. JSON mode adds 4 credits per page, and PDF parsing costs 1 credit per PDF page. The default crawl limit is 10,000 pages — set a lower limit to match your intended crawl size.

Yes. Free accounts include 1,000 credits per month, enough to crawl small docs sites and try every scrapeOption. No credit card required.

Yes. Firecrawl is open source with over 150K GitHub stars and 2.5M+ weekly downloads on npm and PyPI, and the self-hosted API shares the same SDKs and CLI. The open-source stack covers scrape, crawl, map, and search. It does not include Fire-engine, the managed proxy and anti-bot layer, so you bring your own proxies and handle blocked sites yourself, and screenshots, page actions, Agent, Browser, and Interact are hosted-only. Self-hosting fits cooperative sites when you already run infrastructure; the hosted API fits protected sites and zero ops. See docs.firecrawl.dev/contributing/open-source-or-cloud for the full comparison.