How do I crawl an entire website and get content for every page?

Point a crawl API at a starting URL and it follows every internal link automatically, returning the content of each page as clean markdown. No need to know the site structure or list URLs in advance.

Firecrawl's crawl endpoint handles discovery, JavaScript rendering, rate limiting, and robots.txt automatically:

from firecrawl import Firecrawl
from firecrawl.types import ScrapeOptions
 
firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")
 
result = firecrawl.crawl(
    "https://example.com",
    limit=100,
    scrape_options=ScrapeOptions(formats=["markdown"], only_main_content=True)
)

Use limit to cap pages, max_depth to control link depth, and include_paths / exclude_paths to scope the crawl to specific sections. For a quick one-off crawl from the terminal, the Firecrawl CLI also works: firecrawl crawl https://example.com --wait.