Introducing Search

August 28, 2024

•

Eric Ciarla imageEric Ciarla

Launch Week I / Day 3: Introducing the Map Endpoint

Launch Week I / Day 3: Introducing the Map Endpoint image

Welcome to Day 3 of Firecrawl’s first-ever Launch Week! We’re thrilled to unveil the Map Endpoint.

Introducing the Map Endpoint (Alpha)

This powerful new endpoint allows you to transform a single URL into a comprehensive map of an entire website in record time. It’s the fastest and easiest way to gather all the URLs on a website, opening up new possibilities for your web scraping projects.

The Map endpoint is extremely useful when you need to quickly know the links on a website or when you need to scrape pages of a website that are related to a specific topic (using the search parameter).

Getting Started with the Map Endpoint

Using the new Map endpoint is straightforward. Here’s a quick example using our Python SDK:

from firecrawl.firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Map a website:
map_result = app.map_url('https://firecrawl.dev')
print(map_result)

The response will include a list of URLs found on the website:

{
  "status": "success",
  "links": [
    "https://firecrawl.dev",
    "https://www.firecrawl.dev/pricing",
    "https://www.firecrawl.dev/blog",
    "https://www.firecrawl.dev/playground",
    "https://www.firecrawl.dev/smart-crawl",
    ...
  ]
}

What’s Next?

We’re just getting started! The Map endpoint opens up exciting possibilities for future features and integrations. Stay tuned for the remaining two days of Launch Week, where we’ll be unveiling even more tools to supercharge your web scraping projects.

We can’t wait to see how you’ll use the Map endpoint in your projects. As always, we welcome your feedback and suggestions to help us improve and refine this new feature.

Happy mapping, and see you tomorrow for Day 4 of Launch Week!

Ready to Build?

Start scraping web data for your AI apps today.
No credit card needed.

About the Author

Eric Ciarla image
Eric Ciarla@ericciarla

Eric Ciarla is the Chief Operating Officer (COO) of Firecrawl and leads marketing. He also worked on Mendable.ai and sold it to companies like Snapchat, Coinbase, and MongoDB. Previously worked at Ford and Fracta as a Data Scientist. Eric also co-founded SideGuide, a tool for learning code within VS Code with 50,000 users.

More articles by Eric Ciarla

How to Create an llms.txt File for Any Website

Learn how to generate an llms.txt file for any website using the llms.txt Generator and Firecrawl.

Announcing Firestarter, our open source tool that turns any website into a chatbot

Spin up a fully functional RAG chatbot from any website URL using Firecrawl and Upstash—clean markdown in, OpenAI-compatible API out, all in under a minute.

Building Fire Enrich, our open source data enrichment tool

See how we built Fire Enrich, an open source tool that uses Firecrawl, OpenAI, and a multi-agent system to automate data enrichment — fully transparent, extensible, and built for developers.

Cloudflare Error 1015: How to solve it?

Cloudflare Error 1015 is a rate limiting error that occurs when Cloudflare detects that you are exceeding the request limit set by the website owner.

Build an agent that checks for website contradictions

Using Firecrawl and Claude to scrape your website's data and look for contradictions.

Why Companies Need a Data Strategy for Generative AI

Learn why a well-defined data strategy is essential for building robust, production-ready generative AI systems, and discover practical steps for curation, maintenance, and integration.

Getting Started with OpenAI's Predicted Outputs for Faster LLM Responses

A guide to leveraging Predicted Outputs to speed up LLM tasks with GPT-4o models.

How to easily install requests with pip and python

A tutorial on installing the requests library in Python using various methods, with usage examples and troubleshooting tips