How to Create an llms.txt File for Any Website

Creating an llms.txt file for your website is now simpler than ever with the llms.txt Generator. This tool enables you to compile your entire website into a single text file that can be used with any Large Language Model (LLM), improving how AI interacts with your content.

The llms.txt Generator leverages Firecrawl to crawl your website and extracts data using gpt-4o-mini. You can generate both llms.txt and llms-full.txt files through the web interface or via API.

You can access llms.txt directly by making a GET request to:

http://llmstxt.firecrawl.dev/{YOUR_URL}

For the full version, use:

http://llmstxt.firecrawl.dev/{YOUR_URL}/full

If you have a Firecrawl API key, you can include it to unlock full results and remove limits:

http://llmstxt.firecrawl.dev/{YOUR_URL}?FIRECRAWL_API_KEY=YOUR_API_KEY

For the full version with API key:

http://llmstxt.firecrawl.dev/{YOUR_URL}/full?FIRECRAWL_API_KEY=YOUR_API_KEY

1. Visit the Generator: Go to http://llmstxt.firecrawl.dev.

2. Enter Your Website URL: Input the URL of your website.

3. Generate the File: Click the generate button and wait a few minutes as the tool processes your site.

4. Download Your Files: Once ready, download the llms.txt and llms-full.txt files.

While an API key is not required, using a free Firecrawl API key removes any usage limits and provides full access to all features.

• llms.txt Standard: Discover more about the llms.txt standard at http://llmstxt.org.

• Directory of Adopters: Check out the directory of llms.txt adopters at http://directory.llmstxt.cloud.

• Contribute: The llms.txt Generator is open source. Visit the GitHub repository to contribute: https://github.com/mendableai/llmstxt-generator.

• Firecrawl Official Website
• llms.txt Standard
• llms.txt Generator
• Directory of llms.txt Adopters
• llms.txt Generator GitHub Repository