Eric CiarlaCreating an llms.txt file for your website is now simpler than ever with the llms.txt Generator an open source example app powered by Firecrawl. This tool enables you to compile your entire website into a single text file that can be used with any Large Language Model (LLM), improving how AI interacts with your content.
What is an llms.txt File?
An llms.txt file is a standardized markdown file proposed by Jeremy Howard to provide information to help LLMs use a website at inference time. Unlike traditional web content designed for human readers, llms.txt files offer concise, structured information that LLMs can quickly ingest. This is particularly useful for enhancing development environments, providing documentation for programming libraries, and offering structured overviews for various domains such as corporate websites, educational institutions, and personal portfolios.
The llms.txt file is located at the root path /llms.txt of a website and contains sections in a specific order, including a project name, a summary, detailed information, and file lists with URLs for further details. This format allows LLMs to efficiently access and process the most important information about a website.
Introducing llms.txt Generator ✨
The llms.txt Generator leverages Firecrawl to crawl your website and extracts data using gpt-4o-mini. You can generate both llms.txt and llms-full.txt files through the web interface or via API.
Accessing llms.txt via API
You can access llms.txt directly by making a GET request to:
http://llmstxt.firecrawl.dev/{YOUR_URL}
For the full version, use:
http://llmstxt.firecrawl.dev/{YOUR_URL}/full
If you have a Firecrawl API key, you can include it to unlock full results and remove limits:
http://llmstxt.firecrawl.dev/{YOUR_URL}?FIRECRAWL_API_KEY=YOUR_API_KEY
For the full version with API key:
http://llmstxt.firecrawl.dev/{YOUR_URL}/full?FIRECRAWL_API_KEY=YOUR_API_KEY
How to Generate Your llms.txt File
-
Visit the Generator: Go to http://llmstxt.firecrawl.dev.
-
Enter Your Website URL: Input the URL of your website.
-
Generate the File: Click the generate button and wait a few minutes as the tool processes your site.
-
Download Your Files: Once ready, download the
llms.txtandllms-full.txtfiles.
No API Key Required, But Recommended
While an API key is not required, using a free Firecrawl API key removes any usage limits and provides full access to all features.
Learn More and Get Involved
-
llms.txt Standard: Discover more about the
llms.txtstandard at http://llmstxt.org. -
Directory of Adopters: Check out the directory of
llms.txtadopters at http://directory.llmstxt.cloud. -
Contribute: The llms.txt Generator is open source. Visit the GitHub repository to contribute: https://github.com/mendableai/llmstxt-generator.
References
data from the web