Firecrawl x Dify.AI: Add web data to your RAG applications

We are excited to announce that Firecrawl now integrates seamlessly with Dify, empowering your RAG (Retrieval-Augmented Generation) applications like never before!

Dify is an open-source platform for developing LLM (Large Language Model) applications. It allows you to orchestrate everything from simple agents to complex AI workflows, all powered by a robust RAG engine. With Firecrawl, you can now add web data to your RAG applications - all inside Dify’s platform.

Firecrawl enables you to convert any website into clean, LLM-ready data effortlessly. To use it in Dify, configure your settings in the Knowledge dashboard. There you can set your Firecrawl API key and start crawing right away.

To start using it, pass the URL you want to ingest by clicking the Sync from website button in the Knowledge section. Tailor your crawling process with options to set sub-page limits and depths. Use the Exclude and Include path options to ensure you capture exactly the data you need.

Firecrawl rapidly crawls web pages in parallel. After crawling, you can select the desired web data on Dify for preprocessing and cleaning. The processed data is then embedded and stored in Dify’s vector DB as a new knowledge base.

Now, you can create a RAG app that uses web data as contextual knowledge on Dify with the power of Firecrawl. We value your feedback! Connect with us on X/Twitter @firecrawl_dev.