Using OpenAI's Realtime API and Firecrawl to Talk with Any Website

Interacting with any website through a conversational agent in real time is now possible thanks to OpenAI’s new Realtime API and Firecrawl. This powerful combination allows developers to build low-latency, multi-modal conversational experiences that can fetch and interact with live web content on the fly.

In this tutorial, we’ll guide you through the process of integrating Firecrawl’s scraping and mapping tools into the OpenAI Realtime API Console Demo. By the end, you’ll have a real-time conversational agent capable of talking with any website.

Prerequisites

Before you begin, make sure you have the following:

Node.js and npm installed on your machine.
An OpenAI API key with access to the Realtime API.
A Firecrawl API key.
Basic understanding of React and TypeScript.

Step 1: Clone the OpenAI Realtime API Console Demo

First, clone the repository that contains the OpenAI Realtime API Console Demo integrated with Firecrawl.

git clone https://github.com/nickscamara/firecrawl-openai-realtime.git
cd firecrawl-openai-realtime

Step 2: Install Dependencies

Install the required npm packages:

npm install

Step 3: Set Up Environment Variables

Create a .env file in the root directory and add your OpenAI and Firecrawl API keys:

OPENAI_API_KEY=your-openai-api-key
FIRECRAWL_API_KEY=your-firecrawl-api-key

If you’re running a local relay server, set the relay server URL:

REACT_APP_LOCAL_RELAY_SERVER_URL=http://localhost:8081

Step 4: Integrate Firecrawl Tools into the Realtime API Console Demo

Open the ConsolePage.tsx file located at src/pages/ConsolePage.tsx.

Import Firecrawl

At the top of the file, import the Firecrawl SDK:

import FirecrawlApp from '@mendable/firecrawl-js';

Add the ‘scrape_data’ Tool

Within the useEffect hook where tools are added to the client, add the scrape_data tool:

client.addTool(
  {
    name: 'scrape_data',
    description: 'Goes to or scrapes data from a given URL using Firecrawl.',
    parameters: {
      type: 'object',
      properties: {
        url: {
          type: 'string',
          description: 'URL to scrape data from',
        },
      },
      required: ['url'],
    },
  },
  async ({ url }: { url: string }) => {
    const firecrawl = new FirecrawlApp({
      apiKey: process.env.FIRECRAWL_API_KEY || '',
    });
    const data = await firecrawl.scrapeUrl(url, {
      formats: ['markdown', 'screenshot'],
    });
    if (!data.success) {
      return 'Failed to scrape data from the given URL.';
    }
    setScreenshot(data.screenshot || '');
    return data.markdown;
  }
);

This tool allows the assistant to scrape data from any URL using Firecrawl.

Add the ‘map_website’ Tool

Next, add the map_website tool to enable searching for pages with specific keywords on a website:

client.addTool(
  {
    name: 'map_website',
    description: 'Searches a website for pages containing specific keywords using Firecrawl.',
    parameters: {
      type: 'object',
      properties: {
        url: {
          type: 'string',
          description: 'URL of the website to search',
        },
        search: {
          type: 'string',
          description: 'Keywords to search for (2-3 max)',
        },
      },
      required: ['url', 'search'],
    },
  },
  async ({ url, search }: { url: string; search: string }) => {
    const firecrawl = new FirecrawlApp({
      apiKey: process.env.FIRECRAWL_API_KEY || '',
    });
    const mapData = await firecrawl.mapUrl(url, { search });
    if (!mapData.success || !mapData.links?.length) {
      return 'No pages found with the specified keywords.';
    }
    const topLink = mapData.links[0];
    const scrapeData = await firecrawl.scrapeUrl(topLink, {
      formats: ['markdown', 'screenshot'],
    });
    if (!scrapeData.success) {
      return 'Failed to retrieve data from the found page.';
    }
    setScreenshot(scrapeData.screenshot || '');
    return scrapeData.markdown;
  }
);

This tool allows the assistant to search a website for specific content and retrieve it.

Manage Screenshot State

At the top of your ConsolePage component, add state management for the screenshot:

const [screenshot, setScreenshot] = useState<string>('');

Display the Screenshot in the UI

In the UI, display the screenshot by adding the following within the appropriate JSX:

{ screenshot && <img src={screenshot} alt="Website Screenshot" /> }

Step 5: Run the Application

In a new terminal window, start the React application:

npm start

Open your browser and navigate to http://localhost:3000 to interact with your real-time conversational agent.

Testing the Agent

Now, you can test your agent by initiating a conversation. For example, ask:

User: “Can you get the latest blog post from https://mendable.ai?”

The assistant will use the scrape_data tool to fetch content from the specified URL and present it to you.

Conclusion

By integrating Firecrawl’s scraping and mapping tools into the OpenAI Realtime API Console Demo, you’ve created a powerful conversational agent capable of interacting with any website in real time. This setup opens up endless possibilities for building advanced AI applications that can access and process live web content on demand.