How to Build AI Agents with the Claude Agent SDK and Firecrawl

Bex TuychievBex Tuychiev
Nov 10, 2026 (updated)

TL;DR

  • What it is: the Claude Agent SDK gives Python and TypeScript programs the same tools, agent loop and context management that power Claude Code.
  • Where it came from: Anthropic first shipped it as the Claude Code SDK in June 2025 and renamed it the Claude Agent SDK on September 29, 2025.
  • How to install it: pip install claude-agent-sdk (Python 3.10+) or npm install @anthropic-ai/claude-agent-sdk. Both packages bundle the Claude Code binary.
  • How it reaches the web: an MCP server such as the Firecrawl MCP server plugs into mcp_servers and gives the agent search, scrape and interact tools that return clean Markdown.
  • Subagents and structured output: AgentDefinition declares subagents that Claude calls through its Agent tool, and output_format with a JSON Schema returns validated JSON in ResultMessage.structured_output.
  • Release status: PyPI still lists the Python package as Alpha (version 0.2.180 on November 9, 2026), so pin your version.

Claude Code is great until you need an agent that runs without you. It's a CLI tool, built for interactive sessions. If you want the same agent loop inside a Python script that runs on a schedule or plugs into a pipeline, you need the Claude Agent SDK.

This tutorial covers the SDK by building a dependency auditor. It reads a requirements.txt, sends parallel subagents to research each package on PyPI and GitHub through a Firecrawl MCP server, and produces a structured risk report. Every SDK feature we touch transfers to whatever you build after.

What is the Claude Agent SDK?

The Claude Agent SDK is Anthropic's library for running the Claude Code agent inside your own Python or TypeScript application. It gives your code the same built-in tools, permissions, sessions, hooks and subagents that Claude Code uses, so an agent can read files, run commands and search the web with nobody at the terminal.

Anthropic introduced Claude Code as a command line tool for agentic coding in February 2025. The SDK's Python package first shipped as claude-code-sdk in June 2025, extracting that agent loop into a library. On September 29, 2025, Anthropic renamed it the Claude Agent SDK, because the same harness can power finance, customer support and deep research agents as well as coding agents.

At a high level, the SDK lets your scripts do what Claude Code does. The agent handles file operations and shell commands on its own, can search the web for information, and coordinates subtasks by dispatching subagents. You configure what it's allowed to do, point it at external tools through MCP servers or CLI skills, and let it run. If you'd rather extend Claude Code interactively instead of building a standalone script, Claude Code automation through the skills system is built for that workflow.

Claude Agent SDK architecture showing a Python script calling query() into an agent loop with built-in tools, MCP servers like Firecrawl, and subagents, returning structured JSON output via ResultMessage

How is the Claude Agent SDK different from Claude Code and the Claude API?

The Claude Agent SDK runs Claude Code's agent inside a process you operate, while Claude Code itself is the terminal app for interactive work. The Claude API, through Anthropic's Client SDKs, gives you the model alone, so you write the tool loop yourself. Pick the Agent SDK when you want Claude Code's tools and loop without a person typing.

You want toUseWhat you get
Embed Claude Code's agent in your own Python or TypeScript appClaude Agent SDKA library that runs the Claude Code binary, with built-in tools, permissions, sessions and hooks
Work interactively or run one-off tasks from a terminalClaude Code CLIThe terminal interface, built for daily use
Call the model directly from your own codeClient SDK (Claude API)Direct API access; you write the tool loop yourself or use the beta tool runner

The Agent SDK covers more than the dependency auditor below needs. Anthropic's Agent SDK overview lists what it includes:

CapabilityWhat it does
Built-in toolsRead, write and edit files, run commands, search the web
HooksRun your own code at key points in the agent lifecycle
SubagentsSpawn specialized agents for focused subtasks
MCPConnect external tools and data sources such as Firecrawl
PermissionsDecide which tools run automatically and which need approval
SessionsKeep context across exchanges, then resume or fork later
Skills, commands and memoryLoad from .claude/ and ~/.claude/, the same as Claude Code
PluginsPackage skills, agents, hooks and MCP servers and load them by path

Anthropic's docs also note that apps built on the SDK should authenticate with an API key rather than a claude.ai login unless Anthropic has approved it, so usage is billed through the Claude API.

Building a web scraping agent with the Claude Agent SDK

The fastest way to learn the SDK is to wire something up with it. Our project is a dependency auditor that reads a requirements.txt and produces a risk report for each package. I know tools like pip-audit already handle this, but the audit isn't the point. What matters here is how the SDK handles subagents, structured output and MCP integration.

The full script is on GitHub. Open it in a separate tab for reference as we break it down section by section. The gist was written against an earlier SDK release, so where it differs from the snippets below (the model ID, the Agent tool name, the deprecated debug_stderr option), follow the snippets.

This tutorial uses the Firecrawl MCP server for web access. If you prefer the CLI approach, there's a CLI version of the script that uses the Firecrawl CLI as a skill instead. The case for CLIs over IDEs applies here too.

Prerequisites

You need Python 3.10+ and Node.js (the Firecrawl MCP server runs via npx). The Python package bundles the Claude Code binary, so there is no separate Claude Code install. Install the dependencies:

pip install claude-agent-sdk pydantic rich python-dotenv

If you're going with the CLI version, you'll also need to install and register the Firecrawl CLI as a skill:

npm install -g firecrawl-cli
npx -y firecrawl-cli@latest init --all --browser

Create a .env file with your API keys:

FIRECRAWL_API_KEY=your-key-here
ANTHROPIC_API_KEY=your-key-here

The SDK reads ANTHROPIC_API_KEY from the environment automatically. We'll load FIRECRAWL_API_KEY with python-dotenv and pass it to the MCP server config.

The sections below cover the SDK-specific parts of the script. Utility functions for parsing requirements.txt, rendering the Rich table, and saving the markdown report are in the full script. You'll need all of them to run the auditor.

How do you get structured output from the agent?

Before the agent researches anything, we need to define what a report looks like. Two Pydantic models handle this:

from pydantic import BaseModel
 
class PackageReport(BaseModel):
    name: str
    pinned_version: str = ""
    latest_version: str = ""
    last_release: str = ""
    last_commit: str = ""
    status: str = ""        # active | slowing down | stale | abandoned
    risk: str = ""          # low | medium | high
    summary: str = ""
    replacement: str = ""
 
class AuditResult(BaseModel):
    packages: list[PackageReport]

Agents return text by default. Ask one to "audit flask" and you'll get a paragraph you have to parse yourself. output_format changes that. We'll pass AuditResult.model_json_schema() to the SDK, and the agent returns JSON matching this schema instead of prose. AuditResult wraps the full package list so everything arrives in one response, and we validate it into Python objects on our end.

The replacement field defaults to an empty string since it only applies to abandoned packages. Defaults matter because the agent will skip fields it considers irrelevant unless you give it a fallback value.

If the agent can't produce output that matches the schema after its retries, a single-shot query() yields an error result with the subtype error_max_structured_output_retries and then raises, so wrap the loop in error handling before you rely on structured_output.

The agent doesn't always return the exact enum values from your field comments. We got "outdated" instead of "stale" in early testing. If strict validation matters, add a Pydantic validator or post-process the results. The same thing happens with LLM-based data extraction generally.

How do system prompts shape agent behavior?

The agent is general-purpose out of the box. The system prompt turns it into a specialist. Ours needs classification logic for rating packages and a coordination strategy for its subagents:

AUDITOR_PROMPT = """You are a Python dependency auditor coordinating a team of
researchers. You will receive a list of Python packages to audit.
 
For EACH package, dispatch a researcher subagent using the Agent tool. Dispatch
ALL packages at once so they run in parallel.
 
After all researchers report back, classify each package:
 
- active: released within the last 6 months
- slowing down: last release 6-18 months ago
- stale: last release 18 months to 2 years ago
- abandoned: no release in 2+ years, or repo archived
 
Risk levels:
- low: active, no known issues
- medium: slowing down or minor concerns
- high: stale/abandoned, security risks, or no maintained alternative
 
If stale or abandoned, suggest a replacement. Keep summaries to one sentence.
Return the structured audit result with all packages."""

The "dispatch ALL packages at once" line matters. Without it, the agent researches packages sequentially, which takes much longer with eight dependencies.

Each subagent gets a shorter prompt focused purely on data gathering:

RESEARCHER_PROMPT = """You are a Python package researcher. For the package you're
given, use the Firecrawl tools to check its PyPI page for latest version and
release dates, and its GitHub repo for recent commits and activity. Return a
detailed text summary of your findings."""

The researcher prompt is deliberately narrow. It scrapes data and reports back without knowing the classification criteria or risk levels. That separation lets the orchestrator see results from all eight packages when it makes its assessment, while each researcher focuses on getting accurate data for one.

How do you connect an MCP server?

MCP servers are how you give the agent tools beyond what's built in. The SDK has built-in web tools, but they return raw page content the agent has to sift through. Firecrawl strips pages down to clean markdown and handles JS rendering, so the researchers get usable data from PyPI and GitHub instead of HTML noise. It plugs in through the mcp_servers parameter:

import os
 
mcp_servers = {
    "firecrawl": {
        "command": "npx",
        "args": ["-y", "firecrawl-mcp"],
        "env": {"FIRECRAWL_API_KEY": os.environ["FIRECRAWL_API_KEY"]},
    }
}

The SDK starts the Firecrawl server process, discovers its available tools (search, scrape, crawl, interact, agent and more), and registers them alongside the agent's built-in tools. The agent sees them with an mcp__firecrawl__ prefix, such as mcp__firecrawl__firecrawl_scrape. This first run downloads firecrawl-mcp via npx. After that it uses the cached version.

You can add multiple MCP servers to the same dictionary. Each one registers its own tools, and the agent picks from the combined set.

If you'd rather not run a local process, Firecrawl also hosts the server at https://mcp.firecrawl.dev/v2/mcp. The SDK accepts remote servers in the same dictionary with "type": "http" and a url. The hosted endpoint works keyless for Firecrawl Search, Scrape and Parse, and takes your API key as a bearer token for the other tools. The same Firecrawl tools are also available through the Python and Node SDKs, the REST API, the Firecrawl CLI and no-code integrations, so the research step isn't tied to this script.

Can the researchers get package data without scraping PyPI?

The researchers can pull PyPI data as structured records through Firecrawl Alexandria, which the Firecrawl MCP server reaches alongside the live web. Firecrawl Alexandria serves typed records from data providers, so a package lookup returns fields instead of a page.

Its package registries provider returns a package's latest version, publish time, license, repository link, version count and download numbers for npm, PyPI and crates.io.

Alexandria calls need a Firecrawl API key, which the local server config above passes; the hosted keyless tier doesn't include them. A researcher finds the provider with firecrawl_find_tools (for example providers: ["package-registry-metadata-download-stats"]), then calls firecrawl_scrape with an alexandria body naming the provider, the packages/package capability and the options {registry: "pypi", name: "envoy"}. The PyPI half of each audit comes back as data instead of a page to parse. Asking it for envoy on PyPI returns version 0.0.3, published August 13, 2014, which is the same signal our auditor flags as abandoned. The GitHub half (recent commits, archived status) still comes from scraping the repository page.

Using @firecrawl for a client's deep research agent has been amazing. I kept thinking, 'I should hook this into Claude Code.' Finally did it weeks later via MCP for my own research tasks. Why didn't I do this sooner?

— Tim (@timothyvh), developer

How do subagents work?

Parallel subagent dispatch flow where an orchestrator agent sends researcher subagents to audit flask, pydantic, httpx, and envoy packages through Firecrawl MCP scraping PyPI and GitHub, returning a Pydantic-validated AuditResult

AgentDefinition creates a named subagent the orchestrator can dispatch:

from claude_agent_sdk import AgentDefinition
 
agents = {
    "researcher": AgentDefinition(
        description="Researches a Python package's health using Firecrawl web tools",
        prompt=RESEARCHER_PROMPT,
    ),
}

Each subagent runs as a separate process with its own context window. That matters for two reasons:

  • The orchestrator's context doesn't get filled with raw scraped HTML from eight different PyPI pages.
  • Each researcher works independently so a slow scrape on one package doesn't block the others.

The description field is how the orchestrator decides when to use this subagent. With the "dispatch ALL at once" instruction from the system prompt, it fires off eight researchers simultaneously. Each one hits PyPI and GitHub for its assigned package, then reports back. The orchestrator collects all results before producing the final classification.

Claude dispatches subagents through its Agent tool, which is why the system prompt names it. If you want a subagent to use a different model, AgentDefinition accepts a model parameter: an alias ("sonnet", "opus", "haiku" or "inherit") or a full model ID. It also takes a tools list, so you can limit researchers to the Firecrawl tools and keep them away from shell commands.

How does the query() loop run the agent?

All the pieces come together in ClaudeAgentOptions:

from claude_agent_sdk import ClaudeAgentOptions
 
options = ClaudeAgentOptions(
    model="claude-sonnet-5-5",
    system_prompt=AUDITOR_PROMPT,
    permission_mode="bypassPermissions",
    max_turns=25,
    output_format={
        "type": "json_schema",
        "schema": AuditResult.model_json_schema(),
    },
    mcp_servers={...},  # Firecrawl config from above
    agents={...},       # researcher definition from above
)

permission_mode="bypassPermissions" is the one to pay attention to. In the default mode, any tool call that needs approval goes to a can_use_tool callback, which makes sense when a human is watching, but a non-interactive script needs to run unattended. This mode runs tools without prompts, apart from a short list of actions no mode auto-approves. For a narrower setup, Anthropic's permissions guide pairs permission_mode="dontAsk" with allowed_tools=["mcp__firecrawl__*"]: every Firecrawl tool is approved, and any other call that needs approval is denied. allowed_tools on its own only auto-approves. It doesn't remove other tools, and under bypassPermissions it changes nothing. To take a tool away in every mode, list it in disallowed_tools.

max_turns=25 caps the number of agent loop iterations. Each tool call, subagent dispatch, or reasoning step counts as a turn. With eight subagents making multiple scrape calls each, 25 gives enough headroom without letting a confused agent spin forever.

output_format connects to the Pydantic schema from earlier. The agent's final response comes back as JSON matching AuditResult instead of free-form text. Older versions of this script passed debug_stderr to silence subprocess logs; that option is deprecated and no longer read, so leave it out and pass a stderr callback when you want the logs.

The query() call itself is an async generator:

import json
from claude_agent_sdk import ResultMessage, query
 
async def run(path: Path) -> list[PackageReport]:
    packages = parse_requirements(path)
    # ... options config from above ...
 
    pkg_list = "\n".join(
        f"- {name}=={version}" if version else f"- {name}"
        for name, version in packages
    )
    prompt = f"Audit these Python packages:\n\n{pkg_list}"
    result = None
 
    console = Console(stderr=True, force_terminal=True)
 
    with console.status("[bold blue]Auditing packages..."):
        async for message in query(prompt=prompt, options=options):
            if isinstance(message, ResultMessage) and message.structured_output:
                data = message.structured_output
                if isinstance(data, str):
                    data = json.loads(data)
                result = AuditResult.model_validate(data)
 
    return result.packages if result else []

query() yields messages as the agent works: tool calls, subagent dispatches, intermediate reasoning. Most of these you can ignore. The one that matters is ResultMessage at the end, which carries the structured output. We validate it into an AuditResult with model_validate and pull the package list out.

Don't return inside the async for loop, a gotcha that cost some debugging time. It triggers a RuntimeError from the SDK's cancel scope handling. Collect the result in a variable and let the generator exhaust on its own.

Console(stderr=True, force_terminal=True) keeps the Rich spinner visible while the agent runs. The SDK captures stdout for its own communication, so any UI output you want the user to see needs to go through stderr.

Output

The requirements.txt we give the auditor mixes healthy and suspect packages on purpose:

flask==3.1.1
pydantic==2.11.4
httpx==0.28.1
ruff==0.11.12
chardet==5.2.0
pylint==3.3.6
nose2==0.15.1
envoy==0.0.3

Run it:

python audit.py requirements.txt

The orchestrator dispatches eight researchers in parallel. After roughly a minute, you get a color-coded table sorted by risk. The script also saves a markdown report next to the input file.

Terminal output of the dependency auditor showing a Rich table with package names, versions, status, and color-coded risk levels from the Claude Agent SDK script

In the original March 2026 run, a high-risk package and a healthy one came back like this:

envoy (archived since 2018):

FieldValue
Pinned0.0.3
Latest0.0.3
Statusabandoned
Riskhigh
SummaryRepository archived in 2018, last commit in 2014.
Replacementsubprocess (stdlib) or sh

pydantic (active):

FieldValue
Pinned2.11.4
Latest2.12.5
Statusactive
Risklow
SummaryNear-daily commits, 2.13.0 beta in progress.

Check the summaries before you trust them. For envoy, the agent wrote "last commit in 2014", but 2014 is the last PyPI release (0.0.3, August 13, 2014). The GitHub page shows the last commit on January 15, 2017, and the repository archived on April 24, 2018. The classification was still right, and the Pydantic schema can't catch a slip like this, so spot-check the facts behind any high-risk call.

Conclusion

The Python SDK is still marked Alpha on PyPI and ships several releases a week, so pin your version and check the changelog before upgrading. If something breaks, pass a stderr callback in the options config to see what the agent is doing. The full auditor script is 220 lines. Swap out the classification logic for whatever problem you're working on and the scaffolding stays the same.

If you'd rather extend Claude Code itself than build a standalone agent, the Firecrawl plugin for Claude adds the same web tools to your sessions, and the roundup of open source agent frameworks compares other ways to build AI agents.

To try web access without signing up, point the agent at Firecrawl's keyless MCP endpoint. When you need crawl, map or the Agent endpoint, sign up for a free Firecrawl account and grab an API key.

Use Firecrawl in your agent

Add Firecrawl to ChatGPT, Claude, Cursor and more in one click.

Frequently Asked Questions

What is the Claude Agent SDK?

The Claude Agent SDK is Anthropic's library for running the Claude Code agent loop inside your own Python or TypeScript program. It comes with the same built-in tools, permissions, sessions, hooks and subagents as Claude Code, so a script can hand Claude a task and let it read files, run commands, search the web and call MCP servers until the task is done.

How is the Claude Agent SDK different from Claude Code and the Claude API?

Claude Code is the terminal app for interactive, day-to-day work. The Claude Agent SDK embeds that same agent in an application you run yourself. The Client SDKs call the Claude API directly, so you write the tool loop on your own. Pick the Agent SDK when you want Claude Code's tools and agent loop without a person at the keyboard.

Is the Claude Agent SDK free?

Installing the SDK from PyPI or npm costs nothing, but every run calls Claude, so model usage is billed to the API key you configure. Anthropic's docs say products built on the SDK should authenticate with an API key, not a claude.ai login, unless Anthropic has approved it.

How much does running the auditor cost?

Each run dispatches eight subagents, each making 2-4 Firecrawl scrape calls and several Claude API calls. In the original test run with Sonnet as the model, a full audit of eight packages cost roughly $0.50-1.50 depending on how much page content the agent processed. The Firecrawl free plan includes 1,000 Firecrawl credits a month. Using Haiku for the researcher subagents brings the Claude API cost down further.

Is the Claude Agent SDK production-ready?

Treat it as early software. PyPI still lists the Python package as Alpha (version 0.2.180 on November 9, 2026), and it ships several releases a week. The core patterns (query, subagents, MCP, structured output) have stayed stable, but options get deprecated between versions. Pin your dependency version, test after upgrades, add retry logic around the query call, and validate structured output more strictly than this demo does.

Can I use the Firecrawl CLI instead of the MCP server?

Yes. Install the Firecrawl CLI with npm install -g firecrawl-cli, then swap mcp_servers for setting_sources=["user"] in ClaudeAgentOptions. The agent discovers the CLI as a skill and calls it through shell commands instead of the MCP protocol. The CLI is simpler to set up but lives outside your script, while MCP keeps everything in one Python file with no global installs.

How do I debug what the agent is doing?

Pass a stderr callback to ClaudeAgentOptions to receive the Claude Code subprocess logs, which show tool calls, subagent dispatches and errors. The older debug_stderr option is deprecated and no longer read. You can also iterate over every message that query() yields, not only the ResultMessage, to see intermediate steps in your own code.

What happens if a subagent fails?

The orchestrator handles it. If a researcher can't scrape a PyPI page (rate limiting, a network error, or a page that doesn't exist), it reports back what it found or didn't find, and the orchestrator classifies with whatever data is available. The fallback in our script creates a PackageReport with status unknown and risk medium if the entire query() call returns nothing.

Can I use different models for the orchestrator and subagents?

Yes. Set the orchestrator model on ClaudeAgentOptions.model with a full model ID such as claude-sonnet-5-5. Set subagent models on AgentDefinition.model, which takes an alias (sonnet, opus, haiku or inherit) or a full model ID. A common setup is Opus for the orchestrator, which reasons across all the results, and Haiku for researchers, since data gathering doesn't need the strongest model.

Does the SDK work with TypeScript?

Yes. The Claude Agent SDK ships Python and TypeScript versions with nearly identical APIs, and the TypeScript package is @anthropic-ai/claude-agent-sdk on npm. query(), the options object, agent definitions and result messages all have TypeScript equivalents. The Firecrawl MCP server and CLI work the same way from either language, so porting this tutorial to TypeScript is mostly syntax changes.

//
Get started
//
Ready to build?
Start getting web context for free and scale seamlessly as your project expands. No credit card needed.
Are you an AI agent? See setup options