Introducing the Firecrawl Developer Index, built for supercharging coding agents. Read the announcement →
All benchmarks

Developer Retrieval
Benchmark 2026

Compare developer retrieval tools on how reliably they surface the repository behind a described capability, the pull request that fixed a bug, and the docs page that answers a how-to.

Last updated
[ Updated Aug 21, 2026 ]
Systems compared
[ 8 systems ]
Dataset
[ 1,179 tasks ]
[ 01 / 03 ]
·
Results
//
Results
//

How the systems stack up

8 systems on the same 1,179 tasks, ranked by overall recall.

Leaderboard

Overall recall with a 95% confidence interval. Switch tracks to see how the ranking changes.

Recall@10, 95% CI
Firecrawl Developer Index
0.631
[0.605, 0.657]
Parallel
0.577
[0.553, 0.602]
Firecrawl Search
0.576
[0.548, 0.603]
Mintlify
0.546
[0.520, 0.572]
Exa
0.537
[0.511, 0.563]
Native web search[ctl]
0.454
[0.430, 0.477]
Context7
0.168
[0.151, 0.185]
No tools[ctl]
0.013
[0.007, 0.020]

Full results

Every metric for all 8 systems. The best value in each column is highlighted, and control rows are reference points rather than products.

Developer Retrieval Benchmark results, ranked by Overall recall.
#SystemRepo (higher is better)Issue/PR (higher is better)Docs (higher is better)Overall recall (higher is better)Overall MRR (higher is better)
01Firecrawl Developer IndexFirecrawl's curated developer index of issues, PRs, READMEs, and docs, queried through its MCP server.76.1%66.0%47.2%63.1%0.596
02Parallel81.9%62.9%28.2%57.7%0.561
03Firecrawl SearchFirecrawl's general web search (/search), a different surface from the developer index. Included here for context.77.6%56.5%38.6%57.6%0.549
04Mintlify74.3%56.0%33.4%54.6%0.538
05Exa73.3%58.5%29.3%53.7%0.538
06Native web searchcontrolThe agent's own built-in web search, not a developer index. A reference point rather than a product.80.7%27.5%28.0%45.4%0.458
07Context7Docs-only index, second on the docs track. Read its overall score as coverage, not quality.0.8%2.9%46.6%16.8%0.144
08No toolscontrolThe agent answering from memory with no search at all. The floor every row is read against.0.5%0.0%3.4%1.3%0.009
[ 02 / 03 ]
·
Methodology
//
Methodology
//

How is developer retrieval measured?

The dataset behind the three tracks, the two metrics, and how every system is driven and scored.
01

Three retrieval tracks

The retrieval tasks developer agents spend most of their tool calls on.

[ Repo ][ Docs ][ Issue / PR ]
02

Retrieval-forcing questions

Questions are written so the answer cannot come from memory: repository names are hidden and documentation questions are paraphrased from the source.

03

Deterministic gold references

Each item is scored against fixed gold references, so scoring is exact and no model judges the results.

04

Memorization check

Run the same evaluation with search disabled and drop every item the model can answer without retrieval.

[ Same questions, no search tools ][ Non-retrieval items dropped ]
05

Final benchmark

1,179tasks scored. Only retrieval-dependent items are scored.
[ Recall@10 ][ MRR@10 ]

The evaluation harness and a public half of the dataset are open source, so any retrieval system can be scored on the same tasks. Browse the code and data at firecrawl/benchmark-devdex.

Dataset

1,179 tasks scored on the full sets. A 594-item public sample, about half of each track, ships for reproduction.

Repository

Repository queries are created from capability descriptions of real, verifiable repositories. The dataset includes projects across a wide range of popularity levels rather than focusing only on well-known repositories. The repository name is never included in the query.

Example query“how to automatically route LLM classification requests to a cheaper model when confident and fall back otherwise”
Issue to fix

Issue and pull request queries are built from real issue and pull request pairs. For these questions, citing either the issue or the pull request is considered correct.

Example query“scikit-learn logisticregression random_state not workin”
Documentation

Documentation queries are created from real passages in a library's documentation. Each question is phrased differently from the wording used in the original passage. This reduces the possibility of matching the query directly to the source through surface-level language similarity.

Example query“How do I add Pydantic to my project using uv instead of pip?”

Metrics

How Recall@10 and MRR@10 are computed, how every system is driven, and the check that keeps memorized answers out.

Recall@10 and MRR@10

Two metrics, both computed over the agent's first ten citations. Recall@10 is the fraction of an item's correct answers that were cited. MRR@10 is the reciprocal rank of the first correct citation, so citing it first scores higher than citing it tenth.

The overall column is the equal-weight mean of the three track scores. A system is only given an overall score when it runs all three tracks.

How systems are driven

One agent (Claude Opus 4.8) drives every system through the same harness with a single search tool, ten results per call, and the same retrieval depth. Each system is called with its default configuration, and a run that returns nothing counts as a miss rather than being dropped.

Two rows are controls, marked as such, and are reference points rather than products: Native web search is the agent's own built-in search, and No tools is the agent answering from memory with no search at all.

Scoring

Scoring is deterministic: correct answers are matched by reference and canonical URL against fixed golds, with no model judging any number, so the same run reproduces the same score.

A memorisation gate removes any item the model could already answer with search disabled, so the benchmark measures retrieval rather than recall of training data. The floor that leaves is the No-tools control: 0.5% on repo, 0.0% on issue/PR, and 3.4% on docs.

[ 03 / 03 ]
·
FAQ
//
FAQ
//

Frequently asked questions

Common questions about developer retrieval and how to read these results.
It measures how reliably a tool surfaces the right developer answer for a query, across three tracks: finding the GitHub repository behind a described capability, finding the issue or pull request where a bug was fixed, and finding the documentation page that answers a how-to. One agent drives every system with a single search tool and ten results per call, and answers are scored deterministically against known-correct golds, with no model judging the results.
The Firecrawl Developer Index has the highest overall Recall@10 at 63.1%, ahead of Parallel (57.7%), Firecrawl Search (57.6%), Mintlify (54.6%), and Exa (53.7%). It also leads the issue-to-fix track (66.0%) and the documentation track (47.2%). On the repository track the field is close and general web search is competitive, with Parallel (81.9%) and native web search (80.7%) ahead there. Because single-track leads are often statistically tied, the combined score is the one that separates systems.
Recall@10 is the fraction of an item's correct answers that appear in the agent's first ten citations. MRR@10 is the reciprocal rank of the first correct citation, so a system that cites the right answer first scores higher than one that cites it tenth. Recall rewards finding the answer at all; MRR rewards ranking it near the top.
Context7 is a documentation index, and it is second overall on the docs track at 46.6%. But repository and issue/PR retrieval are outside its domain: roughly 78% and 52% of those runs return nothing, so its scores there are near zero. Its overall number therefore reflects coverage of one track out of three, not retrieval quality across all three.
The Developer Index is a curated index of developer content, issues, pull requests, READMEs, and docs, queried through Firecrawl's MCP server. Firecrawl Search is the general /search endpoint, a different surface. On this benchmark the purpose-built Developer Index scores higher overall, 63.1% vs 57.6% Recall@10, and leads on issue-to-fix and documentation retrieval.
No. Every task has a known-correct answer, and scoring checks by reference and canonical URL whether that answer appears in the agent's citations and where it ranks. There is no model grading answers, so the same run reproduces the same score. A memorisation gate also removes any item the model could answer with search disabled, so the benchmark measures retrieval rather than memorised knowledge.
The scores here are computed on the full task sets, and the benchmark ships a 594-item public sample, about half of each track, so the same comparison can be run without the full corpus. Every system is driven by the same agent, with the same single search tool and the same ten-result retrieval depth.
It is re-run and republished when results change materially or when a system is added or updated. Each snapshot is dated, and past changes are listed in the changelog on this page.

Changelog

Dated snapshots of this benchmark and what changed in each run.

  1. Aug 21, 2026First public snapshot (DevDex v1.0.0). Six retrieval tools and two controls compared on Recall@10 and MRR@10 across repository, issue-to-fix, and documentation retrieval.

Each snapshot is published from a pinned run in firecrawl/benchmark-devdex.