Diffbot Search Agent: A new SOTA for web question answering

Maosheng Guo

Machine Learning Engineer at Diffbot · November 9, 2026

Searching for information online is one of the most popular use cases of AI, with over 77% of users using AI for search on a daily basis [1]. However, general purpose harnesses load over 113 tools into their system prompts and must support a wide range of general purpose tasks. We wondered if we could do better by building a specialized harness for the sole task of web research that simultaneously reducing token usage while increasing answering performance.

We present Diffbot Search Agent, a new state-of-the-art for agentic web research work. Diffbot Search Agent combines extremely low latency web retrieval tools with an adaptive deep and wide reasoning loop to optimize the joint objective of answering more complex questions correctly while reducing the number of tokens used.

1. DeepSearchQA F1
2. WideSearch Item F1

0.000.250.500.751.00

Diffbot Search AgentDeepSeek-V4.1-FlashDeepSeek HarnessDeepSeek-V4.1-FlashKimi CodeKimi K3Claude CodeOpus 5.5Claude CodeSonnet 5.5ZCodeGLM-5.3-Flash

Harness	Model	DeepSearchQA F1	WideSearch Item F1
Diffbot Search Agent	DeepSeek-V4.1-Flash	0.969	0.839
DeepSeek Harness	DeepSeek-V4.1-Flash	0.932	0.835
Kimi Code	Kimi K3	0.903	0.630
Claude Code	Opus 5.5	0.902	0.823
Claude Code	Sonnet 5.5	0.889	0.763
ZCode	GLM-5.3-Flash	0.770	0.768

Better Tools, Better Search

What makes research deep?

At Diffbot, we have a long history of working with complex web research. In our analysis, we found that web research questions are complex in one of two ways.

A deep question is a chain of facts where each link depends on the last, and the answer is a number or a name that has to be exactly right. Here is one:

According to Our World in Data, identify the countries which had a population of more than 10 million in 1950, had fewer than 100 deaths from armed conflicts on their territory from 1989 to 2023 (best estimate), and had fewer than 1 million emigrants in 2020. Of these, which had a life expectancy at birth in 1958 closest to Australia's?

Answering it means finding four datasets, joining them by country, applying three filters and comparing a fourth column. The answer is the Netherlands. The hard part is not finding a page. It is reading four large tables without losing track of any one value.

A wide question asks for every member of a set, with several facts about each:

Offer me a list of the U.S. National Parks, including the park's name, the state(s) it is in, the price of the Standard Entrance Pass for one adult without a vehicle, and whether it has hills or mountains.

There are 63 parks, and each fee is on its own page. No single step is hard. The work is wide: 63 small units of work without dropping or duplicating.

Figure 1. A deep question is one chain where each step depends on the last; a wide question is 63 independent rows, each from its own page.

Deep and wide questions stress different elements of an agent. Depth stresses reading and computing. Wide stresses coverage, coordination, and time. A capable research agent should be able to excel at both. We selected one public benchmark for each and used them as our compass: Google DeepMind's DeepSearchQA[2] for deep questions, and ByteDance's WideSearch[3] for wide questions.

The tools a research agent needs

Generic harness from the frontier labs bundle 113 tools to handle every possible situation, but this burdens the LLM with additional choice load. Like an F1 car stripped down to a single seater cockpit, we sought to eliminate all unnecessary weight and ended with 4 core tools a web research agent needs:

1. Web search (web_search). Diffbot Search Agent searches the web with Diffbot Web Search, built on Diffbot's local index of the public web.[4]. Unlike the web search used in generic harnesses, since Diffbot's web search can query its local index in <100ms, it can do many more turns of search.

2. Web fetch (web_fetch). Answers to deep research questions often live in a statistics table, a row in a CSV file, or a figure in a PDF annual report using Diffbot. Extracting just the right chunk of text from a page without flooding the context with irrelevant tokens that lower prefill speed and introducing distractors is a difficult research challenge that Diffbot has solved.

3. Knowledge Graph (diffbot_kg). The Diffbot Knowledge Graph holds over 10 billion entities and trillions of facts, built by crawling and structuring billions of public web pages, with each entity fused from several sources and kept up to date.[5]. This tool is missing from most generic harnesses, but for questions about organizations, people and places, it can answer in one query what the web answers in dozens of pages, especially structured questions such as "pharmaceutical companies with more than 50,000 employees". The web remains better for unstructured probing: recent events, narrative and analysis, and anything outside of known entity types. There is one catch. Diffbot Knowledge Graph is queried in its own custom language, DQL, and general-purpose models translate natural language into DQL poorly. To solve this, we give the tool three narrow stages: A tiny fine-tuned model turns the agent's question into DQL. The harness runs the query. A short summary comes back together with the matching entities themselves, so the answer stays grounded in the data. On questions shaped for the Knowledge Graph, offering the tool kept accuracy the same while cutting tokens by a quarter to 40%, and on the harder set it made the agent 30% faster.

A JavaScript sandbox (execute_js). Many answers need arithmetic over what was read: join two tables, filter by a threshold, find the closest value. Models are unreliable at doing this in memory. A sandboxed JavaScript interpreter with no network access can be called to read any files the agent has fetched. Most general purpose models are well trained in JavaScript and JSON, making this tool much more than a basic calculator.

Tools left out

Shell (Bash or Cmd). Most generic harnesses have a shell tool. While this provides more flexibility, it also introduces scope and security concerns. Generic harnesses solve this by either running a separate LLM to approve shell actions (or use human approval), which is latency we can avoid. Javascript can be locked down in a lightweight V8 isolate with capped memory, no network, and read only access to session files.

A browser or computer-use tool. Diffbot's Extract API abstracts all rendering work into markdown. This leaves the agent to focus on research instead of spending tokens on screenshots and clicks. Most generic harnesses would perform actions using an agent browser, which introduce launch and navigation latency.

Read, grep or file tools. execute_js already reads saved pages by line range.

To-do and memory. we tried prompting the model to manage its own context, and it didn't change its behavior. Instead, we've designed the harness with a compactor and time budget to manage this. Most generic harness have a multitude of memory management tools and we tried to only have one way to keep running working memory.

How to go deep

A deep question takes many compounding steps. If every page the agent reads fills its context, each later step becomes slower and more expensive. At research scale, the model starts to lose track of what it knows. Our first instinct was to ask the model to be frugal: a prompt telling it to keep its context small, and a scratch file it could write notes to (a familiar design from coding harnesses). The results were unremarkable. In testing, the agent wrote to its scratch file on 0 of 26 WideSearch tasks. A model rarely adopts a capability just because it is described. It adopts what makes its next step easier, and what it cannot avoid. The solution is to manage context with the harness, not the model.

Pages spill to disk, with an index

A fetched page does not pipe directly into context. If it is larger than the tool's output budget, web_fetch writes the full markdown to a file and returns an index instead: the file's path, its size in bytes and lines, and its headings with their line numbers. The model sees the outline of a page and reads only the sections it needs. If a singular fact is requested from a page, a small subagent model answers is fed the page and answers. This is one of the most formative design decisions dictating how web_fetch works with Diffbot Search Agent.

A sandbox that reads the files

If pages are saved to disk, the agent needs to read them. execute_js is called from within a sandbox to read saved pages in a REPL environment: variables, functions and parsed tables persist from one call to the next. A 1.6 MB CSV source is parsed once into a variable, queried, joined with a table, and verified against a hypothesis. Like an analyst working in a notebook, sources are verified methodically line by line. Data never passes through the model's context. Only the answers to its questions do.

Here is an "Our World in Data" question as the agent actually ran it.

Figure 3. q665 to scale: two searches, four CSV fetches saved to disk, six sandbox calls that parse, join and filter the four tables, and the answer after 9 steps and 41 seconds.

Four files totalling about 2.4 MB were read, joined and filtered, and none entering the model's context. The model saw four short indexes and the results of its own code. That is how a flash model gets a four-dataset join perfectly in 41 seconds.

The sandbox's memory has a secondary benefit. It lives in the interpreter, not in the conversation, so it survives anything that happens to the conversation. When old tool results are cleared out of the context to save space (we'll get back to this), the tables the model built from them are still in memory, one variable name away. The context holds what the model is thinking about; the sandbox holds what it is working on.

How to go wide

For a wide question, one agent reading 63 pages one after another is not only slow, but also fills its context with indifferentiatable pages. In such scenarios, we introduce a tool for the agent to fan out: parallel_explore.

In parallel_explore, a coordinator splits the job, explorer agents collect answers in parallel, and a compiler assembles a table with all the answers.

Cover a set without overlap. The coordinator divides a set of data into assignable slices. A list of national parks split 5 ways by alphabetical order, or chronological data sliced by year. Each slice is given to an explorer subagent and told explicitly that every other slice is out of scope. Without this explicit instruction, explorers wander into neighboring slices, collecting the same rows twice and burn out the time budget unnecessarily.

Agree on keys before collecting. Every explorer subagent receives the same column structure, key columns, and example conventions. This ensures data from multiple different sources agree on format before any work is done, avoiding drift. When working with a list of national parks, a key column convention might be enforcing official park names without the words "National Park" (i.e. "Acadia", not "Acadia National Park"), or "Black Canyon of the Gunnison", without diacritics. Keys match the agreed convention at collection time, and a compiler merges them exactly.

Rows go in files. Each explorer writes its rows as JSON to its own file and reports one line. A compiler agent loads every file with deterministic helpers (read the rows, deduplicate by key, render the table), looks into any file that came back empty instead of silently dropping it, and writes the merged table. If the compiler fails, a deterministic backstop produces the table anyway. The coordinator gets the result as a stage, not as the answer: it can check it, join it with something else, or send out another pass for the gaps.

Explorers share fetch requests. A shared cache coalesces requests that are still in flight. A page that three explorers requests at the same time are merged into a single request.

Figure 4. parallel_explore. The coordinator splits the job into slices that cover everything without overlap, with the same columns, key and key convention for every explorer. Explorers collect in parallel through a fetch cache they share with the coordinator and write their rows to files; a compiler merges the files by key and returns the table to the coordinator as a stage.

Figure 5. ws_en_043 to scale: six explorers collect in parallel, a compiler merges 63 rows by park name, and the coordinator spot-checks seven fee pages and adds the mountains column. Item F1 0.99.

We arrived at this narrow tool by failing forward from a general one. An earlier iteration gave the coordinator a "spawn agents" tool that could delegate anything to anyone. The model used it eagerly and had a tendency to collect far more "answers" than required: one task ballooned from 126 rows to 282. Free-form results with free-form keys also led to their own source of errors. parallel_explore constrains what explorers return and assigns rows to named columns and agreed keys. Merging largely becomes a deterministic join rather than endless parallel reasoning.

Fanning-out is an expensive tool. The coordinator, not the harness, decides when a question is wide enough to need it.

Architecture: the full picture

Figure 6. One coordinator with seven tools, two of them only for images. Pages go to disk and the sandbox reads them. Wide work fans out to explorers, which can search, fetch and compute, and which read the pages the coordinator already fetched; a compiler, which can only run code, merges their rows by key. Citations and images are added after the answer.

Diffbot Search Agent is one coordinator agent with explorer subagents for wide work and a compiler that assembles what they collect. That places it between two well-known positions — Anthropic prefers multi-agent research systems, where a lead agent plans and sub-agents search in parallel.[6] Cognition argues the opposite: agents that do not share context make inconsistent decisions, preferring a single agent with all the context.[7] We find merit in both positions, and designed a harness that adapts to multi-agent orchestration for wide questions or a singular agent that relies on the harness to manage context for deep questions.

In both strategies, agents share files, not summaries. When a coordinator hands a wide task to explorers, pages that have already been fetched are handed to them as file paths. When an explorer subagent finishes, its rows live in a file and the compiler reads the file. The alternative leads to the natural outcome of telephone — each agent resummarizes its findings to the next agent, which resummarizes those findings along with theirs to the next. Each retelling costs tokens and loses precision with each step.

Results

Benchmarks

Table 3. Production configuration, two runs per benchmark.

Benchmark	Metric	Value
DeepSearchQA, 46-question sample	F1	0.969
	Fully correct	92%
WideSearch, 26 English tasks	Item F1	0.839
	Row F1	0.658

A DeepSearchQA question took about two minutes and 0.4 million tokens at the median, most of them served from the provider's prompt cache.

Comparisons

We ran other agent harnesses on exactly the same questions and graded them with the same judge. To compare harnesses and models rather than search engines, every one used Diffbot Web Search, through our MCP server, as its only search tool, with fetching left to its own tools. Each otherwise ran in isolation, with its own defaults, apart from any configuration noted.

DeepSearchQA F1

1. Diffbot Search AgentDeepSeek-V4.1-Flash0.969
2. Claude CodeDeepSeek-V4.1-Flash0.953
3. Diffbot Search AgentKimi K30.948
4. DeepSeek HarnessDeepSeek-V4.1-Flash0.932
5. Kimi CodeKimi K30.903
6. Claude CodeOpus 5.50.902
7. Diffbot Search AgentGLM-5.3-Flash0.892
8. Claude CodeSonnet 5.50.889
9. Claude CodeGLM-5.3-Flash0.820
10. ZCodeGLM-5.3-Flash0.770

WideSearch Item F1

1. Diffbot Search AgentDeepSeek-V4.1-Flash0.839
2. DeepSeek HarnessDeepSeek-V4.1-Flash0.835
3. Claude CodeDeepSeek-V4.1-Flash0.823
4. Claude CodeOpus 5.50.823
5. Diffbot Search AgentGLM-5.3-Flash0.784
6. ZCodeGLM-5.3-Flash0.768
7. Claude CodeSonnet 5.50.763
8. Claude CodeGLM-5.3-Flash0.735
9. Diffbot Search AgentKimi K30.719
10. Kimi CodeKimi K30.630

Table 4. Comparison of benchmarks across various models and harnesses. Tokens are the median per DeepSearchQA question; cost is the mean.

Model	Harness	DeepSearchQA F1	WideSearch item F1	Tokens per question	Cost per question
DeepSeek-V4.1-Flash	Diffbot Search Agent	0.969	0.839	0.40M	$0.018
	Claude Code	0.953	0.823	0.82M	$0.031
	DeepSeek Harness	0.932	0.835	0.85M	$0.026
Kimi K3	Diffbot Search Agent	0.948	0.719	0.37M	$0.68
	Kimi Code	0.903	0.630	0.51M	$0.66
GLM-5.3-Flash	Diffbot Search Agent	0.892	0.784	0.17M	$0.026
	ZCode	0.770	0.768	0.22M	$0.027
	Claude Code	0.820	0.735	0.24M	$0.038
Opus 5.5	Claude Code	0.902	0.823	0.19M	$0.34
Sonnet 5.5	Claude Code	0.889	0.763	0.25M	$0.22

Within each model, Diffbot Search Agent scores highest on both benchmarks and uses the fewest tokens per deep question. Across the whole table, Diffbot Search Agent on DeepSeek-V4.1-Flash is both the most accurate system and the cheapest. Claude models use fewer tokens, but at list prices each of their tokens cost fifteen to thirty times as much, making answering a single question cost twelve to nineteen times more.

Comparisons by harness

To separate the effect of the model and the Diffbot's tools, we performed a replacement experiment where we put open source models (DeepSeek-V4.1-Flash, GLM-5.3-Flash) in popular generic harnesses and measured their performance on DeepSearchQA and WideSearch as well as comparing it to the closed-source Opus and Sonnet models.

We found that mere addition of the Diffbot Search Agent harness increased F1 on DeepSearchQA up to 12% while simultaneously reducing the median number of tokens per question.

1. Diffbot Search Agent
2. Claude Code
3. DeepSeek Harness
4. Kimi Code
5. ZCode
6. same model, swapped onto our harness

DeepSeek-V4.1-Flash DeepSearchQA F1

1. Diffbot Search Agent0.969
2. Claude Code0.953
3. DeepSeek Harness0.932

DeepSeek-V4.1-Flash WideSearch Item F1

1. Diffbot Search Agent0.839
2. DeepSeek Harness0.835
3. Claude Code0.823

Kimi K3 DeepSearchQA F1

1. Diffbot Search Agent0.948
2. Kimi Code0.903

Kimi K3 WideSearch Item F1

1. Diffbot Search Agent0.719
2. Kimi Code0.630

GLM-5.3-Flash DeepSearchQA F1

1. Diffbot Search Agent0.892
2. Claude Code0.820
3. ZCode0.770

GLM-5.3-Flash WideSearch Item F1

1. Diffbot Search Agent0.784
2. ZCode0.768
3. Claude Code0.735

Try Diffbot Search Agent

Diffbot Search Agent is available as a live demo at diffbot.com. Inspect the tool calls and sources it reads live as it reasons through a response.

For API access, an OpenAI compatible completions endpoinnt is available at https://llm.diffbot.com/v1/chat/completions. Authenticate with a free Diffbot token.

export DIFFBOT_KEY=...   # Your llm.diffbot.com key

curl -N https://llm.diffbot.com/v1/chat/completions \
  -H "Authorization: Bearer $DIFFBOT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "diffbot-search-agent",
    "stream": true,
    "messages": [
      {"role": "user", "content": "List every national park in the US with its state and year established."}
    ]
  }'

References

1. Adobe, "ChatGPT as a search engine", 8 July 2025.
2. Gupta et al., "DeepSearchQA", arXiv:2601.20975, January 2026.
3. Wong et al., "WideSearch", arXiv:2508.07999, August 2025.
4. Diffbot Web Search documentation, diffbot.com/docs/web-search, accessed 26 October 2026.
5. Diffbot Knowledge Graph, diffbot.com/products/knowledge-graph, accessed 28 October 2026.
6. Anthropic, "How we built our multi-agent research system", 13 June 2025.
7. Cognition, "Don't Build Multi-Agents", June 2025.