Web Search
Diffbot Web Search is a first-of-its-kind web search engine combining a comprehensive first party web index, SOTA retrieval and reranking models, and an API that fits entirely in a DGX Spark. In other words, web search in a box.
No ads, no telemetry, and latency no other web search API can match because it's on your network.
Built for AI
Your agent needs to search the web to ground its knowledge. This access is gated by model providers who proxy your request to third party Boogle SERP scrapers. Some of those URLs are fetched, and the final JSON array is returned heavily summarized to your agent. Your query was logged by at least 3 different mediators in this process and took 5 seconds.
With Diffbot Web Search, your query hits a server you own, a crossencoder model on that server retrieves, ranks, and chunks cached content, returning a condensed response without unnecessary web fetching in under 300ms.
┌─────────────┐ │ Agent query │ └──────┬──────┘ ┌───────────┴───────────┐ │ ▼Diffbot ▼today ┌──────────────────────┐ ┌────────────────┐ │ Server you own │ │ Model provider │ │ retrieves, ranks and │ └────────┬───────┘ │ chunks │ │ └───────────┬──────────┘ │ │ ▼ ▼ ┌──────────────┐ ┌─────────────────────────┐ │ SERP scraper │ │ Condensed JSON (<300ms) │ └───────┬──────┘ └─────────────────────────┘ │ ▼ ┌───────────┐ │ Web Fetch │ └─────┬─────┘ │ ▼ ┌───────────────────────┐ │ Summarized JSON (~5s) │ └───────────────────────┘
Try Diffbot Web Search
The easiest way to try Diffbot Web Search is to add our agent skills to your agent harness of choice.
If you prefer, Diffbot Web Search API is available in GET and POST flavors.
curl --request GET \
--url 'https://llm.diffbot.com/api/v1/web_search?text=diffbot' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <DIFFBOT_TOKEN>'Self-hosting
Diffbot Web Search is self-host ready. Reach out to us for more information. Web Search API is hosted on a dual RTX 6000 workstation at Diffbot HQ.