Web search you own — no ads, no telemetry. ​

When it comes to web search, people tend to assume only one solution exists. You know the one — No API. Paid ads that grow in size and number each year. Tracks and sells your data. Rhymes with Boogle.

Now imagine a solution that's built API-first, without any ads or telemetry, and rhymes with Diffbot.

Diffbot Web Search is a first-of-its-kind web search engine combining a comprehensive web index, SOTA retrieval and reranking models, and an API that all fit entirely in a DGX Spark. In other words, web search in a box.

Built for AI ​

Your AI agent needs to search the web to ground its knowledge. This access is gated by model providers who proxy your request to third party Boogle SERP scrapers. Some of those URLs are fetched, and the final JSON array is returned heavily summarized to your agent. Your query was logged by at least 3 different mediators in this process and took 5 seconds.

With Diffbot Web Search, your query hits a server you own, a crossencoder model on that server retrieves, ranks, and chunks cached content, returning a condensed response without unnecessary web fetching in under 300ms.

                  ┌─────────────┐
                  │ Agent query │
                  └──────┬──────┘
             ┌───────────┴───────────┐
             │                       ▼Diffbot
             ▼today      ┌──────────────────────┐
    ┌────────────────┐   │    Server you own    │
    │ Model provider │   │ retrieves, ranks and │
    └────────┬───────┘   │        chunks        │
             │           └───────────┬──────────┘
             │                       │
             ▼                       ▼
     ┌──────────────┐   ┌─────────────────────────┐
     │ SERP scraper │   │ Condensed JSON (<300ms) │
     └───────┬──────┘   └─────────────────────────┘
             │
             ▼
       ┌───────────┐
       │ Web Fetch │
       └─────┬─────┘
             │
             ▼
 ┌───────────────────────┐
 │ Summarized JSON (~5s) │
 └───────────────────────┘

The single hop latency benefits apply with our hosted Web Search API. Speaking of...

The easiest way to get started with Diffbot Web Search is to add our agent skills to your agent harness of choice.

If you prefer, Diffbot Web Search API is available in GET and POST flavors.

curl --request GET \
  --url 'https://llm.diffbot.com/api/v1/web_search?text=diffbot' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer <DIFFBOT_TOKEN>'

Self-hosting ​

Diffbot Web Search is self-host ready. Reach out to us for more information.

See also ​