Web Search ​

Diffbot Web Search is a first of its kind web search engine combining a comprehensive first party web index, SOTA retrieval & reranking, and an API that fits entirely in a single machine. In simpler terms, web search in a box.

No ads, no telemetry, and fast in-network latency.

Web search should not compromise airgapped agents ​

Knowledge grounded agents need to search the web. This is a tricky task for airgapped environments. The second best solutions available today proxy your requests to third party Google SERP scrapers, fetch those URLs, and summarize it to your agent. Your request was just hot potatoed across 3 or more vendors.

The ideal web search solution never leaves your network. Index, retrieval, and reranking in a single machine or kubernetes cluster.

Traditional vs. Diffbot Web Search
                   ┌──────────────┐
                   │ Search query │
                   └───────┬──────┘
              ┌────────────┴─────────────┐
              │ Traditional              │ Diffbot
              ▼                          ▼       
      ┌───────────────┐      ┌───────────────────────┐
      │ 3rd party API │      │ Diffbot Web Search on │
      └───────┬───────┘      │     your hardware     │
              │              └───────────┬───────────┘
              │                          │
              ▼                          ▼
   ┌─────────────────────┐  ┌────────────────────────┐
   │ Google SERP scraper │  │ Results with chunks or │
   └──────────┬──────────┘  │      full content      │
              │             └────────────────────────┘
              │
              ▼
  ┌───────────────────────┐
  │ Web Fetch for content │
  └───────────┬───────────┘
              │
              ▼
 ┌─────────────────────────┐
 │ Results with summarized │
 │         content         │
 └─────────────────────────┘

Features ​

  • ~300ms latency target. Retrieval, ranking, and chunking in a single cluster.
  • Owned index. No variable API costs.
  • No web fetch required. Full document chunks live inside your index.
  • Runs fully offline. Ready for air-gapped environments, with zero telemetry.

Tradeoffs ​

  • No closed media sites. Facebook, Reddit, LinkedIn, Instagram, and other closed media sites are generally off limits.
  • Infrastructure cost. Diffbot Web Search runs on minimal hardware requirements. Other than an NVME storage floor, CPU and GPU can be budget balanced proportionally to latency.
  • Unoptimized tool use in post-training. Most models are post-trained to expect links with content snippets, requiring a secondary web fetch to retrieve full context. Diffbot Web Search's response can contain a page's full content, potentially overwhelming smaller models. Snippet length and token limits are adjustable in the API as a workaround.

Take Diffbot Web Search for a test drive on our reference hosted instance running a 96 core Threadripper CPU and 2x RTX 6000s.

Sign up for a free token to access the API, MCP, or agent skills.

curl --request GET \
  --url 'https://llm.diffbot.com/api/v1/web_search?text=diffbot' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer <DIFFBOT_TOKEN>'

Self-hosting ​

Diffbot Web Search is self-hostable on machines or on your cloud. Reach out to us for more information. Our reference machine is live in our Menlo Park, CA office.

Live at Diffbot HQ in Menlo Park, CA

FAQ ​

Why shrink the web down to 8TB?

8TB balances an achievable disk budget against covering 99% of daily driving search requirements. While this is our target production index size, your needs may vary and can be tuned for enterprise requirements.

What are the hardware requirements?

The entry requirements are low: any GPU, plus enough NVME SSD storage to hold your index.

ComponentRecommended
RAM64GB. A minimum of 4GB works; more is better. Sized to around 10% of your index.
GPUAny GPU with 5GB or more of memory. Production tested on GB10 and RTX 6000s.
CPU2.2GHz, 32 cores. Retrieval tasks are parallelized, so more cores lead to faster retrieval.
Disk~10TB to hold the latest index build (~8TB), on RAID 0 NVMe SSDs.

See also ​