Web Search API ​

Diffbot maintains the largest independently crawled web index outside of Google and Bing (>150TB). The Web Search API allows for fast querying of this index.

Candidates are retrieved and reranked with a cross-encoder model trained to rank factual relevance over popularity, primary sources over domain monopolies, and organic page rank over paid spend.

The net effect is search results that assume the user is not an idiot. A search for "hypothyroidism" returns relevant research study papers alongside content from publicly administered websites and reputable non-profit institutions.

Web Search API is available as a GET and POST.

Key features ​

  • Latency: Roughly 200ms hot latency, ~400ms cold. We are aiming for sub-100ms P90 latency.
  • Highlights: Splices out the most relevant chunk of a page and returns it in the content of each search result. Not an AI summary; that would be slow and undeterminably inaccurate. Chunk size can be dialed in with the maxTokens parameter for context-size sensitive applications.
  • Local hosting: The entire web index, including the retrieval/reranking models and the API, can be hosted on-prem, enabling internet "connectivity" in air-gapped or high-risk workflows. Reach out to sales@diffbot.com for more information on this option.

Access the Web Search API via REST or through the diffbot-python library. For use with agents, install one of the harness plugins available in diffbot-skills.

Pricing & rate limits ​

A free Diffbot token includes 100,000 queries per month with a 60 QPM rate limit and no concurrency. Should you need more, contact sales@diffbot.com for pricing down to $0.02/1K, >30,000 QPM, and concurrency limits north of 500.

Building an open source project? For the low low price of $0 we'll upgrade your token to 1,000,000 queries per month. Star and open an issue in your desired interface (diffbot-python, diffbot-typescript, or diffbot-skills) to request this.

Limitations ​

  • The index is English only for the moment. Multi-language support is ongoing.
  • Content that has explicitly opted out of crawling (robots.txt) is not included in the index.
  • Platform-gated content is generally not included in the index, including content from sites like Reddit and most social media platforms.