Diffbot Global Index

The Diffbot Global Index lets you search the web like a database. Our Automatic APIs regularly scour the web, indexing articles, comments, images, products and more—all available for realtime search across any structured field.

Sign up for a Diffbot plan Global Index Documentation

Search for...

News

Tens of thousands of news sources are monitored constantly for new content. Search millions of articles and filter by author, date, topic, region, sentiment, language and more.

Comments

Article and blog post comments are automatically extracted and structured alongside article content. Find comments by author, search by topic or simply query for all comments matching your specific term.

Images and Video

The Global Index indexes all images and videos featured in articles, allowing you to filter by caption, source, image tags or image similarity.

More coming soon

Discussions, products, videos and more page types are on their way.


Features

JSON response

New API, Same Great JSON

The Global Index API returns JSON objects in the same format as our Automatic APIs. No refactoring necessary.

realtime indexing

Realtime indexing

Powered by Crawlbot, the Global Index is always up-to-date, regularly monitoring tens of thousands of sites.

Faceted Search

Faceted Searching

Powerful faceting engine lets you apply filters to any search: facet by date, site, topic, author and practically any other field.

Add custom sources

Add custom sources

Need to include content from an obscure source? Contact Support to add the site to our regularly monitored sources.