Technology
The stack that structures the world
Diffbot designs and researches machine learning services that work together to structure all of human knowledge. Many of these services are exposed as public endpoints for anyone to use.
01 A wild URL appears
Rendering
A URL arrives — requested by you, or discovered by one of Diffbot's global crawlers. The rendering engine then fetches, renders, and annotates raw HTML to make a web page legible to downstream services.
How it’s built
- 01
Fetch
In exchange for a URL and some TLS packets of affirmation, a web server returns HTML.
- 02
Render
Fetched HTML is rendered in a custom Chromium build, partially. Instead of loading all resources as requested, a model prioritizes just the resources needed to faithfully render the essential content of the page. Low priority resources, such as ad trackers, tracking scripts, and animations are dropped. Partial rendering reduces latency by several orders of magnitude and allows Diffbot to operate structured extraction at web scale. ↗ US Patent 9,678,928
- 03
Annotate
Every essential element in the HTML source is annotated with a dump of its low-level features: rendered pixels at every position, fonts, styles and layout, the DOM, the HTTP conversation and every dependent resource, even the state of the JavaScript VM and the CSS layout engine. Annotated features make it possible for Extract to classify and structure facts from unstructured HTML.
Go to
02 Signal from noise
Extract
The beating heart of Diffbot's engine. Extraction takes an annotated page, classifies it into a standard type, then attempts to extract a schema of that type.
Page types
type:articletype:producttype:discussiontype:imagetype:videotype:eventtype:jobtype:list
How it’s built
- 01
Classify
98% of the public web can be classified into one of roughly 20 standard page types, including article, product, discussion, image, video, event, job, lists, and more. Classification, exposed as the Analyze API, classifies and extracts a page in a single request. → Analyze API
- 02
Extract fields
Once a page is classified, it can be automatically extracted following a standard ontology. Text, visual layout, DOM, and render state features define semantic regions where fields (like product name, price, or author) are expected to exist. Once found, these values are normalized — units to metric, categories to a taxonomy. Finally, large textual regions are handed to the Natural Language service for further entity, relationship, and sentiment resolution. → Ontology
- 03
Custom Rules
For cases when the standard ontology does not cover a field you need, Extract can be extended with traditional CSS selector rules. → Custom API
Extract a page
Classify and extract structured, opinionated JSON from a single URL without hallucinations. (Surprise! It's not an LLM)
03 Scaling up
Bulk & Crawl
Twin async services that scale Extract across the global public web. Bulk extracts a static set of URLs. Crawl spiders a set of seed URLs for links to extract. These extracted links become a new set of URLs to spider. The recursive flow continues until all links are found, or a stop pattern is reached.
Go to
Bulk Extract APICrawl APICrawl PatternsSearch Crawl Data
Crawl every page
Walk an entire site from a single seed URL and extract thousands of pages into a single structured dataset.
04 Crawling the public web
Global Index
Diffbot has maintained a whole web crawl for over 20 years and is one of the last remaining independent web crawlers of that era. Built on C++, the Global Index doesn't just crawl, but also revisits pages on a freshness policy set per page type. Sharded across hundreds of SSDs, the Global Index is a holding lobby representing all of the public web to be further structured into consumable services like Web Search and the Knowledge Graph.
Go to
05 Resolving Entities
Natural Language
Some of the most critical factual information on the web lives not in selectable HTML elements but deep within unstructured plain text. By resolving entities, relationships, coreferences, salience, and sentiment, Natural Language is able to programmatically decipher M&A deals, executive changes, new product launches, and even more market signals within milliseconds — much faster and more consistently than an LLM.
Structure natural language
Resolve entities, relationships, facts, and sentiment from plain text.
06 Structuring the public web
Knowledge Graph®
Armed with a bag of facts extracted from billions of pages in every language, we can now build a Knowledge Graph. This requires multiple passes of inference, record linking, relationship extraction, and knowledge fusion. A full build is completed every 4-5 days.
How it’s built
- 01
Record Linking
Diffbot might exist as a single entity, but we exist on the web across thousands of documents — directory listings, a LinkedIn profile, a Wikipedia page, this website! Record and entity linking models cluster extractions that refer to the same entity, score the cluster into one record, and (if one exists) links them to an existing node in the Knowledge Graph. The result of this step are the nodes of the Knowledge Graph.
- 02
Relationship extraction
Nodes in the Knowledge Graph are linked together by relationships, also known as the edges of a graph. Edges in the KG are common sense facts that connect two entities together, such as the relationship between a parent and child, or a company and its location. Qualifiers for each edge (e.g. current or past) are also included at this step. This is one of the most difficult steps in building a KG that represents all of human knowledge. ↗ KnowledgeNet, EMNLP 2019
- 03
Knowledge fusion
Given a cluster of candidate facts about an entity, some will inevitably conflict, or represent stale information. Knowledge fusion reconciles these facts by evaluating truthfulness, trustworthiness of its source, and current established facts.
Go to
Search (DQL) APIEnhance APIOntology
Query the Knowledge Graph
Run structured queries against the world's largest Knowledge Graph of the public web for companies, people, news, and more.
07 Search for every day humans
Web Search
Plain text search across all of the public web — unstructured web content in the Global Index and structured entities in the Knowledge Graph. Built to serve extremely fast responses (<200ms); even faster on your own hardware. Candidates retrieved from the index are scored by a cross-encoder reranker model tuned to favor factual relevance, primary sources, and organic page rank. Easy for humans, easier for agents.
Go to
Search the web
Ultra-low latency web search without ads or telemetry. Built for self-hosting and API access.
08 Research as a service
Diffbot LLM
Large language models fine tuned for zero hallucination tolerance web research. Unlike most commercial frontier models, Diffbot LLM attempts to use every web access tool at its disposal (particularly Diffbot API) to ground every response on live public sources. Weights are open and available on Hugging Face.
Go to
09 Infrastructure
Served on bare metal
Diffbot’s first server cost about $400 and came off a pallet. That one box grew into rows of Diffbot’s own cabinets in a California datacenter — a hardware setup designed for crawling, rendering and extraction at web scale. Everything from the low-level vertical stack to production public APIs are served from these owned cabinets.
- Datacenter
- Owned cabinets, California
- Hardware
- Assembled in-house, per task
- Storage
- SSD-dense index shards
- Network
- Direct P2P links between graph builders
- Code
- Rust / C++ / Java
- Browser
- Custom Chromium build
BuildServers assembled in the office 
RollDown the road to the datacenter 
RackInto Diffbot’s own cabinets 
RunRows of cabinets, California
Knowledge Graph® is a registered trademark of Diffbot Technologies Corp., US Reg. No. 5,515,826.
Test Drive our APIs
No token required
Enter a query and hit Search to see the response.