Custom Headers
Pass referrers, user-agents, and cookies to Extract APIs.
Diffbot supports setting/sending the following custom headers for Extract API, Bulk Extract API, and Crawl API. These headers will be used when requesting content from third-party sites:
- User-Agent
- Referer
- Cookie
- Accept-Language
- X-Evaluate
User-Agent, Referer, Accept-Language
There are several ways to attach custom headers to API requests.
Direct
Use X-Forward as a prefix with any header you want forwarded to the target website. For example, to send the User-Agent foobar, we would use the header X-Forward-User-Agent: foobar.
You can confirm that your headers are working by creating a RequestBin and using the RequestBin URL as the extraction target.
Here's an example as a cURL request:
bash
curl --location --request GET 'api.diffbot.com/v3/article?token=<DIFFBOT_TOKEN>&url=https%3A%2F%2Fen17uofqrlcgv.x.pipedream.net%2F' \
--header 'X-Forward-User-Agent: foobar' \
--header 'X-Forward-Referrer: Diffbot.com' \
--header 'X-Forward-Accept-Language: hr'These headers are discarded after this call, meaning they need to be added again on a subsequent call.
Rule-based
To permanently attach headers to a rule for an API, you can download raw rule data, modify it, and upload it back to your token, replacing the old rule setup.
A rule with permanently attached headers might look like this:
json
{
xForwardHeaders: {
User-Agent: [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Safari/604.1.38"
],
X-Evaluate: "function() { start(); setTimeout(function() { setTimeout(function() { end(); }, 5000); }, 25000);}",
Cookie: "foo=bar"
},
rules: [ ],
api: "all",
urlPattern: "(http(s)?://)?(.*\.)?mysite.com.*",
testUrl: "",
},Re-uploading this JSON will add these headers to all calls issued towards mysite.com. Notice that User-Agent is a JavaScript array. If you supply User-Agent as an array (only possible through this method), Diffbot will randomly pick one from the list when accessing a site. This is great for throwing off bot-detection algorithms.
TIP
X-Evaluate is explained below.
Dashboard
The Dashboard also allows you to permanently add some headers to a rule. When creating a new custom rule, use the Custom Headers section to enter any headers you wish to add. This will save the headers in the same way as the JSON approach above.
Crawlbot and Bulk also support per-job settings, set via the Dashboard. A Crawlbot job has individual inputs for each of the headers that are supported, while a Bulk job allows the sending of arbitrary headers. Note that when entering headers on a Bulk job in the Dashboard, it is not necessary to include the X-Forward- prefix for the header name. So for example, to send a User-Agent header with value abc on a Bulk job, you would simply enter User-Agent:abc.
Cookie
The Cookie header allows you to:
- simulate a login session
- remove annoying GDPR and newsletter popups
- ignore ads and content you don't want to extract
For a comprehensive guide on using the Cookie header to simulate a login session, please see How do I extract content behind logins?.
X-Evaluate
See Custom Javascript.