Create a Bulk Extract Job ​

POSThttps://api.diffbot.com/v3/bulk

To create a bulk job, make a POST request to this endpoint.

See also: Managing a Bulk Job and Retrieving a Bulk Job

Payload Setup ​

This endpoint only accepts a Content-Type header set to application/x-www-form-urlencoded (not multipart/form-data), with body content in querystring format (key/value pairs), for example:

name=bulkTest&token=<DIFFBOT_TOKEN>&urls=https://www.diffbot.com https://blog.diffbot.com&apiUrl=https://api.diffbot.com/v3/analyze

Example ​

bash
curl --location 'https://api.diffbot.com/v3/bulk?token=<DIFFBOT_TOKEN>' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'name=bulkTest' \
--data-urlencode 'urls=https://www.diffbot.com https://blog.diffbot.com' \
--data-urlencode 'apiUrl=https://api.diffbot.com/v3/analyze'
json
{
  "response": "Successfully added urls for spidering.",
  "jobs": [
    {
      "jobStatus": {
        "message": "Job is initializing.",
        "status": 0
      },
      "maxHops": -1,
      "downloadJson": "...json",
      "urlProcessPattern": "",
      "jobCompletionTimeUTC": 0,
      "maxRounds": -1,
      "type": "bulk",
      "pageCrawlSuccessesThisRound": 0,
      "urlCrawlRegEx": "",
      "pageProcessPattern": "",
      "apiUrl": "https://api.diffbot.com/v3/analyze",
      "useCanonical": 1,
      "jobCreationTimeUTC": 1649950325,
      "repeat": 0,
      "downloadUrls": "...csv",
      "obeyRobots": 1,
      "roundsCompleted": 0,
      "pageCrawlAttempts": 0,
      "notifyWebhook": "",
      "pageProcessSuccessesThisRound": 0,
      "customHeaders": {},
      "objectsFound": 0,
      "roundStartTime": 0,
      "urlCrawlPattern": "",
      "seedRecrawlFrequency": -1,
      "urlProcessRegEx": "",
      "pageProcessSuccesses": 0,
      "urlsHarvested": 0,
      "crawlDelay": -1,
      "currentTime": 1649950325,
      "useProxies": 0,
      "sentJobDoneNotification": 0,
      "currentTimeUTC": 1649950325,
      "name": "bulkTest",
      "notifyEmail": "",
      "pageCrawlSuccesses": 0,
      "pageProcessAttempts": 0
    }
  ]
}

API Definition

Generated from v1.1 of the Bulk Extract API OpenAPI specification.

Extract a list of URLs asynchronously

Request

Authentication
tokenquery
Cache and insert token into example requests (15d expiry)
Request body application/x-www-form-urlencoded
namestring required

Job name. This should be a unique identifier and will be used to modify your bulk job and retrieve its output.

urlsstring required

Space-delimited list of URLs to process. If you are on the Startup plan, jobs must contain at least 50 URLs.

apiUrlstring required

The full Extract (or Custom) API to be used for each URL. For instance, to process each URL via the Article API, supply https://api.diffbot.com/v3/article. You may also include API parameters, e.g. https://api.diffbot.com/v3/article?fields=meta,tags.

customHeadersstring

Set custom headers to be used for processing each URL. Send multiple customHeaders values in your POST body, with header keys/values delimited by a colon (and URL-encoded).

notifyEmailstring

Send a message to this email address when the bulk job completes.

notifyWebhookstring

Pass a URL to be notified when the bulk job completes. You will receive a POST with a JSON response.

obeyRobotsstring

Pass obeyRobots=0 to ignore a site’s robots.txt instructions.

repeatnumber (float)

Specify the number of days as a floating-point (e.g. repeat=7.0) to repeat this job. By default bulk jobs will not be repeated.

pageProcessPatternstring

Enter ||-separated strings to limit pages processed to those whose HTML contains any of the content strings. If a page does not contain at least one of the strings, it will be ignored.

Example request

bash
curl --request POST \
  --url 'https://api.diffbot.com/v3/bulk?token=<DIFFBOT_TOKEN>' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-raw '{}'

Response

200Bulk job created
responsestring
jobsobject[]
jobStatusobject
messagestring
statusinteger
maxHopsinteger
downloadJsonstring
urlProcessPatternstring
jobCompletionTimeUTCinteger
maxRoundsinteger
typestring
pageCrawlSuccessesThisRoundinteger
urlCrawlRegExstring
pageProcessPatternstring
apiUrlstring
useCanonicalinteger
jobCreationTimeUTCinteger
repeatinteger
downloadUrlsstring
obeyRobotsinteger
roundsCompletedinteger
pageCrawlAttemptsinteger
notifyWebhookstring
pageProcessSuccessesThisRoundinteger
customHeadersobject
objectsFoundinteger
roundStartTimeinteger
urlCrawlPatternstring
seedRecrawlFrequencyinteger
urlProcessRegExstring
pageProcessSuccessesinteger
urlsHarvestedinteger
crawlDelayinteger
currentTimeinteger
useProxiesinteger
sentJobDoneNotificationinteger
currentTimeUTCinteger
namestring
notifyEmailstring
pageCrawlSuccessesinteger
pageProcessAttemptsinteger
json
{
    "response": "Successfully added urls for spidering.",
    "jobs": [
        {
            "jobStatus": {
                "message": "Job is initializing.",
                "status": 0
            },
            "maxHops": -1,
            "downloadJson": "...json",
            "urlProcessPattern": "",
            "jobCompletionTimeUTC": 0,
            "maxRounds": -1,
            "type": "bulk",
            "pageCrawlSuccessesThisRound": 0,
            "urlCrawlRegEx": "",
            "pageProcessPattern": "",
            "apiUrl": "https://api.diffbot.com/v3/analyze",
            "useCanonical": 1,
            "jobCreationTimeUTC": 1649950325,
            "repeat": 0,
            "downloadUrls": "...csv",
            "obeyRobots": 1,
            "roundsCompleted": 0,
            "pageCrawlAttempts": 0,
            "notifyWebhook": "",
            "pageProcessSuccessesThisRound": 0,
            "customHeaders": {},
            "objectsFound": 0,
            "roundStartTime": 0,
            "urlCrawlPattern": "",
            "seedRecrawlFrequency": -1,
            "urlProcessRegEx": "",
            "pageProcessSuccesses": 0,
            "urlsHarvested": 0,
            "crawlDelay": -1,
            "currentTime": 1649950325,
            "useProxies": 0,
            "sentJobDoneNotification": 0,
            "currentTimeUTC": 1649950325,
            "name": "bulkTest",
            "notifyEmail": "",
            "pageCrawlSuccesses": 0,
            "pageProcessAttempts": 0
        }
    ]
}
401Token missing or invalid
errorCodeinteger
errorstring
json
{
    "errorCode": 401,
    "error": "Not authorized API token."
}
500No URLs were found in the request
errorCodestring
errorstring
json
{
    "errorCode": "500",
    "error": "no urls found"
}
505Job name rejected — names can consist only of letters, numbers, underscores, or hyphens
errorCodeinteger
errorstring
json
{
    "errorCode": 505,
    "error": "Crawl names can consist only of letters, numbers, underscores or hyphens."
}