Manage a Bulk Extract Job ​

GEThttps://api.diffbot.com/v3/bulk

A single endpoint allows both control and status requests for one or more active bulk jobs (and crawls) with any given token.

See also: Creating a Bulk Job and Retrieving a Bulk Job

View the Status of Bulk Jobs ​

Your token's active bulk jobs (along with any active crawls) will be returned in a jobs object when a GET request supplying just a token parameter is made to this endpoint.

Note that this endpoint without any query parameters returns exactly the same output as its Crawl Job equivalent.

bash
curl --request GET \
     --url https://api.diffbot.com/v3/bulk?token=<DIFFBOT_TOKEN> \
     --header 'Accept: application/json'

To retrieve a single job's details, provide the job's name in addition to your token in your request.

Pause a Bulk Job ​

To pause a bulk job, send a GET request to this endpoint supplying your token, the name of the bulk job to pause, and the pause parameter set to 1.

bash
curl --request GET \
     --url https://api.diffbot.com/v3/bulk?token=<DIFFBOT_TOKEN>&name=bulkTest&pause=1 \
     --header 'Accept: application/json'

To resume a paused bulk job, pass pause=0 in the same GET request.

Delete a Bulk Job ​

To delete a bulk job, send a GET request to this endpoint supplying your token, the name of the bulk job to delete, and the delete parameter set to 1. Job deletions are irreversible.

bash
curl --request GET \
     --url https://api.diffbot.com/v3/bulk?token=<DIFFBOT_TOKEN>&name=bulkTest&delete=1 \
     --header 'Accept: application/json'

Restart a Bulk Job ​

To restart a bulk job, send a GET request to this endpoint supplying your token, the name of the bulk job to restart, and the restart parameter set to 1. This will erase all previously processed data and re-process all of the submitted URLs.

bash
curl --request GET \
     --url https://api.diffbot.com/v3/bulk?token=<DIFFBOT_TOKEN>&name=bulkTest&restart=1 \
     --header 'Accept: application/json'

Status Codes ​

The jobStatus object will return the following status codes and associated messages:

StatusMessage
0Job is initializing
6Job paused
7Job in progress
8All crawling temporarily paused by root administrator for maintenance.
9Job has completed and no repeat is scheduled

API Definition

Generated from v1.1 of the Bulk Extract API OpenAPI specification.

Pause, delete, restart, or view the status of a bulk job.

Request

Authentication
tokenquery
Cache and insert token into example requests (15d expiry)
Query parameters
namestring

Job name as defined when the bulk job was created.

pausestring

Pass pause=1 to pause a bulk job. Pass pause=0 to resume a paused job.

restartstring

Pass restart=1 to restart a bulk job. This will erase all processed data and re-process all of the submitted URLs.

deletestring

Pass delete=1 to delete a job, and all associated data, completely.

Example request

bash
curl --request GET \
  --url 'https://api.diffbot.com/v3/bulk?token=<DIFFBOT_TOKEN>' \
  --header 'Accept: application/json'

Response

200Job managed
responsestring
jobsobject[]
jobStatusobject
messagestring
statusinteger
maxHopsinteger
downloadJsonstring
urlProcessPatternstring
jobCompletionTimeUTCinteger
maxRoundsinteger
typestring
pageCrawlSuccessesThisRoundinteger
urlCrawlRegExstring
pageProcessPatternstring
apiUrlstring
useCanonicalinteger
jobCreationTimeUTCinteger
repeatinteger
downloadUrlsstring
obeyRobotsinteger
roundsCompletedinteger
pageCrawlAttemptsinteger
notifyWebhookstring
pageProcessSuccessesThisRoundinteger
customHeadersobject
objectsFoundinteger
roundStartTimeinteger
urlCrawlPatternstring
seedRecrawlFrequencyinteger
urlProcessRegExstring
pageProcessSuccessesinteger
urlsHarvestedinteger
crawlDelayinteger
currentTimeinteger
useProxiesinteger
sentJobDoneNotificationinteger
currentTimeUTCinteger
namestring
notifyEmailstring
pageCrawlSuccessesinteger
pageProcessAttemptsinteger
json
{
    "response": "Successfully added urls for spidering.",
    "jobs": [
        {
            "jobStatus": {
                "message": "Job is initializing.",
                "status": 0
            },
            "maxHops": -1,
            "downloadJson": "...json",
            "urlProcessPattern": "",
            "jobCompletionTimeUTC": 0,
            "maxRounds": -1,
            "type": "bulk",
            "pageCrawlSuccessesThisRound": 0,
            "urlCrawlRegEx": "",
            "pageProcessPattern": "",
            "apiUrl": "https://api.diffbot.com/v3/analyze",
            "useCanonical": 1,
            "jobCreationTimeUTC": 1649950325,
            "repeat": 0,
            "downloadUrls": "...csv",
            "obeyRobots": 1,
            "roundsCompleted": 0,
            "pageCrawlAttempts": 0,
            "notifyWebhook": "",
            "pageProcessSuccessesThisRound": 0,
            "customHeaders": {},
            "objectsFound": 0,
            "roundStartTime": 0,
            "urlCrawlPattern": "",
            "seedRecrawlFrequency": -1,
            "urlProcessRegEx": "",
            "pageProcessSuccesses": 0,
            "urlsHarvested": 0,
            "crawlDelay": -1,
            "currentTime": 1649950325,
            "useProxies": 0,
            "sentJobDoneNotification": 0,
            "currentTimeUTC": 1649950325,
            "name": "bulkTest",
            "notifyEmail": "",
            "pageCrawlSuccesses": 0,
            "pageProcessAttempts": 0
        }
    ]
}
json
{
    "response": "Successfully added urls for spidering.",
    "jobs": [
        {
            "jobStatus": {
              "message": "Job paused.",
              "status": 6
            },
            "maxHops": -1,
            "downloadJson": "...json",
            "urlProcessPattern": "",
            "jobCompletionTimeUTC": 0,
            "maxRounds": -1,
            "type": "bulk",
            "pageCrawlSuccessesThisRound": 0,
            "urlCrawlRegEx": "",
            "pageProcessPattern": "",
            "apiUrl": "https://api.diffbot.com/v3/analyze",
            "useCanonical": 1,
            "jobCreationTimeUTC": 1649950325,
            "repeat": 0,
            "downloadUrls": "...csv",
            "obeyRobots": 1,
            "roundsCompleted": 0,
            "pageCrawlAttempts": 0,
            "notifyWebhook": "",
            "pageProcessSuccessesThisRound": 0,
            "customHeaders": {},
            "objectsFound": 0,
            "roundStartTime": 0,
            "urlCrawlPattern": "",
            "seedRecrawlFrequency": -1,
            "urlProcessRegEx": "",
            "pageProcessSuccesses": 0,
            "urlsHarvested": 0,
            "crawlDelay": -1,
            "currentTime": 1649950325,
            "useProxies": 0,
            "sentJobDoneNotification": 0,
            "currentTimeUTC": 1649950325,
            "name": "bulkTest",
            "notifyEmail": "",
            "pageCrawlSuccesses": 0,
            "pageProcessAttempts": 0
        }
    ]
}
json
{
  "response": "Successfully deleted job."
}
json
{
    "jobs": [
        {
            "jobStatus": {
                "message": "Job has completed and no repeat is scheduled.",
                "status": 9
            },
            "maxHops": -1,
            "downloadJson": "...json",
            "urlProcessPattern": "",
            "jobCompletionTimeUTC": 0,
            "maxRounds": -1,
            "type": "bulk",
            "pageCrawlSuccessesThisRound": 0,
            "urlCrawlRegEx": "",
            "pageProcessPattern": "",
            "apiUrl": "https://api.diffbot.com/v3/analyze",
            "useCanonical": 1,
            "jobCreationTimeUTC": 1649950325,
            "repeat": 0,
            "downloadUrls": "...csv",
            "obeyRobots": 1,
            "roundsCompleted": 0,
            "pageCrawlAttempts": 0,
            "notifyWebhook": "",
            "pageProcessSuccessesThisRound": 0,
            "customHeaders": {},
            "objectsFound": 0,
            "roundStartTime": 0,
            "urlCrawlPattern": "",
            "seedRecrawlFrequency": -1,
            "urlProcessRegEx": "",
            "pageProcessSuccesses": 0,
            "urlsHarvested": 0,
            "crawlDelay": -1,
            "currentTime": 1649950325,
            "useProxies": 0,
            "sentJobDoneNotification": 0,
            "currentTimeUTC": 1649950325,
            "name": "bulkTest",
            "notifyEmail": "",
            "pageCrawlSuccesses": 0,
            "pageProcessAttempts": 0
        }
    ]
}
500No job found by that name
errorCodestring
errorstring
json
{
    "errorCode": "500",
    "error": "no collection record."
}