Manage a Crawl Job

GEThttps://api.diffbot.com/v3/crawl

A single endpoint allows both control and status requests for one or more active crawl jobs with any given token.

View the Status of Crawl Jobs

Your token's active crawl jobs (along with any active bulk jobs) will be returned in a jobs object when a GET request supplying just a token parameter is made to this endpoint.

Note that this endpoint without any query parameters returns exactly the same output as its Bulk Job equivalent.

bash

curl --request GET \
     --url https://api.diffbot.com/v3/crawl?token=<DIFFBOT_TOKEN> \
     --header 'Accept: application/json'

To retrieve a single crawl job's details, provide the job's name in addition to your token in your request.

bash

curl --request GET \
     --url https://api.diffbot.com/v3/crawl?token=<DIFFBOT_TOKEN>&name=crawlTest \
     --header 'Accept: application/json'

Pause a Crawl Job

To pause a crawl job, send a GET request to this endpoint supplying your token, the name of the crawl job to pause, and the pause parameter set to 1.

bash

curl --request GET \
     --url https://api.diffbot.com/v3/crawl?token=<DIFFBOT_TOKEN>&name=crawlTest&pause=1 \
     --header 'Accept: application/json'

To resume a paused crawl job, pass pause=0 in the same GET request.

Delete a Crawl Job

To delete a crawl job, send a GET request to this endpoint supplying your token, the name of the crawl job to delete, and the delete parameter set to 1. Job deletions are irreversible.

bash

curl --request GET \
     --url https://api.diffbot.com/v3/crawl?token=<DIFFBOT_TOKEN>&name=crawlTest&delete=1 \
     --header 'Accept: application/json'

Restart a Crawl Job

To restart a crawl job, send a GET request to this endpoint supplying your token, the name of the crawl job to restart, and the restart parameter set to 1. This will erase all previously processed data and re-process all of the submitted URLs.

bash

curl --request GET \
     --url https://api.diffbot.com/v3/crawl?token=<DIFFBOT_TOKEN>&name=crawlTest&restart=1 \
     --header 'Accept: application/json'

Status Codes

The jobStatus object will return the following status codes and associated messages:

Status	Message
0	Job is initializing
1	Job has reached maxRounds limit
2	Job has reached maxToCrawl limit
3	Job has reached maxToProcess limit
4	Next round to start in _ seconds
5	No URLs were added to the crawl
6	Job paused
7	Job in progress
8	All crawling temporarily paused by root administrator for maintenance
9	Job has completed and no repeat is scheduled
10	Failed to crawl any seed Indicates a problem retrieving links from the seed URL(s)
11	Job automatically paused because crawl is inefficient. Successfully downloaded 10000+ consecutive pages without a single successfully processed page

API Definition

Generated from v1.1 of the Crawl API OpenAPI specification.

Pause, delete, restart, or view the status of a crawl job.

Request

Authentication
`token`query	`<DIFFBOT_TOKEN>` Cache and insert token into example requests (15d expiry)

Query parameters
`name`string	Job name as defined when the crawl job was created.
`roundStart`string	Pass `roundStart=1` to force the start of a new crawl “round” (manually repeat the crawl). If `onlyProcessIfNew` is set to 1 (default), only newly-created pages will be processed.
`pause`string	Pass `pause=1` to pause a crawl job. Pass `pause=0` to resume a paused job.
`restart`string	Pass `restart=1` to restart a crawl job. This will erase all processed data and re-process all of the submitted URLs.
`delete`string	Pass `delete=1`to delete a job, and all associated data, completely.

Example request

bash

curl --request GET \
  --url 'https://api.diffbot.com/v3/crawl?token=<DIFFBOT_TOKEN>' \
  --header 'Accept: application/json'

Response

200Job managed
`response`string
`jobs`object[]
`↳ jobStatus`object
`↳ message`string
`↳ status`integer
`↳ maxHops`integer
`↳ downloadJson`string
`↳ urlProcessPattern`string
`↳ jobCompletionTimeUTC`integer
`↳ maxRounds`integer
`↳ type`string
`↳ pageCrawlSuccessesThisRound`integer
`↳ urlCrawlRegEx`string
`↳ pageProcessPattern`string
`↳ apiUrl`string
`↳ useCanonical`integer
`↳ jobCreationTimeUTC`integer
`↳ repeat`integer
`↳ downloadUrls`string
`↳ obeyRobots`integer
`↳ roundsCompleted`integer
`↳ pageCrawlAttempts`integer
`↳ notifyWebhook`string
`↳ pageProcessSuccessesThisRound`integer
`↳ customHeaders`object
`↳ objectsFound`integer
`↳ roundStartTime`integer
`↳ urlCrawlPattern`string
`↳ seedRecrawlFrequency`integer
`↳ urlProcessRegEx`string
`↳ pageProcessSuccesses`integer
`↳ urlsHarvested`integer
`↳ crawlDelay`number
`↳ currentTime`integer
`↳ useProxies`integer
`↳ sentJobDoneNotification`integer
`↳ currentTimeUTC`integer
`↳ name`string
`↳ notifyEmail`string
`↳ pageCrawlSuccesses`integer
`↳ pageProcessAttempts`integer

json

{
    "response": "Successfully added urls for spidering.",
    "jobs": [
        {
            "jobStatus": {
                "message": "Job is initializing.",
                "status": 0
            },
            "maxHops": -1,
            "downloadJson": "...json",
            "urlProcessPattern": "",
            "jobCompletionTimeUTC": 0,
            "maxRounds": -1,
            "type": "crawl",
            "pageCrawlSuccessesThisRound": 0,
            "urlCrawlRegEx": "",
            "pageProcessPattern": "",
            "apiUrl": "https://api.diffbot.com/v3/analyze",
            "useCanonical": 1,
            "jobCreationTimeUTC": 1649950325,
            "repeat": 0,
            "downloadUrls": "...csv",
            "obeyRobots": 1,
            "roundsCompleted": 0,
            "pageCrawlAttempts": 0,
            "notifyWebhook": "",
            "pageProcessSuccessesThisRound": 0,
            "customHeaders": {},
            "objectsFound": 0,
            "roundStartTime": 0,
            "urlCrawlPattern": "",
            "seedRecrawlFrequency": -1,
            "urlProcessRegEx": "",
            "pageProcessSuccesses": 0,
            "urlsHarvested": 0,
            "crawlDelay": -1,
            "currentTime": 1649950325,
            "useProxies": 0,
            "sentJobDoneNotification": 0,
            "currentTimeUTC": 1649950325,
            "name": "crawlTest",
            "notifyEmail": "",
            "pageCrawlSuccesses": 0,
            "pageProcessAttempts": 0
        }
    ]
}

json

{
    "response": "Successfully added urls for spidering.",
    "jobs": [
        {
            "jobStatus": {
              "message": "Job paused.",
              "status": 6
            },
            "maxHops": -1,
            "downloadJson": "...json",
            "urlProcessPattern": "",
            "jobCompletionTimeUTC": 0,
            "maxRounds": -1,
            "type": "crawl",
            "pageCrawlSuccessesThisRound": 0,
            "urlCrawlRegEx": "",
            "pageProcessPattern": "",
            "apiUrl": "https://api.diffbot.com/v3/analyze",
            "useCanonical": 1,
            "jobCreationTimeUTC": 1649950325,
            "repeat": 0,
            "downloadUrls": "...csv",
            "obeyRobots": 1,
            "roundsCompleted": 0,
            "pageCrawlAttempts": 0,
            "notifyWebhook": "",
            "pageProcessSuccessesThisRound": 0,
            "customHeaders": {},
            "objectsFound": 0,
            "roundStartTime": 0,
            "urlCrawlPattern": "",
            "seedRecrawlFrequency": -1,
            "urlProcessRegEx": "",
            "pageProcessSuccesses": 0,
            "urlsHarvested": 0,
            "crawlDelay": -1,
            "currentTime": 1649950325,
            "useProxies": 0,
            "sentJobDoneNotification": 0,
            "currentTimeUTC": 1649950325,
            "name": "crawlTest",
            "notifyEmail": "",
            "pageCrawlSuccesses": 0,
            "pageProcessAttempts": 0
        }
    ]
}

json

{
  "response": "Successfully deleted job."
}

json

{
    "jobs": [
        {
            "jobStatus": {
                "message": "Job has completed and no repeat is scheduled.",
                "status": 9
            },
            "maxHops": -1,
            "downloadJson": "...json",
            "urlProcessPattern": "",
            "jobCompletionTimeUTC": 0,
            "maxRounds": -1,
            "type": "crawl",
            "pageCrawlSuccessesThisRound": 0,
            "urlCrawlRegEx": "",
            "pageProcessPattern": "",
            "apiUrl": "https://api.diffbot.com/v3/analyze",
            "useCanonical": 1,
            "jobCreationTimeUTC": 1649950325,
            "repeat": 0,
            "downloadUrls": "...csv",
            "obeyRobots": 1,
            "roundsCompleted": 0,
            "pageCrawlAttempts": 0,
            "notifyWebhook": "",
            "pageProcessSuccessesThisRound": 0,
            "customHeaders": {},
            "objectsFound": 0,
            "roundStartTime": 0,
            "urlCrawlPattern": "",
            "seedRecrawlFrequency": -1,
            "urlProcessRegEx": "",
            "pageProcessSuccesses": 0,
            "urlsHarvested": 0,
            "crawlDelay": -1,
            "currentTime": 1649950325,
            "useProxies": 0,
            "sentJobDoneNotification": 0,
            "currentTimeUTC": 1649950325,
            "name": "crawlTest",
            "notifyEmail": "",
            "pageCrawlSuccesses": 0,
            "pageProcessAttempts": 0
        }
    ]
}

500No crawl job by that name
`errorCode`string
`error`string

json

{
    "errorCode": "500",
    "error": "no collection record."
}

Authentication

Custom API