Create a Crawl ​

POSThttps://api.diffbot.com/v3/crawl

To create a crawl, make a POST request to this endpoint with Content-Type set to application/x-www-form-urlencoded and include the minimum settings specified below. Please note that this API does not accept JSON payloads.

Creating a crawl job will instruct Diffbot to immediately start spidering through the provided seed URLs for links and process them with a specified Extract API.

Additional settings are available to crawl only links that match a certain URL pattern, or extract only some crawled links.

See also: Managing a Crawl Job and Retrieving Crawl Job Data

Example ​

This request initiates a crawl on example.com using the Analyze API. This is a one page site, so crawling should be complete within seconds.

bash
curl --location 'https://api.diffbot.com/v3/crawl?token=<DIFFBOT_TOKEN>' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'name=test-crawl' \
--data-urlencode 'seeds=https://example.com' \
--data-urlencode 'apiUrl=https://api.diffbot.com/v3/analyze' \
--data-urlencode 'maxToCrawl=100'
json
{
  "response": "Successfully added urls for spidering.",
  "jobs": [
    {
      "maxToProcessPerSubdomain": -1,
      "jobStatus": {
        "message": "Job is initializing.",
        "status": 0
      },
      "maxHops": -1,
      "onlyProcessIfNew": 1,
      "downloadJson": "https://api.diffbot.com/v3/crawl/download/<REDACTED>.json",
      "urlProcessPattern": "",
      "jobCompletionTimeUTC": 0,
      "maxRounds": -1,
      "restrictDomain": 1,
      "type": "crawl",
      "pageCrawlSuccessesThisRound": 0,
      "urlCrawlRegEx": "",
      "pageProcessPattern": "",
      "apiUrl": "https://api.diffbot.com/v3/analyze",
      "useCanonical": 1,
      "maxToCrawlPerSubdomain": -1,
      "jobCreationTimeUTC": 1726163287,
      "repeat": 0,
      "downloadUrls": "https://api.diffbot.com/v3/crawl/download/<REDACTED>.csv",
      "obeyRobots": 1,
      "roundsCompleted": 0,
      "pageCrawlAttempts": 0,
      "notifyWebhook": "",
      "pageProcessSuccessesThisRound": 0,
      "customHeaders": {},
      "objectsFound": 0,
      "roundStartTime": 0,
      "urlCrawlPattern": "",
      "maxToProcess": 100000,
      "seedRecrawlFrequency": -1,
      "seeds": "https://example.com",
      "urlProcessRegEx": "",
      "pageProcessSuccesses": 0,
      "urlsHarvested": 0,
      "crawlDelay": 0.25,
      "currentTime": 1726163288,
      "useProxies": 0,
      "maxToCrawl": 100000,
      "sentJobDoneNotification": 0,
      "currentTimeUTC": 1726163288,
      "restrictSubdomain": 0,
      "name": "test-crawl",
      "notifyEmail": "",
      "pageCrawlSuccesses": 0,
      "pageProcessAttempts": 0
    }
  ]
}

API Definition

Generated from v1.1 of the Crawl API OpenAPI specification.

Create and start a job to spider and extract pages through a site.

Request

Authentication
tokenquery
Cache and insert token into example requests (15d expiry)
Request body application/x-www-form-urlencoded
namestring required

Job name. This should be a unique identifier and can be used to modify your crawl or retrieve its output.

seedsstring required

Seed URL(s). Must be URL encoded. Separate multiple URLs with whitespace to spider multiple sites within the same crawl. If the seed contains a non-www subdomain (“https://blog.diffbot.com” or “https://docs.diffbot.com”) Crawl will restrict spidering to the specified subdomain.

apiUrlstring required

Full Extract API URL through which to process pages. E.g., &apiUrl=https://api.diffbot.com/v3/analyze to process matching links via the Analyze API, which will automatically determine the page type to extract as. The Extract API URL can include querystring parameters to tailor the output.

For example, &apiUrl=https://api.diffbot.com/v3/product?fields=querystring,meta will process matching links using the Product API, and also return the querystring and meta fields.

customHeadersstring

Set multiple custom headers to be used while crawling and processing pages sent to Diffbot APIs. Each header should be sent in its own customHeaders argument, with a colon delimiting the header name and value, and should be URL-encoded. For example, &customHeaders=Accept-Language%3Aen-us. See more on using custom headers.

useCanonicalinteger

Pass useCanonical=0 to disable deduplication of pages based on a canonical link definition. See more. (default: 1)

obeyRobotsinteger

Pass obeyRobots=0 to ignore a site’s robots.txt instructions. (default: 1)

restrictDomaininteger

Pass restrictDomain=0 to allow limited crawling across subdomains/domains. See more. (default: 1)

useProxiesinteger

Set value to 1 to force the use of proxy IPs for the crawl. This will utilize proxy servers for both crawling and processing of pages. (default: 0)

maxHopsinteger

Specify the depth of your crawl. A maxHops=0 will limit processing to the seed URL(s) only – no other links will be processed; maxHops=1 will process all (otherwise matching) pages whose links appear on seed URL(s); maxHops=2 will process pages whose links appear on those pages; and so on.

By default (maxHops=-1) Crawl will crawl and process links at any depth.

(default: -1)

notifyEmailstring

Send a message to this email address when the crawl hits the maxToCrawl or maxToProcess limit, or when the crawl completes.

notifyWebhookstring

Pass a URL to be notified when the crawl hits the maxToCrawl or maxToProcess limit, or when the crawl completes. You will receive a POST with X-Crawl-Name and X-Crawl-Status in the headers, and the job’s JSON metadata in the POST body. Note that in webhook POSTs the parent jobs will not be sent—only the individual job object will be returned.

repeatnumber

Specify the number of days as a floating-point (e.g. repeat=7.0) to repeat this crawl. By default crawls will not be repeated.

seedRecrawlFrequencynumber

Useful for specifying a frequency, in number of days, to recrawl seed urls, which is independent of the overall recrawl frequency given by repeat. Defaults to seedRecrawlFrequency=-1 to use the default frequency. (default: -1)

maxRoundsinteger

Specify the maximum number of crawl repeats. By default (maxRounds=0) repeating crawls will continue indefinitely. (default: 0)

urlCrawlPatternstring

Specify ||-separated strings to limit pages crawled to those whose URLs contain any of the content strings. You can use the exclamation point to specify a negative string, e.g. !product to exclude URLs containing the string “product,” and the ^ and $ characters to limit matches to the beginning or end of the URL.

The use of a urlCrawlPattern will allow Crawl to spider outside of the seed domain; it will follow all matching URLs regardless of domain.

urlCrawlRegExstring

Specify a regular expression to limit pages crawled to those URLs that contain a match to your expression. This will override any urlCrawlPattern value.

maxToCrawlinteger

Specify max pages to spider. Default: 100,000. (default: 100000)

maxToCrawlPerSubdomaininteger

Specify max pages to spider per subdomain. Default: no limit (-1) (default: -1)

crawlDelaynumber

Wait this many seconds between each URL crawled from a single IP address. Specify the number of seconds as an integer or floating-point number (e.g., crawlDelay=0.25).

urlProcessPatternstring

Specify ||-separated strings to limit pages processed to those whose URLs contain any of the content strings. You can use the exclamation point to specify a negative string, e.g. !/category to exclude URLs containing the string “/category,” and the ^ and $ characters to limit matches to the beginning or end of the URL.

urlProcessRegExstring

Specify a regular expression to limit pages processed to those URLs that contain a match to your expression. This will override any urlProcessPattern value.

pageProcessPatternstring

Specify ||-separated strings to limit pages processed to those whose HTML contains any of the content strings.

maxToProcessinteger

Specify max pages to process through Diffbot APIs. Default: 100,000. (default: 100000)

maxToProcessPerSubdomaininteger

Specify max pages to process per subdomain. Default: no limit (-1) (default: -1)

onlyProcessIfNewinteger

By default repeat crawls will only process new (previously unprocessed) pages. Set to 0 (onlyProcessIfNew=0) to process all content on repeat crawls. (default: 1)

Example request

bash
curl --request POST \
  --url 'https://api.diffbot.com/v3/crawl?token=<DIFFBOT_TOKEN>' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-raw '{"name":"test-crawl","seeds":"https://example.com","apiUrl":"https://api.diffbot.com/v3/analyze"}'

Response

200Crawl job created
responsestring
jobsobject[]
maxToProcessPerSubdomaininteger
jobStatusobject
messagestring
statusinteger
maxHopsinteger
onlyProcessIfNewinteger
downloadJsonstring
urlProcessPatternstring
jobCompletionTimeUTCinteger
maxRoundsinteger
restrictDomaininteger
typestring
pageCrawlSuccessesThisRoundinteger
urlCrawlRegExstring
pageProcessPatternstring
apiUrlstring
useCanonicalinteger
maxToCrawlPerSubdomaininteger
jobCreationTimeUTCinteger
repeatinteger
downloadUrlsstring
obeyRobotsinteger
roundsCompletedinteger
pageCrawlAttemptsinteger
notifyWebhookstring
pageProcessSuccessesThisRoundinteger
customHeadersobject
objectsFoundinteger
roundStartTimeinteger
urlCrawlPatternstring
maxToProcessinteger
seedRecrawlFrequencyinteger
seedsstring
urlProcessRegExstring
pageProcessSuccessesinteger
urlsHarvestedinteger
crawlDelaynumber
currentTimeinteger
useProxiesinteger
maxToCrawlinteger
sentJobDoneNotificationinteger
currentTimeUTCinteger
restrictSubdomaininteger
namestring
notifyEmailstring
pageCrawlSuccessesinteger
pageProcessAttemptsinteger
json
{
  "response": "Successfully added urls for spidering.",
  "jobs": [
    {
      "maxToProcessPerSubdomain": -1,
      "jobStatus": {
        "message": "Job is initializing.",
        "status": 0
      },
      "maxHops": -1,
      "onlyProcessIfNew": 1,
      "downloadJson": "https://api.diffbot.com/v3/crawl/download/<REDACTED>.json",
      "urlProcessPattern": "",
      "jobCompletionTimeUTC": 0,
      "maxRounds": -1,
      "restrictDomain": 1,
      "type": "crawl",
      "pageCrawlSuccessesThisRound": 0,
      "urlCrawlRegEx": "",
      "pageProcessPattern": "",
      "apiUrl": "https://api.diffbot.com/v3/analyze",
      "useCanonical": 1,
      "maxToCrawlPerSubdomain": -1,
      "jobCreationTimeUTC": 1726163287,
      "repeat": 0,
      "downloadUrls": "https://api.diffbot.com/v3/crawl/download/<REDACTED>.csv",
      "obeyRobots": 1,
      "roundsCompleted": 0,
      "pageCrawlAttempts": 0,
      "notifyWebhook": "",
      "pageProcessSuccessesThisRound": 0,
      "customHeaders": {},
      "objectsFound": 0,
      "roundStartTime": 0,
      "urlCrawlPattern": "",
      "maxToProcess": 100000,
      "seedRecrawlFrequency": -1,
      "seeds": "https://example.com",
      "urlProcessRegEx": "",
      "pageProcessSuccesses": 0,
      "urlsHarvested": 0,
      "crawlDelay": 0.25,
      "currentTime": 1726163288,
      "useProxies": 0,
      "maxToCrawl": 100000,
      "sentJobDoneNotification": 0,
      "currentTimeUTC": 1726163288,
      "restrictSubdomain": 0,
      "name": "test-crawl",
      "notifyEmail": "",
      "pageCrawlSuccesses": 0,
      "pageProcessAttempts": 0
    }
  ]
}
505Too many collections for token — the 1,000-crawl limit has been reached
errorCodeinteger
errorstring
json
{
    "errorCode":505,
    "error":"Too many collections for token"
}