Retrieve Crawl Job Data ​

GEThttps://api.diffbot.com/v3/crawl/data

To download the results of your crawl job's extracted URLs, make a GET request to this endpoint. By default the complete extracted JSON data will be downloaded.

See also: Creating a Crawl and Managing a Crawl Job

Download a URL Report ​

The URL Report is a CSV file that provides a diagnostic on every URL crawled by a Crawl API job and does not include extraction output. To retrieve the URL report, pass type=urls to your GET request on this endpoint.

Example ​

bash
curl --get https://api.diffbot.com/v3/crawl/data \
  --data-urlencode "token=<DIFFBOT_TOKEN>" \
  --data-urlencode "name=test-crawl"

API Definition

Generated from v1.1 of the Crawl API OpenAPI specification.

Download the extracted results of a crawl job.

Request

Authentication
tokenquery
Cache and insert token into example requests (15d expiry)
Query parameters
namestring required

Name of the crawl job whose data you wish to download.

formatstring

Request format=csv to download the extracted data in CSV format (default: json). Note that CSV files will only contain top-level fields.

typestring

Request type=urls to retrieve the URL Report (CSV).

numstring

Pass an integer value (e.g. num=100) to request a subset of URLs, most recently processed first.

Example request

bash
curl --request GET \
  --url 'https://api.diffbot.com/v3/crawl/data?name=%3Cname%3E&token=<DIFFBOT_TOKEN>' \
  --header 'Accept: application/json'

Response

200The crawl job's extracted data
colstring

Name of the Crawl or Bulk job this record was extracted from

parentUrlstring

Immediate parent contained a link to this record

pageUrlstring

The URL of this record

idinteger

A reference

typestring

A standard entity type assigned either on job creation or classified by Analyze API. All other keys on this job record follow the ontology set by its type.

tokenstring

Your token

timestampstring

Time when record was crawled, in ISO 8601 format.

json
[
    {
        "col": "example-test",
        "parentUrl": "https://www.example.com",
        "pageUrl": "https://www.example.com/",
        "id": 1492074227214006500,
        "type": "other",
        "title": "Example Domain",
        "content": "Example Domain \n\n# Example Domain \n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations. \n\n[Learn more](https://iana.org/domains/example)",
        "token": "<REDACTED>",
        "timestamp": "2026-07-24T23:15:09Z"
    }
]
400No crawl job by that name
errorCodeinteger
errorstring
json
{
  "errorCode": 400,
  "error": "No collection record"
}