2016 Changelog ​

December 2016

  • 2016-12-09

    • Product API: overriding the sku, mpn or related fields using custom rules will now affect the productId field as well.
    • Crawls using the Analyze API will now correctly index video pages.
    • Improved the reliability of the fields=links argument in all Automatic APIs.

  • 2016-12-01

    Updates to our rendering engine to properly support more Unicode scripts

November 2016

  • 2016-11-30

    • Updates to our status page for improved coverage and reliability.
    • Crawlbot crawls can now have repeat settings adjusted or added after a crawl completes.
    • Fixed a Crawlbot issue wherein users could completely erase the seeds field.

  • 2016-11-13

    • POSTing to our APIs is speedier, particularly when content includes slow-loading third-party assets.
    • Crawlbot now has limited support for crawling/processing content across multiple domains.

  • 2016-11-06

    • Improvements to background image detection and extraction across all Automatic APIs. This resolves many issues with proper extraction from sites that use background CSS properties for image delivery.
    • Improvements to specification extraction in the Product API.
    • Improvements to HTML <figure> parsing in the Article API.

October 2016

  • 2016-10-24

    • The diffbotUri value for Custom APIs is now calculated from the entirety of custom content (and can be used to detect changes between extractions).
    • Various improvements to caption detection and parsing in the Article API.
    • Crawlbot now adheres to the "Diffbot" user-agent in robots.txt directives, so that our crawling can be whitelisted when crawling partner or other sites.

  • 2016-10-04

    • Increased the size limits for content POSTed to Diffbot APIs.
    • Bulk Service jobs now require a minimum of 50 URLs for Startup plan customers.
    • Bulk Service and Crawlbot jobs now automatically retry failing URLs.

September 2016

  • 2016-09-12

    • Numerous improvements to normalizedSpecs in the Product API.
    • Diffbot Automatic APIs now process PDFs. PDF URLs will be converted to HTML and then analyzed for extractable content. PDFs are not currently supported while crawling.
    • Crawlbot fixes to reduce DNS errors when starting new crawls or crawl rounds.
    • Crawlbot and Bulk Processing: deletion of a nonexistent job will no longer return a "success" message.
    • Improved handling of UTF-8 encoded characters within Crawlbot.

June 2016

  • 2016-06-24

    • Fixed an issue where large Crawlbot and Bulk job downloads would prematurely terminate.
    • Added beta support for executing custom Javascript before processing a page via an extraction API. See Analyze API example (works with all Automatic and Custom APIs).

  • 2016-06-17

    • Added support for custom headers to the Crawlbot and Bulk Job interfaces.
    • Added beta field inferredCategory_beta to the Product API, which provides an automatically-determined category for any extracted product.
    • The Bulk Service will no longer normalize URLs before processing them—all pages will be sent to the specified Diffbot API as-is.
    • Various improvements to performance and quality of Diffbot's rendering engine.
    • Corrected an issue in Custom APIs where a replacement rule would errantly trim blank spaces.

May 2016

  • 2016-05-24

    • Crawlbot now supports custom headers while crawling; Bulk Processing jobs now support custom headers for all URLs.
    • Fixed an issue in Crawlbot where internal JSON objects were sometimes being returned in JSON data downloads.
    • Various improvements to date-parsing and normalization in the Article API.
    • Improvements to "Replacement" and "Ignore" filters in Custom APIs and manual rules.

  • 2016-05-12

    • Added column to the Crawlbot and Bulk API URL Report indicating if a proxy IP was used.
    • Added the argument useProxies to Crawlbot, which allows for proxy IPs to be used for specific crawls

  • 2016-05-05

    Added proxy usage tracking to the Account API.

April 2016

  • 2016-04-28

    • Added available colors (if found on the page) to the normalizedSpecs object in the Product API. Updated format of normalizedSpecs to return multiple values, if available, for a single key.
    • Fixed an issue where image URLs with spaces in the filename would be incorrectly returned.
    • Improved proxy support in the extraction APIs to help diversify origin IPs. Read more.

  • 2016-04-15

    • Released the tagConfidence argument in the Article API, allowing for the return of tags with lower relevance scores if desired.
    • Improved Crawlbot handling of DNS and other connection issues; increased range of TLDs supported by Crawlbot.
    • Fixed an issue where duplicate tags were being returned when sentiment analysis was being performed.

March 2016

  • 2016-03-25

    • Updated Crawlbot seeds behavior, so that if a non-www subdomain is specified as the only seed URL, crawling will restrict itself to that subdomain.
    • Significant updates to the beta normalizedSpecs field in the Product API. See more details.
    • Added the field parentUrlDocId to Crawlbot and Bulk Processing JSON objects. This field can be used to match objects to URLs in the Crawlbot or Bulk Processing URL Report.

  • 2016-03-10

    • Added the originalType field to extracted objects when utilizing the Analyze API's' fallback argument.
    • Fixed an issue with our Semantria integration that could lead to errant timeout responses.

February 2016

  • 2016-02-25

    • Fixed an issue in the Article API to prevent in-line Javascript and CSS from being returned in the html field from unsupported video players.
    • Discussion API: Improved extraction from single-post (no reply) conversations.
    • Improvements to video extraction within the Video API.

January 2016

  • 2016-01-29

    • Added beta fields quantityPrices, priceRange and multiplePrices to the Product API.
    • Improved availability detection and extraction in the Product API.
    • Improved offerPrice detection in the Product API to reduce the chance of returning an incorrect value from unavailable products or items without a visible price.

  • 2016-01-26

    Significant speed improvements to the Global Index.

  • 2016-01-21

    • Released an official endpoint for Custom API management. Please see the documentation for information on programmatic management of custom rules and APIs.
    • Improved video extraction in the Article API to include new providers and HTML5 <video> elements.
    • max:date queries in the Search and Global Index APIs are now inclusive of the date specified.

  • 2016-01-14

    • Improved specification extraction in the Product API.
    • Fixed an issue where the estimatedDate field (Article API) would sometimes not be correctly computed.

  • 2016-01-07

    • Fixed an issue where the <base> element could be incorrectly use to calculate relative paths.
    • Added initial functionality to categorize articles in the Article API based on article text content. If you would like to test this beta feature, contact us.
    • Improved handling of media sources without a specified protocol (e.g. src="//www.youtube.com...). Media element URLs will now match the protocol of the analyzed page.