2017 Changelog ​

November 2017

2017-11-10

  • Significant improvements to Video API site support.

October 2017

2017-10-30

  • Custom API fields using the attribute filter will now return all matching selector values, not just the first attribute match.

2017-10-25

Crawlbot and Bulk Service data retrieval no longer requires access to port :18100. Data downloads are also now HTTPS-only.

2017-10-16

  • Fixed a rare issue where custom rules could be accidentally deleted.
  • Significant performance improvements in the Search API.
  • Improved crawling performance and site coverage in the Global Index.
  • Improved ability to identify, analyze and return background images in all extraction APIs.

August 2017

2017-08-31

  • Fixed an issue in the Video API where the url value would retain HTML escaping if present within the original page source.
  • Fixed a rare crawling issue that occasionally resulted in "Bad IP" status messages for individual pages.
  • Fixed an issue where empty <video> elements could be returned in the Article API.

2017-08-15

Fixed an issue in the Global Index in which complicated Boolean (OR) queries would return no results.

2017-08-08

  • Improved date normalization to include Hijri and Jalali dates
  • Fixed support for unicode characters in API Toolkit rules

May 2017

2017-05-22

  • Many improvements to brand detection in the Product API.
  • Resolved an issue where humanLanguage could be mis-identified on some Spanish-language pages.

2017-05-15

  • Crawlbot: resolved an issue where IP-address-only webhooks would not receive notifications.
  • Crawlbot: improved link spidering/harvesting resilience to markup errors and other invalid HTML source.
  • Fixed an issue where custom APIs would not display in Crawlbot and Bulk Processing dashboard.

2017-05-11

  • Improved link-detection when returning page links via our &fields=links argument.
  • Improved support for and handling of the srcset (and sizes) image attributes in all APIs.
  • Added detection of Afrikaans (af) in the humanLanguage field.
  • Improved duplicate detection in the Diffbot Global Index.

April 2017

2017-04-21

  • The beta category field has been added to the Product API. See documentation.
  • All extraction APIs now support the sending of completely custom headers using X-Forward- terminology. Previously only four defined headers were supported.

2017-04-10

  • In the Article and Discussion APIs' tags element, DBPedia uri values are now properly URL-encoded.
  • Fixed an issue when sorting by date in the Search API.
  • Various improvements and fixes to the Global Index

January 2017

2017-01-12

  • The Account API now tracks Global Index search calls/requests.
  • Improved SKU detection and extraction in the Product API.
  • Article API: Added support for the start attribute (ol elements) and data- attributes in normalized HTML.
  • In the Article API, identified image captions will no longer be returned in the text field content.
  • Various improvements to replacement rule regular expressions in Custom APIs.
  • PDF processing improvements.