2017 Changelog ​

November 2017

  • 2017-11-10

    • Significant improvements to Video API site support.

October 2017

  • 2017-10-30

    • Custom API fields using the attribute filter will now return all matching selector values, not just the first attribute match.

  • 2017-10-25

    Crawlbot and Bulk Service data retrieval no longer requires access to port :18100. Data downloads are also now HTTPS-only.

  • 2017-10-16

    • Fixed a rare issue where custom rules could be accidentally deleted.
    • Significant performance improvements in the Search API.
    • Improved crawling performance and site coverage in the Global Index.
    • Improved ability to identify, analyze and return background images in all extraction APIs.

August 2017

  • 2017-08-31

    • Fixed an issue in the Video API where the url value would retain HTML escaping if present within the original page source.
    • Fixed a rare crawling issue that occasionally resulted in "Bad IP" status messages for individual pages.
    • Fixed an issue where empty <video> elements could be returned in the Article API.

  • 2017-08-15

    Fixed an issue in the Global Index in which complicated Boolean (OR) queries would return no results.

  • 2017-08-08

    • Improved date normalization to include Hijri and Jalali dates
    • Fixed support for unicode characters in API Toolkit rules

May 2017

  • 2017-05-22

    • Many improvements to brand detection in the Product API.
    • Resolved an issue where humanLanguage could be mis-identified on some Spanish-language pages.

  • 2017-05-15

    • Crawlbot: resolved an issue where IP-address-only webhooks would not receive notifications.
    • Crawlbot: improved link spidering/harvesting resilience to markup errors and other invalid HTML source.
    • Fixed an issue where custom APIs would not display in Crawlbot and Bulk Processing dashboard.

  • 2017-05-11

    • Improved link-detection when returning page links via our &fields=links argument.
    • Improved support for and handling of the srcset (and sizes) image attributes in all APIs.
    • Added detection of Afrikaans (af) in the humanLanguage field.
    • Improved duplicate detection in the Diffbot Global Index.

April 2017

  • 2017-04-21

    • The beta category field has been added to the Product API. See documentation.
    • All extraction APIs now support the sending of completely custom headers using X-Forward- terminology. Previously only four defined headers were supported.

  • 2017-04-10

    • In the Article and Discussion APIs' tags element, DBPedia uri values are now properly URL-encoded.
    • Fixed an issue when sorting by date in the Search API.
    • Various improvements and fixes to the Global Index

January 2017

  • 2017-01-12

    • The Account API now tracks Global Index search calls/requests.
    • Improved SKU detection and extraction in the Product API.
    • Article API: Added support for the start attribute (ol elements) and data- attributes in normalized HTML.
    • In the Article API, identified image captions will no longer be returned in the text field content.
    • Various improvements to replacement rule regular expressions in Custom APIs.
    • PDF processing improvements.