2015 Changelog ​

December 2015

  • 2015-12-21

    • Crawlbot and Bulk jobs pending delete (per your Diffbot plan) are now identified in the Crawlbot and Bulk interfaces.
    • The API Toolkit now uses Diffbot's custom rendering engine for live web page previews. This should reduce inaccuracies when creating custom rules.

  • 2015-12-18

    • Fixed an issue where plain-text POSTed to the Article API would not perform text analysis (tags, sentiment, language-detection).
    • Improved Crawlbot behavior on Ajax-heavy sites so that pages with the exact same HTML source are no longer deduplicated.
    • Fixed an issue within the Crawlbot and Bulk interfaces where the "Last 500" URL Report was incorrectly returning the first 500.
    • Improved author detection within the Article API.

  • 2015-12-07

    The Analyze API now supports POSTed content.

November 2015

  • 2015-11-27

    The Account API now returns a list of child or sub-tokens.

  • 2015-11-19

    • Fixed an issue in the Analyze API where products with an API-Toolkit-overridden price field would not reflect changes in the "details" field (offerPriceDetails, regularPriceDetails, etc.).
    • Fixed an Article API issue for certain top-level domains where articles dated in the near future (e.g., tomorrow) would incorrectly be returned with a date from the prior year.

  • 2015-11-11

    • Crawlbot will now successfully spider URLs that contain (invalid) UTF-8 characters.
    • Global Index API: search-by-tag can optionally be performed using a tag-match shorthand.

October 2015

  • 2015-10-16

    • Fixed an issue where Crawlbot and Bulk API data downloads did not include a filename.
    • The breadcrumb element is now a default field in the Article API.

  • 2015-10-22

    • We now offer an Account API for tracking token API usage and billing history.
    • Global Index API: negative search queries (diffbot AND -"machine learning") are now functioning as documented;

  • 2015-10-08

    • APIs no longer ignore "format characters"—invisible characters that may have an effect on neighboring characters. For example, ‌.
    • Crawlbot and Bulk Service URL Reports now offer an option to download the last 500 URLs crawled.
    • Global Index API: Faceted date queries will no longer return a min value of 0.

  • 2015-10-01

    Hey, as of today we're publishing a changelog. It's visible... here.

September 2015

  • 2015-09-23

    • Additional token support within a single account has been added. Additional tokens are available on a case-by-case basis to paying customers. Please contact support@diffbot.com if you would like to discuss additional tokens.
    • API Toolkit now allows direct update of URL pattern / regular expression without having to create a new ruleset.
    • API Toolkit rule output automatically trims fields to remove leading or trailing blank spaces.
    • The diffbotUri field is now computed based on rule-based output, if a custom rule is used to override default output.
    • The resolvedPageUrl is correctly returned in Custom APIs (if a submitted page is redirected).

  • 2015-09-16

    • Each tag in the tags element now returns a list of all matching rdfTypes.
    • Email invoices now return both dollars and cents.

August 2015

  • 2015-08-23

    • Performance improvements to Article API to prevent intermittent extra-long responses.

  • 2015-08-13

    • Semantria output updated to include additional fields.
    • Fix for timeout parameter when sending data to Semantria.

  • 2015-08-10

    Invoices are now visible and printable within the Developer Dashboard (under "Account > Billing History").

  • 2015-08-06

    Tokens are now case-insensitive across all Diffbot APIs.

July 2015

  • 2015-07-16

    • Article API now returns the siteName, publisherCountry and publisherRegion of an article, if it can be determined or if already known.
    • If price data is overridden with a custom rule, the corresponding "details" field (offerPriceDetails) will be computed from the overridden value.

June 2015

  • 2015-06-28

    • Spotify embeds are properly maintained/returned with Article API html.
    • Multipage articles, when concatenated, will no longer return duplicate images (that appear on multiple pages).

  • 2015-06-11

    Article and discussion tagging now supports Spanish and Chinese language tags, in addition to English and French.

February 2015

  • 2015-10-02

    • Analyze API now supports a "fallback" API via the argument fallback. By passing an API value, any otherwise unsupported pages will be forcibly processed via that API. E.g., passing fallback=article will result in any "other" pages being processed by the Article API.