googlechromelabs / confluence

Service and API for Web API Confluence: Metrics for web platform health

Web API Confluence Dashboard

A web service and UI for describing API confluence metrics. These metrics are intended to capture the how browser vendors are stretching the elastic band of the web platform.

Stretching is good: Browsers should be implementing new APIs to add value to the platform.

Breaking is bad: Implementing too many new APIs before other browsers catch up, or failing to remove APIs other browsers don't intend to ship causes fragmentation.

The purpose of API Confluence Metrics is to capture the ways in which different browsers risk breaking the elastic band.

Data for this project is collected using BrowserStack.

Table of Contents generated with DocToc

• The Catalog
  • Querying the catalog
    • Examples
• The Metrics
  • API Count
    • Definition
    • Rationale
  • Lone Omission
    • Definition
    • Rationale
  • Lone Removal
    • Definition
    • Rationale
  • Browser-Specific
    • Definition
    • Rationale
• Contributing
  • Filing issues and contributing code
  • Running locally
  • Collecting data

The Catalog

The dashboard contains an API catalog that lists attributes/properties and operations/methods exposed on constructor functions and their prototypes. The catalog constitutes the raw data from which aggregate API confluence metrics are computed. See CatalogDataCollection.md for details on how the catalog is created.

Querying the catalog

The catalog supports structured queries. Some query atoms apply to all cataloged browser releases, while others apply to the releases currently in view (i.e., the releases currently shown as columns in the table of APIs).

Query atoms may be joined by whitespace, conjunction (and or &), or disjunction (or or |), with parentheses to disambiguate as needed. Atoms are one of the following:

• (Not-)in-releases clause: A phrase of the form in:release or notin:release where release is identified by case-insensitive [release-name-prefix][release-version-prefix][os-name-prefix][os-version-prefix]. Any of these, except [release-name-prefix] may be empty. For example, in:fir59 describes APIs shipped in all releases of Firefox 59 (that are included in the catalog). These atoms apply to all releases.
• Count-of-releases clause: A phrase of the form count:n where n is a non-negative integer describes APIs that are shipped in exactly n releases currently in view.
• Keyword: An atom matching the regular expression [a-zA-Z0-9_#-]+ describes APIs that contain the atom by case-insensitive substring match.

Examples

window# count:1: APIs on intefaces with the case-insensitive window suffix that are shipped in exactly one of the releases in view.

count:1 or count:2 or count:3 or count:4: On a view with showing four or fewer releases, APIs that are shipped by at least one release in view.

in:chrome65 and notin:chrome66: APIs removed in Chrome 66.

The Metrics

API confluence metrics are a count of “APIs” that meet specific criteria with respect to browser releases that include these “APIs”.

Definition: API: For the purposes of these metrics, an “API” is an interface name + attribute or operation pair.

Definition: The Browser: Each API Confluence Metric is computed with respect to some particular browser; this is what’s meant by The Browser. E.g., the “Lone Removal metric for Safari on 2016-09-01” describes APIs that Safari once provided (but no longer does) that where the latest release of all other browsers a year later contains the APIs; in this case Safari is The Browser.

Definition: Grace Period: Most metrics are deliberately calculated with respect to releases of browsers other than The Browser sometime in the past. This avoids penalizing The Browser for making a change (i.e., shipping or removing an API) when other browsers respond in kind. Currently, the Grace Period used for all metrics that have one is one year. The "a year later" in the above example refers to the Lone Removal Grace Period.

API Confluence metrics are API counts assessed for a particular browser at a particular point in time. Most metrics are computed on every date that any browser has a major release. Some metrics are only computed on dates when The Browser has a major release.

API Count

Definition

The API Count metric contains three values; the total number of APIs provided as of the latest browser release, the number of APIs removed (since the previous release) and the number of APIs added (since the previous release). This metric is computed on dates when The Browser has a major release.

Rationale

When browsers move too slowly, it holds back the platform. When browsers move too quickly, they risk “leaving other browsers behind”. Steady growth is good; wild variation is bad.

Lone Omission

Definition

The Lone Omission metric indicates the number of APIs that The Browser does not provide provide for the duration of the Grace Period, but all other browsers do provide throughout the Grace Period.

Rationale

Failing to ship an API that other major vendors provide requires web developers to use special code paths to remain interoperable. Smaller values are good; larger values are bad.

Lone Removal

Definition

The Lone Removal metric indicates the number of APIs removed from a The Browser prior to the Grace Period, that have not been added back in the latest relase following the Grace Period, and that are provided in all other browsers in the latest relase following the Grace Period.

Rationale

Removing an API from only one browser risks breaking existing sites that (reasonably) assume that all browsers support the API. Smaller values are good; larger values are bad.

Browser-Specific

Definition

The Browser-Specific metric indicates the number of APIs that The Browser provides for the duration of the Grace Period, but all other browsers do not provide throughout the Grace Period.

Rationale

Adding APIs that are provided by only one browser makes that browser more and more like its own platform (rather than an implementation of a common web platform). Smaller values are good; larger values are bad.

Contributing

Want to contribute to Web API Confluence? Great!

Filing issues and contributing code

Please use GitHub’s issue tracker and pull request features.

Running locally

1. Clone this repository.

2. Install: npm install

3. Launch the local server:

mkdir -p data/json

Then, either:

1. Copy the latest data:

cd data/json
curl https://storage.googleapis.com/web-api-confluence-data-cache/latest/json/org.chromium.apis.web.ApiCountData.json > org.chromium.apis.web.ApiCountData.json
curl https://storage.googleapis.com/web-api-confluence-data-cache/latest/json/org.chromium.apis.web.ReleaseWebInterfaceJunction.json > org.chromium.apis.web.ReleaseWebInterfaceJunction.json
curl https://storage.googleapis.com/web-api-confluence-data-cache/latest/json/org.chromium.apis.web.BrowserMetricData.json > org.chromium.apis.web.BrowserMetricData.json
curl https://storage.googleapis.com/web-api-confluence-data-cache/latest/json/org.chromium.apis.web.WebInterface.json > org.chromium.apis.web.WebInterface.json
curl https://storage.googleapis.com/web-api-confluence-data-cache/latest/json/org.chromium.apis.web.Release.json > org.chromium.apis.web.Release.json
cd ../..

or

2. Collect the data yourself.

Finally, use npm run serve to launch a local instance of the service. This will load local data, which can take up to a minute to be ready to serve.

4. Hack away! npm run serve uses webpack --watch to observe local changes. Making changes to server code will require a service restart, but client-side changes will be reflected soon after they are saved.

Collecting data

NOTE: The current data collection process requires a BrowserStack account, two separate git clones, and a whole lot of RAM. We hope to streamline and simplify this process soon. If you have all the prerequisites, read on…

1. Clone mdittmer/web-apis and follow the data collection instructions for historical data collection using BrowserStack.

2. Create /path/to/confluence/data/object-graph and copy /path/to/web-apis/data/og/*.json into it.

3. Create /path/to/confluence/data/json and run ./scripts/og_to_confluence.sh to derive confluence data from the object graphs.

4. To run the service locally based on your generated data invoke node main/serve.js "LOCAL" "DEV". If you want live reloading of client code, change the parameters passed to main/serve.js in scripts/serve.sh and start webpack alongside the service with `npm run serve.

Caveat: In order to serve the data you collect, you must ensure that a { <browser name}: { <browser version prefix>: <release date> } } for every version you have imported appears in data/version_history.json.

confluence's Issues

Consider adding Modernizr data as "APIs"

Modernizr is a hand-curated list of feature detects. In many ways the status of each of these detects is as relevant to API confluence as the presence/absense of a JS API. In some cases they're the same (eg. Modernizr.JSON is a simple combination of JS API checks (and so redundant with our API confluence data), but in other cases they're much more interesting things that couldn't be automated generically (eg. passive event listeners).

I think it could be reasonable to just add ALL the Modernizr feature detects as additional "APIs" in our confluence list (though we might want an option to filter them out in some views).

/cc @patket

Test & implement API Extractor

@jingt06 PTAL & link from PR comments.

Batch data during initial load and paging

The client worker script and ApiMatrix should be smart enough to fetch smaller batches of data and only produce a matrix of data relevant to the current page (and maybe a couple surrounding pages).

Add non-grace-period view on browser-specific APIs

@foolip says that in addition to the "1 year grace period" view of browser-specific APIs, he'd like to also see a picture of the debt we're creating the minute we are the first to ship a new API. I.e. at any given point in time, how many APIs exist for the one browser but no others.

I don't think this data will be particularly actionable (we don't consider shipping first to be problem in and of itself), but we should see a saw-tooth pattern that gives some indication of how we're creating temporary "stretch" in the band which later gets relaxed.

"Browser-Specific" APIs don't show the WebKit legacy

The metrics doc defines browser-specific as:

How many APIs does this engine ship that appear in fewer than 2 other engines for at least a year?

README.md, confluence.html, and the metric calculator (lib/confluence/browser_specific.es6.js) document and implement this as:

How many APIs does this engine ship that appear in no other engines for at least a year?

This is important to fix especially because the common WebKit lineage of Safari and Chrome deflate their metric.

API velocity graph point popup cannot be hidden

Steps to reproduce:

1. In an API velocity graph, click any point
2. Scroll to another graph and click that point
3. There are now two popups showing. Click outside to close the second of them.

The first popup now sits around, impossible to dismiss

Catalog search is broken

A promise rejection error is logged when the search action is committed.

Rename: browser -> release; vendor -> browser

Based on discussion of #57 we should rename classes and variables named browser* (except browserName and browserVersion) to release*, because we intend a particular release of a browser. Similarly, comments and variable names that refer to vendor actually refer to a particular browser (which has multiple releases).

@arobins does that sound right?

Add document explaining how interfaces and APIs are identified.

Catalog search broken on master

Some flow in catalog search is throwing (client-side) on master branch.

Download CSV is failing

I get "failed - Network error" whenever I try to use "download CSV"

window.Screen APIs are (mostly) missing

window.Screen.prototype should have a bunch of properties eg. availTop, but I don't see them in the raw data (Except for a couple Chrome releases under an interface named ))).

Add GitHub link to the UI

So people know where to go to file bugs / feature requests

Write metric computation tests

Computation routines are landing on the tail end of @jingt06 's internship. We haven't written tests for them yet 😢

Custom "API Confluence" page views causing performance problems

Creating custom views by interacting with confluence graphs (except for API Velocity, they are okay) triggers an expensive query and data rehashing procedure.

The real problem here is that the client-side part (data caching and rehashing) is done largely synchronously on the same worker that serves all other data queries. Hence, attempting to load a custom confluence view, then navigating to the catalog view yields an empty table. Returning to the confluence view yields no graphs (because they also query via the worker for their data).

Give a better name to custom views

When custom views are created for drilling down in metrics graphs, their tabs are labeled "#VIEW0", "#VIEW1", etc. in the UI. Almost anything would be better than this.

Write markdown docs on data collection and deployment

Some (all?) of this can go in README.md

Safari version < 7.1 API Velocity is suspicious

Safari 5.1 and 6.0 (according to the data) ship < 500 APIs. That does not seem right.

Enable clicking on graph points to get a catalog page

Eg. in the "Safari API diff history" I'm surprised to see so many APIs listed as removed. I want to somehow see a list of what these APIs are.

Switch license/copyright to BSD/Chromium Authors

Same as done with webidl-diff here and here.

Add colour to points that are "shipped this browser"

Metrics views create a point on each line for shipping from any browser. When the browser shipped is the browser associated with the current line, make the point a different colour.

Use friendly Safari version numbers

I think the more useful version number for Safari is the Version/ one (eg. 10.0.03), not the Safari/ one (eg. 602.4.8).

Use ReadOnlyDAO for project's read-only DAOs

Standard DAO decorator introduced in https://github.com/foam-framework/foam2/pull/318/files#diff-2981b40014961d72e93be57efd49e5e2R20.

Implausible removals claimed between Chrome 53.0.2785.101 and 53.0.2785.113 on Windows 10.0

Can be seen at http://web-confluence.appspot.com/#!/catalog?releaseKeys=%5B%22Chrome_53.0.2785.113_Windows_10.0%22,%22Chrome_53.0.2785.101_Windows_10.0%22%5D&releaseOptions=%7B%22Chrome_53.0.2785.113_Windows_10.0%22:true,%22Chrome_53.0.2785.101_Windows_10.0%22:false%7D&numAvailable=null&currentPage=0&gap=5&itemPerPage=15&searchKey=%22%22

It claims that lots of things on CSSStyleDeclaration have changed.

Better use space in catalog view

The catalog view currently has a few issues:

• Small fixed width
• Check/x tiles are huge; let's tighten them up
• Browser headings are pretty verbose/wide; tighten them up too

Add deep linking

All app states should be linkable. This means that we would have different URLs (possibly just a different anchor/fragment) for:

• API catalog (and custom views spawned by analysis)
  • Page number
  • Expand all state
  • Search query
  • Currently selected browsers
• Analysis
  • Currently selected view
  • State for custom views

This may prove to be too much state. Perhaps we want the ability to generate a link for API-catalog-style views, and current app state on the current machine could go in IndexedDB or similar so that custom views aren't lost on a single client.

Test and implement "additional metrics" from design review

These metrics included:

• Overall API confluence: what fraction of the total APIs are present in at least 3 or 4 engines at any given time?
• First to implement: How many APIs did this browser ship that were later shipped by at least 2 or 3 other browsers?

Improve "add browser" UI

The list is getting a little unwieldy. Maybe do something multi-level - first pick one of the 5 browsers, then the OS, then the version (sorted with most recent first)?

Explain "elastic band"

Maybe just link to this post?

Multiple vendor graph controls

It would be nice if the user could add/select vendors to include in time series graphs. This would eliminate the need for separate graphs to show "all browsers" and "each browser individually".

Consolidate metrics

Metrics each contain roughly the same data with different semantics/labels. Also, the set ops refactor exposed similar patterns in how metrics are computed. The metric computer classes should probably be consolidated into:

• A single metric data class
• A metric computation interface
• Three metric computation implementations

Firefox: CSS2Properties currently copied into CSSStyleDeclaration

Firefox's CSS2Properties is a non-standard interface that extends a very small CSSStyleDeclaration. The union of these interfaces is what is meant by a "standard CSSStyleDeclaration" implementation in Firefox. This is a known issue, but is not high priority for the Firefox team.

UI polish: Spacing around main "Web API Confluence" heading

Drop IE data from collections loaded by server

Metrics are computed assuming 4 modern engines (Edge, Blink, WebKit, Gecko). IE (Trident) data should not be included in analysis.

Move to karma/jasmine + jasmine/node combined tests

Similar to https://github.com/GoogleChrome/webidl-diff test setup. This should fix our misleading high coverage metrics (we're missing tests for all the code in #33 but it claims to have no impact on coverage). This will be necessary before we can fix #35.

Add definition and rationale to confluence metrics pages

This info provides important context to anyone inspecting the graphs.

Simplify metric compute() implementations with custom mLangs and/or sinks

Some metric compute() implementations contain a lot of Promise synchronization and exploiting side-effects in accumulating (or reducing) objects and arrays. They should be cleaned up to concisely express the queries they intend, which will probably require custom mLangs and/or sinks, such as DISTINCT.

UI polish: "X" button to close browser selections should stick to top-right corner of cell

StyleSheet#disabled not found via the web-apis.html tool

In https://bugs.chromium.org/p/chromium/issues/detail?id=695984 I was using the tool to see if the disabled attributes were supported in all the engines. I got my answer for the 3 element interfaces, but couldn't find the StyleSheet interface at all. Collection bug or display bug?

Some metrics graphs overlap with their titles

UI polish: Some special character is showing up next to (C) copyright symbol in footer

Start with default browser set

Often you just want to see the latest Chrome, Firefox, Edge and Safari data. Perhaps pre-populate the browser list with that (all Windows, except Safari)?

Add mobile browsers?

The mobile browsers should have mostly the same surface area as their desktop counterparts, but there are some interesting differences. We talked about adding Chrome Android, right? What are the trade offs here?

Refactor chart-based models to share setup code

We have two flavours of time series charts:

• Plain time series (but multiple may appear on one graph)
• "Tri-time-series": Used for number of APIs + add/remove delta on a single series

Right now, there is no common base class for these, even though a lot of setup work is the same.

UI polish: Main view should be full width

The tool is for data analysis (not reading prose) on desktop/laptop machines so there's no reason to limit the main view's width.

CSSStyleDeclaration properties missing on Firefox and Safari

We're hoping that we can get the list of supported CSS properties showing up consistently in CSSStyleDeclaration. In Firefox the properties are on CSS2Properties.prototype instead of CSSStyleDeclaration.prototype. I think we had some plan for merging CSS2Properties "up" into CSSStyleDeclaration, right? I don't see CSS2Properties members showing up in the catalog for Firefox.

In Safari, the properties appear to sort of show up as own-properties on instances of CSSStyleDeclaration:

Object.getOwnPropertyNames(document.body.style) -> contains "borderStyle"
Object.getOwnPropertyDescriptor(document.body.style,'borderStyle') -> Object
Object.hasOwnProperty(document.body.style, 'borderStyle') -> false, WTF?

Merge charts

Now that the model for chart data points has been unified, the chart code for various metrics (except API Velocity) is virtually identical. We should just have one chart.

Opening custom views from metrics graphs is broken

The table loads the correct header, but the data don't show up. Perhaps it's just taking forever to fetch from Datastore, but it seems like something is actually failing.

Add UI control to remove custom views

It should be possible to remove custom views that were created by drilling down into analysis graphs.

Test and implement: Pushing data to Cloud Datastore

We have tests that use the emulator to ensure that FOAM models are behaving as expected, but there is no code checked in that, when run in a Compute/App Engine context, will commit data to Cloud Datastore (yet).