ecosystem-infra / wpt-results-analysis Goto Github PK

It is currently difficult to tell what change caused an increase (or decrease!) in pass-rate of a given browser at a given point in time. This could be due to new tests being added, or a change in the browser itself, or ...

To help with this, we should make the browser version visible in some way - either on the individual data points or via annotations.

Currently the Action runs on Monday/Thursday at midday UTC, which is slightly awkward with STP releases going out on a Thursday during the PT workday and hence not showing up till the following Monday. I don't have a super concrete suggestion here, but one option would be to move it to Tuesday/Friday.

@gsnedders pointed out on https://app.element.io/#/room/#wpt:matrix.org that this repo doesn't have a license. That's an accident of how the repo came into existence, and should be fixed.

It should be possible to deep-link to any of the views in https://ecosystem-infra.github.io/wpt-results-analysis/compat, e.g. something like https://ecosystem-infra.github.io/wpt-results-analysis/compat?focus=position-sticky&label=stable should take you directly to that graph.

Modifying the UI should also cause the URL to update (via history.pushState).

After #80 + #82 + #83 we should regenerate all of the historical BSF data, since it will have slightly changed. As it is, BSF data is only regenerated starting from the latest date in the CSV files:

I feel that if you come in now, there's a paragraph of text and you need to explicitly choose a Focus area. I think it would be better to have one area preselected and get people excited about, and aware of, the presence of graphs.

Currently, changing the graph always keeps the same max value for the y-axis (100%), but the min will change depending on the range in the datatable. This means that you cannot visually compare across graphs flicking between features; you need to reorient yourself with the new y-axis range.

We should fix the min y-axis instead. A quick and cheap way would just be to manually look at the data and pick a value say 10% less than the global minimum, but the 'correct' way would be to fetch all the data up-front and find the global minimum properly.

From chrishtr:

3. Label the score at the top as "Chrome Canary" "Firefox Nightly" "Safari Developer Preview" And move the stable checkbox up to the top above individual features

We can likely go a little further, and include the major version in the summary name. We could also add a single sentence lower down to explain that some features are enabled in the 'experimental' releases.

walkTrees doesn't seem to ever be called for path=/content-security-policy/gen/top.meta/script-src-self; I wonder if this is because it is an identical tree object as /content-security-policy/gen/top.http-rp/script-src-self?

    it('should handle tests that arent in all (three) browsers', () => {
      const expectedBrowsers = new Set(['chrome', 'firefox', 'safari']);

      let chromeTree = new TreeBuilder()
          .addTest('TestA', 'OK')
          .addSubtest('TestA', 'Test (upper)', 'PASS')
          .build();

      let firefoxTree = new TreeBuilder()
          .addTest('TestA', 'OK')
          .addSubtest('TestA', 'test (lower)', 'PASS')
          .build();

      let safariTree = new TreeBuilder()
          .addTest('TestA', 'OK')
          .addSubtest('TestA', 'Test (upper)', 'PASS')
          .addSubtest('TestA', 'test (lower)', 'PASS')
          .build();
      
      let runs = [
          { browser_name: 'chrome', tree: chromeTree },
          { browser_name: 'firefox', tree: firefoxTree },
          { browser_name: 'safari', tree: safariTree },
      ];

      let scores = browserSpecific.scoreBrowserSpecificFailures(runs, expectedBrowsers);
      assert.deepEqual(scores, new Map([['chrome', 0.5], ['firefox', 0.5], ['safari', 0.0]]));
    });

This currently fails with a result of:

      assert.deepEqual(scores, new Map([['chrome', 0.3333333333333333], ['firefox', 0], ['safari', 0]]));

This also isn't hypothetical: this affects the scoring of clear-site-data/storage.https.html among many other tests.

There will be places to iframe the dashboard in, like web.dev and MDN Insights. However, for those places, having complete bug lists would be overwhelming. While that can be addressed with iframe height and no scrolling, it would be nice – if possible – to just pass in parameters in the querystring.

E.g. wpt.fyi/compat2021?nobugs=1, wpt.fyi/compat2021?notitle=1

To give a quick overview of each browser, we should present summary numbers at the top of the page (analogous to Lighthouse's summaries). These should be based on the most recent run for each focus area.

To normalize for number of tests, we will do something like give each area 20 points of the total number, and contribute passing_percent * 20 from them; thus giving a number between 0-100.

Bonus points if the number changes color as it approaches 100. Extra bonus points if there's fancy (performant) animations and swish ;).

After discussing options with @chrishtr we think that hosting this on wpt.fyi/compat2021 would make the most sense.

Someone should probably just write an RFC for this, but it would be good to have it there.

The "Update gh-pages" action last ran on 27 Oct 2021, 13:27 BST, and produced 4722ba4

However, the only seems to have picked up runs to 2021-10-25, and not any of the more recent runs on wpt.fyi, especially for stable where we had our first Safari 15 run in that intervening period: https://wpt.fyi/runs?label=master&label=stable&max-count=100&from=2021-10-21T00%3A00&product=chrome&product=firefox&product=safari&aligned

We should rename it to "Compat 2021", "#Compat2021" or similar for consistency between messages and channels.

The Stable/Nightly option should rather be a toggle, I think, so it's clear what the current one is and how to switch. People I've shown it to misses the Stable releases checkbox text.