puppeteer / ispuppeteerwebdriverbidiready

Is Puppeteer ready for WebDriver BiDi yet?

https://puppeteer.github.io/ispuppeteerwebdriverbidiready/

ispuppeteerwebdriverbidiready's Issues

Build & deploy job fails

https://github.com/whimboo/ispuppeteerfirefoxready/actions/runs/148572467

The ssh-private-key argument is empty. Maybe the secret has not been configured, or you are using a wrong secret name in your workflow file.

Count multiple expectation status as failure

Right now the graph doesn't display those test results that have multiple statuses in the expected json file. This actually leaves a disconnect behind, which makes it hard to explain.

@mathiasbynens could we please count multiple status entries as failures instead?

Thanks!

Display historical data of test expectations as graph

To improve the page to display current numbers and not those from the local copy of the expected data, we could do the following:

1. Grab the JSON history data of the file

2. For each entry:
   2.1. Retrieve the push date via entry[index][pushdate][0]
   2.2. Extract the revision via entry[index][node] and request the raw content of the file

3. The number of PASS, FAIL, SKIP, and total tests could be then displayed in a graph like:

I assume that requesting the data via fetch will work cross-domain? If that is the case it shouldn't be that hard to get a nice status history displayed.

Any suggestions for such a library we could use for the graphs?

Link to the Firefox delta graphs from the front page (instead of directly to the json file)

With the changes from #25 we got two new graphs added:

• https://puppeteer.github.io/ispuppeteerwebdriverbidiready/?firefox-delta
• https://puppeteer.github.io/ispuppeteerwebdriverbidiready/?firefox-delta-all

Sadly both are not accessible from the front page. My suggestion would be to add a link to the delta page from the front page via the Firefox (WebDriver BiDi) has XXX failing or skipped tests. The link to the JSON file we could then show on that particular delta page only.

@OrKoN @juliandescottes

Extend timeline of the graph to a reasonable amount of days

Right now the graph only shows the most recent 8 results and makes it impossible to actually overview the improvements that we have made over the last milestone (3 months duration). It would be great to extend the timeline to at least some (4?) weeks if possible.

We would appreciate if that could be done. Maybe it requires a different type of graph, which just displays two lines?

CC @OrKoN

Annotate each value

Ideally the individual values in the chart can be viewed by hovering over each data point, similar to how it's done here: https://mathiasbynens.be/demo/jquery-size

Dashboard does no longer update since Nov 2021

As it looks like the dashboard does no longer update since around mid November 2021:
https://puppeteer.github.io/ispuppeteerfirefoxready/

@mathiasbynens is there some service not running? Or does it need a manual trigger?

It would be great if you could have a look. Thanks a lot.

Use Puppeteer's own test expectations file

Since the Puppeteer 17.1.3 release Puppeteer has a new expectation file:
https://github.com/puppeteer/puppeteer/blob/main/test/TestExpectations.json

Given that we consider dropping our own expectation file, which is currently used to build the data for ispuppeteerfirefoxready, the code in this repository should probably use the Puppeteer's own file.

@mathiasbynens what do you think? Should we change it, or maybe abandon this page? After talking to @OrKoN he had the idea with ispuppeteerbidiready instead.

Dashboard is not updated anymore

This is happening because I moved the puppeteer-expected.json file from /remote to /remote/tests, and as such fetching the data fails now:

https://github.com/puppeteer/ispuppeteerfirefoxready/blob/6d0eae213db0ba4753754ac0ee51b92039529d72/preprocess-data.js#L39

We would have to fetch the data from both URLs.

I'll prepare a PR.

Not correct reports for `skipped` test

I was looking around what test are still need for this and it seems that there is one more thing to consider when getting the correct data.
Some tests are disabled globally with it.only and not from TestExpecations.json.
The fix should be straight forward:
Get the Chrome CDP skipped test and remove them from failing and skipped for BiDi, not we should not remove them from passing as we already have test that pass in BiDi but fail in CDP.
I believe if the test is skipped in all 3 places we should remove its count from the dashboard as well.