Comments (6)
A few things at play here:
-
The
TO_DATE
in the build.sh script usesdate -d "yesterday 13:00" '+%Y-%m-%d'
, so when we run on 2021-10-27 we are only passing a--to
of 2021-10-26 tocompat2021/main.js
-
main.js
then fetches exclusively (https://github.com/Ecosystem-Infra/wpt-results-analysis/blob/main/compat-2021/main.js#L95), so the last fetch URL for a run on 2021-10-27 will be: -
Even for one day more than that, the wpt.fyi API is giving me back Safari 14.1 still:
It's not until I change it to 27 to 28 that I get the Safari 15.0 run:
https://wpt.fyi/api/runs?&label=master&label=stable&product=chrome&product=firefox&product=safari&aligned=true&max-count=1&from=2021-10-27&to=2021-10-28Looking at the JSON, looks like the
time_start
for that 15.0 run was2021-10-27T01:21:50.549Z
So we will pick up that Safari 15.0 run tomorrow I think (the gh-pages action runs Mon,Wed,Fri), but it's a reasonable criticism to note that we have a 36-48 hour lag on picking up a given run even if we were running the action every day of the week.
Minimizing this delay would likely involve making build.sh pick up runs until 'today' (not 'yesterday') and then carefully aligning the action times so that the runs for that day have (probably) already happened and have already been ingested to https://github.com/ecosystem-Infra/wpt-results . Unfortunately that isn't work I have time for, but PRs are welcome :).
from wpt-results-analysis.
While I know @stephenmcgruer has now manually triggered an update (especially for WPT), and Compat 2021 now shows STP 135, various people have questioned both that and the Browser-Specific-Failure (BSF) graph not being updated more readily.
It's a shame that the action only runs a few times a week (and changing that to daily should be trivial!).
AIUI, the big challenge here is the wpt.fyi API for queries such as the above (e.g., https://wpt.fyi/api/runs?&label=master&label=stable&product=chrome&product=firefox&product=safari&aligned=true&max-count=1&from=2021-10-27&to=2021-10-28) returns the latest test runs where the test_start
time is on 2021-10-27, and inevitably those test runs may only complete at some point into 2021-10-28 (or potentially even further into the future; I don't think we strictly have an upper bound on test execution time!), hence we don't actually know what the latest test run for a given date is until all test runs have completed.
Minimizing this delay would likely involve making build.sh pick up runs until 'today' (not 'yesterday') and then carefully aligning the action times so that the runs for that day have (probably) already happened and have already been ingested to https://github.com/ecosystem-Infra/wpt-results .
build.sh
does this itself, as part of the BSF update, no? One could imagine updating multiple times per day (after all, we're using little GitHub Actions capacity here!), and just overwriting the data for the last two or three days in case there's been any change about what runs are being used there.
Unfortunately that isn't work I have time for, but PRs are welcome :).
Unfortunately, that's difficult as long as copyright is held by the authors and/or their employers and this isn't licensed under an open-source license (c.f. #58). 🙃
from wpt-results-analysis.
I'm noticing this issue today again because STP 136 was just released, but triggering all the updates manually still doesn't update the data due to the TO_DATE
limit.
@stephenmcgruer do you recall why this was included in 2fe8143 in the first place? browser-specific-failures.js already uses aligned runs... Was it something like runs appearing out of order and wanting to leave a margin for that noise to settle?
from wpt-results-analysis.
@stephenmcgruer do you recall why this was included in 2fe8143 in the first place? browser-specific-failures.js already uses aligned runs... Was it something like runs appearing out of order and wanting to leave a margin for that noise to settle?
From what I recall, it was a conservative guard against failure due to no runs detected, especially due to the caching logic (which is likely irrelevant to the GitHub action but is relevant locally).
I don't think there's anything fundamentally impossible about bringing the cut-off date closer, but we previously weren't interested in cutting-edge run results for compat2021 and so decided better safe than sorry ;)
from wpt-results-analysis.
Thanks @stephenmcgruer! Sounds like there are issues we might run into, but we can give it a try to shorten the pipeline and see what happens.
from wpt-results-analysis.
Closing in favor of web-platform-tests/results-analysis#65 ahead of archiving this repo.
from wpt-results-analysis.
Related Issues (17)
- change Action cron days? HOT 1
- [compat2021] Views should be linkable
- [compat2021] Implement summary numbers
- [compat2021] Make browser version changes visible HOT 13
- [compat2021] Make the summary score labels more specific HOT 2
- [compat2021] put it on wpt.fyi HOT 4
- [compat2021] Unify the y-axis between graphs HOT 2
- [compat2021] The Stable/Nightly option should rather be a toggle HOT 1
- [compat2021] Rename page title to Compat 2021 / similar HOT 2
- [compat2021] Enabling easy iframing of the dashboard HOT 1
- [compat2021] Show summary scores over time as default graph view HOT 2
- Add a LICENSE file HOT 1
- Migrate to wpt org? HOT 2
- Incorrect scoring of BSF
- BSF walkTrees not called for some directories? HOT 1
- Regenerate all BSF data HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wpt-results-analysis.