Add integration testing for version sets that must pass before updating version index about fuelup

[mitchmindtree]

Just to be clear - is it ok if the tests are run against a fuel-core binary, or is it necessary to build fuel-core from source, or run it within docker just like how the Sway repo does it as well? Or was running from docker out of necessity?

Whatever's easiest I think! I don't use docker and haven't had any troubles when running the E2E tests locally.

[eightfilms]

Correct me if I'm wrong: the e2e tests in the sway repo here are enough to guarantee that the latest forc release works with whatever the version of fuel-core was being used in the tests (especially since there are tests that run on the config run_on_node), however it is not enough to guarantee that if a new fuel-core version was released without a new forc release, we don't know for sure that the new fuel-core is compatible - hence the need for this issue.

So the problem is again the decision of where these checks live like we discussed in #66, whether it should be a push-based check or a pull-based check.

If we opt for a push-based check, I imagine we could have fuel-core, on release, trigger a workflow_dispatch event within the sway_repo to run integration tests with the new fuel-core version. Then we'd need a way to communicate whether it passed or failed to fuelup. I'm not sure how useful this would be to the sway repo alone to know that there could be breaking changes on new fuel-core releases, if it is not useful at all then perhaps it isn't a good idea to do this just for fuelup.

The alternative is to use this repo to, again, poll for new releases, checkout the code and run them on the CI here(?). Again, it seems like both solutions aren't perfect but I think it'd be messier if we left it up to the sway or fuel-core repos. I think sticking to the maxim of ignorance of fuelup is a good way to go here?

[eightfilms]

It would be a shame to have some test here that could have been added to Sway and caught some incompatibility bug earlier in the release process.

Agreed, I think tests should be written in the Sway repo, and we just need to decide if we want to check compatibility here (by pulling code from both repos and running tests), or in the Sway repo.

[mitchmindtree]

Correct me if I'm wrong: the e2e tests in the sway repo here are enough to guarantee that the latest forc release works with whatever the version of fuel-core was being used in the tests (especially since there are tests that run on the config run_on_node), however it is not enough to guarantee that if a new fuel-core version was released without a new forc release, we don't know for sure that the new fuel-core is compatible - hence the need for this issue.

Yes exactly 👍

So the problem is again the decision of where these checks live like we discussed in #66, whether it should be a push-based check or a pull-based check.

I think it might make sense to pull for these, particularly as we only need to test upon detecting a new release from either fuel-core or forc. I'm imagining something along the lines of:

1. Cron checking for new releases
2. New release detected
3. Work out fuel-core and forc versions to test together
4. Fetch sway repo (probably another good case for using shell script 🥲)
5. Run the E2E tests with forc and fuel-core versions to test. Possibly use a regex to filter for the integration tests.
6. Upon success, update toolchain versions in the index.
7. Upon failure, leave toolchain versions as they are and record somewhere the version set that failed (this should be checked before running the tests, otherwise if there's a failure the cron-job will just keep running the same failed tests over and over).

^ @binggh what are your thoughts on something like this?

[eightfilms]

Sounds good! Unfortunate that we have to once again rely on scripting for this but given that it's a part of the CI and not really a part of the fuelup package itself it seems acceptable to me.

I'd probably implement it somewhere along the lines of the current index-versions script as well - checking for new releases on a 30 minute time interval.

Just to be clear - is it ok if the tests are run against a fuel-core binary, or is it necessary to build fuel-core from source, or run it within docker just like how the Sway repo does it as well? Or was running from docker out of necessity?