rust-gcc / testing Goto Github PK

TestCase::new() and TestCase::Skip do not make a lot of sense when creating the two possible variants of a Test case. Furthermore, does it make sense to keep a builder pattern for non-skipped test cases? If the answer is yes, does it make sense to simply return Skip cases even when calling builder functions, effectively making the call a no-op? Should we panic!() to indicate that the user might be doing something unexpected or silently ignore that as we currently do?

cargo run -- ... takes forever while generating testsuite.yml. (30min on my laptop)
It is helpful to show how far it has progressed.

[info] generating test cases for `gccrs-parsing`... this might take a while

↓

[info] generating test cases for `gccrs-parsing`
54% done.

ftf's timeouts are in seconds, not minutes

This will avoid repetitions throughout multiple passes and allow us to have sensible common defaults for all existing and future passes such as the crate name to give to rustc

It would also be nice if this was simply a trait extension on Rust's std::process::Command

Since we now use a custom runner, there's no point not to do so

This was disabled due to some issues, maybe due to the runner, maybe due to how we were exporting said AST dump. We need to investigate the failure and re-enable it in the nightly run.

This will avoid us having to change the CI file everytime we want to add new passes to the project

Launching the testsuite using the readme instructions produces no test, the resulting testsuite.yml has the following content:

tests:

> cargo run -- \
	--gccrs './rust1' --rustc rustc \
	--gccrs-path gccrs/ --rust_path rust/ \
	--output-dir sources/ --yaml testsuite.yml \
	--pass gccrs-parsing

The executable should be named "crab1", and the rust_path argument is wrong - should be rust-path

We have to be careful with the way bottleboard fetches files from the github actions but that should be an issue

This test case Rust-GCC/gccrs#1297 is causing an infinite loop and could cause any automation to lock up.

Run the code and its dumped version against each other, checking for stdout and the exit code. On our testsuite is probably a good starting point

With #![rustc_private] you could use the RunCompiler API. Not sure if you want to do that though. I think just executing rustc with Command is good enough.

Originally posted by @flip1995 in #7 (comment)

Let's experiment and see if it makes a good difference in test generation time. Currently, generating all test suites takes around 7 minutes which is quite long but not "horrible"