(integration) tests are flaky about surrealdb-migrations

[turbohz]

I see, I never considered that critical that the tests ran fast,

Ideally yes. But then it will take a lot of time to run tests. Apart from the "branches" tests, tests are generally using the same surrealdb instance but in separate ns/db so there should never be any conflict.

I fixed an issue where the integration test was using a db named test, instead of a random named one.

Could be just that one, it might well be, yeah.

I never experienced failed test or surrealdb process crash. Are you sure nothing else come in the way? Enough CPU? Enough RAM? I suppose you were not able to identify the cause.

I work on a beefy desktop PC. Since the two process used are put in the background, I could not catch what happened to them, I just got errors when the test tried to connect.

Could be an issue with my setup, or bugs in surrealdb related with the strict mode, which is what I was testing.

I'll close the issue for now, and reopen if I find out anything more concrete.

[turbohz]

This means, before-integration-tests.sh should not be required to be manually ran.

[turbohz]

I also noticed that sometimes the surrealdb processes crash, or stop for some reason.

[Odonno]

The 2 main goals with tests are:

• Tests should run in less than 10s, so parallelizing a maximum of tests is a priority. The ones that can't be run in parallel are the ones on branch feature. Might be a good idea to split in 2 test libraries. One in parallel and one in serial (one library per limited scope like branches). I can see that.
• Tests should run in CI. So having to launch a limited number of instances of surrealdb is good. And to mimic the workflow of the CI in your local computer, we need the same workflow.

While implementing #56 I noticed that tests will pass sometimes, and then they will not.

Never noticed that. I suppose you have no idea to determine the cause? I think it can happen to have ns/db conflict between 2 tests but that should be like one in a zillion.

Ideally each integration test should run in a fresh, uninitialized test database.

Ideally yes. But then it will take a lot of time to run tests. Apart from the "branches" tests, tests are generally using the same surrealdb instance but in separate ns/db so there should never be any conflict. We still need a right cleanup for "branches" test.

From my experience with other languages, they often have hooks to run stuff before/after a test suite or individual tests. I assume something similar can be done in Rust. This means, before-integration-tests.sh should not be required to be manually ran.

Kind of like setupTest in jest or a beforeAll/afterAll in most languages. Yes, I can definitely see that. Not sure if it is possible at the moment. I mean, I did not found a solution for this.

I also noticed that sometimes the surrealdb processes crash, or stop for some reason.

I never experienced failed test or surrealdb process crash. Are you sure nothing else come in the way? Enough CPU? Enough RAM? I suppose you were not able to identify the cause.

[Odonno]

On another note, I am also interested in a migration to nextest. But that should be in its own PR.

[Odonno]

Thank you for your input. It emerges with good idea like splitting test library in 2. Will create an issue for this.