As detailed by @elitan:
- checkout new branch
- nhost CLI starts applying changes
- before nhost CLI finishes applying changes, do
git pull
and pull new migrations.
- nhost CLI does not pick up new changes because it was busy applying changes from step 1 but there are new migrations from step 3.
Originally posted by @elitan in #9 (comment)
This is a fundamental race condition. I spent the entire weekend 😪 trying to implement a reasonable solution. Reverted all my changes. Here's why:
- Hasura CLI needs to read the files it wants to apply with
hasura migrate/metadata apply
. Since, the user has done a git pull
, those files have already been changed/removed.
- This is not a race condition that can be solved with sophistication from our end.
Potential solutions I implemented:
Sol 1. Build real-time activation channels for every service inside our code. When Hasura is performing migrations/metadata, no other resource/function should be allowed to trigger another set of migrations. This basically locks the environment. The post-pull trigger will have to wait for Hasura to not be occupied, i.e. be active.
Logical problem with this solution: The environment is technically active, because while performing health-checks, Hasura isn't performing migrations/metadata anyway. But auth and storage might be. Our post-pull trigger ends up launching migrations before Hasura even becomes active.
Sol 2. Used a queue channel which stores the number of times we want migrations to be run. Heck, just imagine a counter. By default we only run migrations once in our dev environment for now. On git checkout
, the queue buffer is incremented by one, and migrations are performed again.
Logical problem with this solution: It doesn't matter how many number of times we tell the environment to run migrations, because by the time the CLI is finished performing even a single set of migrations+metadata, the local files have been changed by git. And at the end of the day, Hasura CLI needs access to those migration files inside nhost/migrations
to apply them.
So, the Hasura CLI just goes on and applies the same migrations X number of times, where X is the queue buffer.
This is the same problem I mentioned over here: #9 (comment)
Sol 3. Put mutually exclusive locking on migrations/metadata operations. Basically, if the migrations are currently being performed, then lock the Hasura service inside our code, and don't let any other resource trigger migrations concurrently, avoiding conflicts, and maintaining the sequence of migrations.
Same problem with this solution, as before: By the time the loop increments, git has already changed local files. Hasura CLI can't apply migrations if they don't exist inside nhost/migrations
now.
Sol 4. [This is also what I'd pushed in v1.0-internal-17
] - Don't perform migrations on the immediate git pull
. Let the environment fail with an error and the user must re-start the environment to apply the new migrations.
In short: User must wait for environment to restart after git checkout
, before pulling/merging/fetching their code.
Support git checkout
, followed by a 2 minute break, followed by git pull
. But DO NOT support git checkout {{branch}} & git pull
together.
Potential hazards: None! Most probably, the immediate git pull
will be over in the middle of the health checks of the refreshed environment (as happened in ~50 times I tested this). Since Hasura isn't running at that time, no migrations will be applied and CLI will exit with a fatal error.
This is the safest solution in my opinion. Not just in terms of implementation from our end, but also for the user. User can also do nhost purge --data
when in doubt, and rebuild the environment with new migrations.
I think I've spent far too much time on solving this race condition, than I should have. Almost the entire weekend.
P.S. - We must mention this race condition, somewhere in the documentation for the CLI, if we write one. Perhaps, somewhere in the "advanced features" section. This can some up again in the future. This is why I documented this in a separate issue.