Dead nodes are not removed from horde registry about horde

[derekkraan]

Thanks for your help @farhadi. I'll release an rc so that people can test a bit before I publish the final release.

[derekkraan]

https://hex.pm/packages/horde/0.4.0-rc.1

[derekkraan]

Hi Ali,

Could you try with the current Horde master branch and let me know if you're still seeing this behaviour? I have been meaning to roll a new release, as some issues such as this should be fixed.

Derek

[farhadi]

I tested with current master branch using included hello_world example.

First I start two nodes and everything is fine:

iex([email protected])1> Horde.Cluster.members(HelloWorld.HelloRegistry) 
{:ok,
 %{
   "MbAqzbUD9KNrbLIxEY+faA==" => %{
     keys_pid: #PID<0.206.0>,
     members_pid: #PID<0.198.0>,
     own_pid: #PID<0.210.0>,
     registry_pid: #PID<0.202.0>
   },
   "cg+xMC3lJCOUYmgcTI27Pw==" => %{
     keys_pid: #PID<20632.204.0>,
     members_pid: #PID<20632.196.0>,
     own_pid: #PID<20632.208.0>,
     registry_pid: #PID<20632.200.0>
   }
 }}

iex([email protected])1> Horde.Cluster.members(HelloWorld.HelloRegistry) 
{:ok,
 %{
   "MbAqzbUD9KNrbLIxEY+faA==" => %{
     keys_pid: #PID<17991.206.0>,
     members_pid: #PID<17991.198.0>,
     own_pid: #PID<17991.210.0>,
     registry_pid: #PID<17991.202.0>
   },
   "cg+xMC3lJCOUYmgcTI27Pw==" => %{
     keys_pid: #PID<0.204.0>,
     members_pid: #PID<0.196.0>,
     own_pid: #PID<0.208.0>,
     registry_pid: #PID<0.200.0>
   }
 }}

But after restarting one of the nodes, I receive this Discarding message {delta, ... error and members returns 3 items in both nodes:

iex([email protected])2> Horde.Cluster.members(HelloWorld.HelloRegistry) 
{:ok,
 %{
   "B1wfSzsxs3Py18QSzNmOfQ==" => %{
     keys_pid: #PID<20632.204.0>,
     members_pid: #PID<20632.196.0>,
     own_pid: #PID<20632.208.0>,
     registry_pid: #PID<20632.200.0>
   },
   "MbAqzbUD9KNrbLIxEY+faA==" => %{
     keys_pid: #PID<0.206.0>,
     members_pid: #PID<0.198.0>,
     own_pid: #PID<0.210.0>,
     registry_pid: #PID<0.202.0>
   },
   "cg+xMC3lJCOUYmgcTI27Pw==" => %{
     keys_pid: #PID<20632.204.0>,
     members_pid: #PID<20632.196.0>,
     own_pid: #PID<20632.208.0>,
     registry_pid: #PID<20632.200.0> 
   }
 }}

iex([email protected])2> Horde.Cluster.members(HelloWorld.HelloRegistry) 
{:ok,
 %{
   "B1wfSzsxs3Py18QSzNmOfQ==" => %{
     keys_pid: #PID<0.204.0>,
     members_pid: #PID<0.196.0>,
     own_pid: #PID<0.208.0>,
     registry_pid: #PID<0.200.0>
   },
   "MbAqzbUD9KNrbLIxEY+faA==" => %{
     keys_pid: #PID<17991.206.0>,
     members_pid: #PID<17991.198.0>,
     own_pid: #PID<17991.210.0>,
     registry_pid: #PID<17991.202.0>
   },
   "cg+xMC3lJCOUYmgcTI27Pw==" => %{
     keys_pid: #PID<0.204.0>,
     members_pid: #PID<0.196.0>,
     own_pid: #PID<0.208.0>,
     registry_pid: #PID<0.200.0>
   }
 }}

And more restarts leads to more errors and more items returned by Horde.Cluster.members

[derekkraan]

Hi Ali,

I've done some testing, PR #57 should fix the issue. Can you please test and report back whether it works as expected?

Cheers
Derek

[farhadi]

Thanks for the quick update.
I tested with your PR and found a few issues which I tried to fix here: https://github.com/farhadi/horde/commit/dad7a2dbfae5e639e6c55ea0db0e29c9519c2288
can you take a look at my changes.

[derekkraan]

Can you describe the issue that is fixed by this commit?

[farhadi]

• mark_dead was defined between two clauses of handle_info
• state.members was not defined (I used DeltaCrdt.read(state.members_pid, 30_000)) to retrieve members
• I think mark_alive(state) was not needed here so I removed it
• I also made mark_dead a private function

[derekkraan]

Ah ok. I agree with all the changes except this one:

I think mark_alive(state) was not needed here so I removed it

This is needed because we also need to cover the scenario that there is a netsplit, and the node is not actually dead.

Can you restore mark_alive and open a PR on the branch remove_dead_members?

[farhadi]

Here it is: #58