Testing Horde.UniformQuorumDistribution not behaving as expected. about horde

[Ivor]

Ok thanks!

I think you have given me a good start. I'll get back to you ASAP.

Thanks for all the help!

[derekkraan]

Hi Ivor,

Thanks for the bug report.

In a previous version of Horde, members weren't removed from the members array, but marked as :dead. Later this changed, with the (in hindsight) predictable result that the UniformQuorumDistribution no longer works as intended.

So your observations are correct and this is not expected behaviour.

This is actually a trickier question than it seems at first blush. I'm going to let it simmer in the back of my brain for a bit, maybe I'll come up with something good.

In the meantime, a StaticQuorumDistribution would solve your problem (ie, that compares the number of nodes in the cluster with a fixed integer, say, 3 [in which case 2 nodes would be mandatory]). I'd be happy to give pointers if you wanted to take a crack at that.

[derekkraan]

I've created #47 for implementation of StaticQuorumDistribution. #46 is also possibly interesting for you.

[Ivor]

Thanks Derek!

I am definitely interested in taking a stab at the StaticQuorumDistribution. It makes sense to have a behaviour as you mention in #47

Could you give me some pointers as to how one would go about this.
Could one simply add a similar list to "members" from which the :dead members are not removed and used that list in the static version?

For production, we'll be running this with AWS and will read a list of active nodes from there. I am hoping we can create a solution that can handle the quorum based on the nodes returned by the EC2 cluster, so whatever EC2 spins up for us should be taken as the members at that stage. I think we'll be able to find a way to read that somehow.

Thanks in advance!

[derekkraan]

The part that makes the UniformQuorumDistribution tricky is that you have to know about the dead nodes in order to have an idea of how many nodes there should be. With StaticQuorumDistribution you have a static number, so you don't need to know about the dead nodes, just the alive ones.

If you have a secondary source of information about cluster membership (ie, the AWS API), then you could create an AWSUniformQuorumDistribution that compares the alive nodes in the cluster to the nodes that AWS says are present. This could also be a nice solution.

Perhaps you could start with #46, if you have solved that one then doing #47 should be a little clearer. A distribution strategy basically must have two functions: choose_node/2 and has_quorum?/1. You can fill this in as you wish with the information available to you (whether this is a static number of the number of nodes in the cluster, or a list of nodes from AWS).

Does that help? Happy to answer any additional questions you might have.

[Ivor]

Quickly looking at choose_node/2 and has_quorum?/1 I see that choose_node is used to proxy a process to a new node and has_quorum? to determine whether the node is part of the main partition or whether its processes should be terminated.

You are currently passing in the members which will make it slightly complicated to work with alternative ways of determining quorum. I wonder if it would make sense to pass in state instead so that each quorum can do with state what it wants to determine these answers. The existing distribution strategies could read members and take it from there, while static could read a number from the state. We could then also configure a cluster_membership_function [MODULE, :action, args] option that something like an AWSUniformDistribution strategy could call to determine cluster membership.

Does changing those two functions to take the state and get the members from the state, instead of passing in the members, sound like a reasonable change?

[derekkraan]

The tricky part with this approach (seems to me) is that you would have to have a way to get your own data into the state of SupervisorImpl.

Another option is to store that information in another GenServer (or ETS table) which you could refer to from the distribution strategy.

For the AWS strategy this would probably be an ETS table for optimal performance. For UniformStaticDistribution I would recommend making it configurable via config :horde, UniformStaticDistribution, quorum: 3

[mgwidmann]

Read this to get a better understanding of what a UniformQuorumDistribution actually does so thank you, the example test really helped. Would like to see once this is in better working order a bit more information in the documentation about why this would be used. I too would like to use it in an EC2 environment within an autoscaling group. It doesn't make much sense to do a static integer gathered via the AWS API since it can scale up or down at any time so you'd want to have it updated automatically when the a node joined/left the cluster.

[derekkraan]

@mgwidmann, if you can get a list of nodes from EC2 in your autoscaling group, then you can fairly easily build your own distribution strategy that dynamically adjusts quorum based on the number of nodes in your group. This isn't released yet, so you'll have to either wait a couple of weeks or use master.

[derekkraan]

The way we do membership has changed leading up to 0.5.0. Horde will now never add or remove a member from the cluster itself. This means that you should manage cluster membership yourself (with something like libcluster) and it also means that the UniformQuorumDistribution should also work the way you are expecting.