Comments (10)
Ok thanks!
I think you have given me a good start. I'll get back to you ASAP.
Thanks for all the help!
from horde.
Hi Ivor,
Thanks for the bug report.
In a previous version of Horde, members weren't removed from the members array, but marked as :dead
. Later this changed, with the (in hindsight) predictable result that the UniformQuorumDistribution
no longer works as intended.
So your observations are correct and this is not expected behaviour.
This is actually a trickier question than it seems at first blush. I'm going to let it simmer in the back of my brain for a bit, maybe I'll come up with something good.
In the meantime, a StaticQuorumDistribution
would solve your problem (ie, that compares the number of nodes in the cluster with a fixed integer, say, 3 [in which case 2 nodes would be mandatory]). I'd be happy to give pointers if you wanted to take a crack at that.
from horde.
I've created #47 for implementation of StaticQuorumDistribution
. #46 is also possibly interesting for you.
from horde.
Thanks Derek!
I am definitely interested in taking a stab at the StaticQuorumDistribution. It makes sense to have a behaviour as you mention in #47
Could you give me some pointers as to how one would go about this.
Could one simply add a similar list to "members" from which the :dead
members are not removed and used that list in the static version?
For production, we'll be running this with AWS and will read a list of active nodes from there. I am hoping we can create a solution that can handle the quorum based on the nodes returned by the EC2 cluster, so whatever EC2 spins up for us should be taken as the members at that stage. I think we'll be able to find a way to read that somehow.
Thanks in advance!
from horde.
The part that makes the UniformQuorumDistribution
tricky is that you have to know about the dead nodes in order to have an idea of how many nodes there should be. With StaticQuorumDistribution
you have a static number, so you don't need to know about the dead nodes, just the alive ones.
If you have a secondary source of information about cluster membership (ie, the AWS API), then you could create an AWSUniformQuorumDistribution
that compares the alive nodes in the cluster to the nodes that AWS says are present. This could also be a nice solution.
Perhaps you could start with #46, if you have solved that one then doing #47 should be a little clearer. A distribution strategy basically must have two functions: choose_node/2
and has_quorum?/1
. You can fill this in as you wish with the information available to you (whether this is a static number of the number of nodes in the cluster, or a list of nodes from AWS).
Does that help? Happy to answer any additional questions you might have.
from horde.
Quickly looking at choose_node/2
and has_quorum?/1
I see that choose_node
is used to proxy a process to a new node and has_quorum?
to determine whether the node is part of the main partition or whether its processes should be terminated.
You are currently passing in the members which will make it slightly complicated to work with alternative ways of determining quorum. I wonder if it would make sense to pass in state
instead so that each quorum can do with state what it wants to determine these answers. The existing distribution strategies could read members and take it from there, while static could read a number from the state. We could then also configure a cluster_membership_function
[MODULE, :action, args]
option that something like an AWSUniformDistribution
strategy could call to determine cluster membership.
Does changing those two functions to take the state and get the members from the state, instead of passing in the members, sound like a reasonable change?
from horde.
The tricky part with this approach (seems to me) is that you would have to have a way to get your own data into the state of SupervisorImpl
.
Another option is to store that information in another GenServer (or ETS table) which you could refer to from the distribution strategy.
For the AWS strategy this would probably be an ETS table for optimal performance. For UniformStaticDistribution I would recommend making it configurable via config :horde, UniformStaticDistribution, quorum: 3
from horde.
Read this to get a better understanding of what a UniformQuorumDistribution actually does so thank you, the example test really helped. Would like to see once this is in better working order a bit more information in the documentation about why this would be used. I too would like to use it in an EC2 environment within an autoscaling group. It doesn't make much sense to do a static integer gathered via the AWS API since it can scale up or down at any time so you'd want to have it updated automatically when the a node joined/left the cluster.
from horde.
@mgwidmann, if you can get a list of nodes from EC2 in your autoscaling group, then you can fairly easily build your own distribution strategy that dynamically adjusts quorum based on the number of nodes in your group. This isn't released yet, so you'll have to either wait a couple of weeks or use master
.
from horde.
The way we do membership has changed leading up to 0.5.0. Horde will now never add or remove a member from the cluster itself. This means that you should manage cluster membership yourself (with something like libcluster) and it also means that the UniformQuorumDistribution should also work the way you are expecting.
from horde.
Related Issues (20)
- ETS lookup crash in Registry.lookup
- Horde Process topology HOT 12
- DynamicSupervisor.start_child hangs on infinite timeout HOT 3
- Is it possible to start a Horde dynamic supervisor with a registered name? HOT 2
- match error when stopping a process HOT 1
- Stopping Child via Horde.DynamicSupervisor.terminate_child with via_tuple HOT 1
- Occasional Horde Registry Lookup Misses HOT 2
- GenServer.cast cant find process via {:via, Horde.Registry, ...} HOT 6
- race condition in name conflict in same registry process results in all processes losing HOT 20
- Default random node distribution strategy HOT 5
- Match error while doing process handoff in dynamic supervisor
- Use new PartitionSupervisor
- Timeouts and Crashes in the Cluster HOT 2
- Disappearing values from Horde.Registry after spawning more than two nodes. HOT 2
- Why we need to handle {:error, {:already_started, pid}} ? HOT 4
- SayHello GenServer never (re)started on `count3` node?
- uniform_random_distribution.ex not in deps 8.7 HOT 1
- Old child specs remain in Supervisor CRDT after process handoff
- Registry.lookup doesn't always return new item that was just registered HOT 13
- Example of How to Start Children like a Regular Supervisor
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from horde.