klusterd's Introduction

klusterd

klusterd is a sample Akka Cluster project that is packaged using Docker and deployed to Kubernetes.

Taking klusterd for a test drive

First, we'll use sbt to package the application as a Docker image.

$ cd klusterd
$ sbt "docker:publishLocal"

This creates a local Docker image for us:

$ docker images | grep klusterd
vyshane/klusterd                      1.0                  43ee995e8adb        1 minute ago      690.7 MB

To deploy our klusterd image to our Kubernetes cluster:

$ cd ../deployment
$ ./up.sh

We can see that a klusterd pod comes up:

$ kubectl get pods
NAME                   READY     STATUS    RESTARTS   AGE
klusterd-6srxl         1/1       Running   0          1m

Let's tail its log:

$ kubectl logs -f klusterd-6srxl

We can see that klusterd is running at the IP address 172.17.0.3 and listening on port 2551.

INFO  14:58:06.022UTC akka.remote.Remoting - Starting remoting
INFO  14:58:06.235UTC akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:2551]

Since we have only launched one klusterd node, it is its own cluster seed node.

INFO  14:58:06.446UTC akka.actor.ActorSystemImpl(klusterd) - Configured seed nodes: akka.tcp://[email protected]:2551

We have a cluster of one.

INFO  14:58:06.454UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Node [akka.tcp://[email protected]:2551] is JOINING, roles []
INFO  14:58:06.482UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:2551] to [Up]
INFO  14:58:06.505UTC akka.tcp://[email protected]:2551/user/cluster-monitor - Cluster member up: akka.tcp://[email protected]:2551

Now, let's scale this cluster up. Let's ask Kubernetes for 2 more nodes:

$ kubectl scale --replicas=3 rc klusterd

We can see from klusterd-6srxl's logs that two more nodes join our cluster shortly after.

INFO  15:11:46.525UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Node [akka.tcp://[email protected]:2551] is JOINING, roles []
INFO  15:11:46.533UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Node [akka.tcp://[email protected]:2551] is JOINING, roles []
INFO  15:11:47.484UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:2551] to [Up]
INFO  15:11:47.505UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:2551] to [Up]
INFO  15:11:47.507UTC akka.tcp://[email protected]:2551/user/cluster-monitor - Cluster member up: akka.tcp://[email protected]:2551
INFO  15:11:47.507UTC akka.tcp://[email protected]:2551/user/cluster-monitor - Cluster member up: akka.tcp://[email protected]:2551

We can scale the cluster down:

$ kubectl scale --replicas=2 rc klusterd

And a node leaves the cluster. In our case, the original pod klusterd-6srxl was killed. Here are the logs from another node showing what happened:

WARN  15:19:31.855UTC akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fklusterd%40172.17.0.3%3A2551-0 - Association with remote system [akka.tcp://[email protected]:2551] has failed, address is now gated for [5000] ms. Reason: [Disassociated] 
WARN  15:19:35.376UTC akka.tcp://[email protected]:2551/user/cluster-monitor - Cluster member unreachable: akka.tcp://[email protected]:2551
WARN  15:19:35.405UTC akka.tcp://[email protected]:2551/system/cluster/core/daemon - Cluster Node [akka.tcp://[email protected]:2551] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://[email protected]:2551, status = Up)]
INFO  15:19:45.384UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Leader is auto-downing unreachable node [akka.tcp://[email protected]:2551]
INFO  15:19:45.387UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Marking unreachable node [akka.tcp://[email protected]:2551] as [Down]
INFO  15:19:46.412UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Leader is removing unreachable node [akka.tcp://[email protected]:2551]
INFO  15:19:46.414UTC akka.tcp://[email protected]:2551/user/cluster-monitor - Cluster member removed: akka.tcp://[email protected]:2551
WARN  15:19:52.756UTC akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fklusterd%40172.17.0.3%3A2551-3 - Association with remote system [akka.tcp://[email protected]:2551] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://[email protected]:2551]] Caused by: [No response from remote for outbound association. Associate timed out after [15000 ms].]
INFO  15:19:52.758UTC akka.tcp://[email protected]:2551/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fklusterd%40172.17.0.3%3A2551-4 - No response from remote for outbound association. Associate timed out after [15000 ms].

To turn off everything:

./down.sh

klusterd's People

Stargazers

Watchers

klusterd's Issues

split brain problem

Hey, I've test that if I initially start 4 replica, than there is high risk to trigger brain splitting(because there is no "first seed node" to avoid brain split). Though we can start the klustrd as 1 replica,than scale out, but this might also happen if the rc restart.
The log,when I initially start 4 replica:

root@SZV1000050172:/home/docker/workspace/xuanhaoran/klusterd-master/deployment# kubectl get pods |grep klusterd
klusterd-42t9x               1/1       Running             0          4m
klusterd-gh8iy               1/1       Running             0          4m
klusterd-h7h2v               1/1       Running             0          4m
klusterd-zsi8h               1/1       Running             0          4m

root@SZV1000050172:/home/docker/workspace/xuanhaoran/klusterd-master/deployment# kubectl logs -f klusterd-42t9x
[DEBUG] [03/20/2017 08:59:57.965] [main] [EventStream] StandardOutLogger started
INFO    - Slf4jLogger started
[DEBUG] [03/20/2017 08:59:58.592] [main] [EventStream(akka://klusterd)] logger log1-Slf4jLogger started
[DEBUG] [03/20/2017 08:59:58.594] [main] [EventStream(akka://klusterd)] Default Loggers started
INFO  08:59:58.696UTC akka.remote.Remoting - Starting remoting
INFO  08:59:58.918UTC akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:2551]
INFO  08:59:58.939UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Starting up...
INFO  08:59:59.097UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Registered cluster JMX MBean [akka:type=Cluster]
INFO  08:59:59.097UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Started up successfully
INFO  08:59:59.133UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar
INFO  08:59:59.170UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Metrics collection has started successfully
INFO  08:59:59.190UTC akka.actor.ActorSystemImpl(klusterd) - Configured seed nodes: akka.tcp://[email protected]:2551, akka.tcp://[email protected]:2551
INFO  08:59:59.794UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Welcome from [akka.tcp://[email protected]:2551]
INFO  08:59:59.814UTC akka.tcp://[email protected]:2551/user/cluster-monitor - Cluster member up: akka.tcp://[email protected]:2551
INFO  09:00:00.184UTC akka.tcp://[email protected]:2551/user/cluster-monitor - Cluster member up: akka.tcp://[email protected]:2551

root@SZV1000050172:/home/docker/workspace/xuanhaoran/klusterd-master/deployment# kubectl logs -f klusterd-gh8iy
[DEBUG] [03/20/2017 09:01:00.898] [main] [EventStream] StandardOutLogger started
INFO    - Slf4jLogger started
[DEBUG] [03/20/2017 09:01:01.302] [main] [EventStream(akka://klusterd)] logger log1-Slf4jLogger started
[DEBUG] [03/20/2017 09:01:01.303] [main] [EventStream(akka://klusterd)] Default Loggers started
INFO  09:01:01.360UTC akka.remote.Remoting - Starting remoting
INFO  09:01:01.582UTC akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:2551]
INFO  09:01:01.596UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Starting up...
INFO  09:01:01.735UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Registered cluster JMX MBean [akka:type=Cluster]
INFO  09:01:01.735UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Started up successfully
INFO  09:01:01.744UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar
INFO  09:01:01.745UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Metrics collection has started successfully
INFO  09:01:01.772UTC akka.actor.ActorSystemImpl(klusterd) - Configured seed nodes: akka.tcp://[email protected]:2551, akka.tcp://[email protected]:2551
WARN  09:01:02.044UTC akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fklusterd%40172.16.37.3%3A2551-0 - Association with remote system [akka.tcp://[email protected]:2551] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://[email protected]:2551]] Caused by: [Connection refused: /172.16.37.3:2551]
WARN  09:01:02.046UTC akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fklusterd%40172.16.90.2%3A2551-1 - Association with remote system [akka.tcp://[email protected]:2551] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://[email protected]:2551]] Caused by: [Connection refused: /172.16.90.2:2551]
INFO  09:01:12.349UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://[email protected]:2551] - Welcome from [akka.tcp://[email protected]:2551]
INFO  09:01:12.381UTC akka.tcp://[email protected]:2551/user/cluster-monitor - Cluster member up: akka.tcp://[email protected]:2551
INFO  09:01:12.665UTC akka.tcp://[email protected]:2551/user/cluster-monitor - Cluster member up: akka.tcp://[email protected]:2551

as you can see ,actually there are two akka cluster created:

akka.tcp://[email protected]:2551 + akka.tcp://[email protected]:2551(seed)
akka.tcp://[email protected]:2551 + akka.tcp://[email protected]:2551(seed)

Recommend Projects