sivachinnayan / consul-cluster-setup Goto Github PK

View Code? Open in Web Editor NEW

0.0 1.0 0.0 11 KB

setup consul cluster setup

License: MIT License

JavaScript 100.00%

consul-cluster-setup's Introduction

consul-cluster-setup

setup consul cluster setup

In Node two follow the below instructions

$sudo -s
$ echo '{"service": {"name": "web", "tags": ["rails"], "port": 80,
  "check": {"script": "curl localhost >/dev/null 2>&1", "interval": "10s"}}}' > /etc/consul.d/web.json 
  
$ echo '{"services":[{"id":"webservice-poll","name":"webservice-poll","tags":["poll"],"port":80,"checks":[{"name":"Sys check ","script":"some_scripts.sh param1 param2","interval":"5s","timeout":"1s"},{"name":"Check if service is alive","http":"http://localhost:80/checkservice","interval":"5s","timeout":"1s"},{"name":"HTTP on port 80","http":"http://localhost:80/","interval":"5s","timeout":"1s"},{"id":"check particular tcp port","name":"check a port","tcp":"localhost:3415","interval":"10s","timeout":"1s"}]}]}' > /etc/consul.d/checkbservice.json

Run Node1 as Server

$consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -node=agent-one -bind=172.20.20.10 -client=0.0.0.0 -config-dir /etc/consul.d/ -ui

172.20.20.10 - private ipaddress of node1
bootstrap-expect [num] - num tells the server that it expect atleast one node to join.
consul can run either as server or agent, -server makes the consul agent as server
-node gives a name to identify the node

Notes: Expose client = 0.0.0.0 inorder to accomplish port forwarding to localhost

Run Node2 as client and join to cluster

$consul agent --data-dir /tmp/consul -node=agent-two -bind=172.20.20.11 -config-dir /etc/consul.d/

Note

172.20.20.11 - private ip address of node2
-bind telling consul to bind the localip address, so while advertising itself it will use this ipaddress. Can also see -advertise or -advertise-wan
--data-dir - tells the consul to refer or store all intermediate meta-datas for the consul

Join Node2 from node1

Login to machine one - ie, node

$vagrant ssh node1
node1@$consul members 
node1@$consul join 172.20.20.11

The join command will make the node join to cluster and election process happens to elect cluster leader.

#Run Node3 as client and join to cluster

$consul agent --data-dir /tmp/consul -node=agent-three -bind=172.20.20.12 -config-dir /etc/consul.d/

172.20.20.12 - local private ipaddres of node3
node = agent-three, this is used to identify node3 among other nodes in cluster.

#join node3 to cluster

$vagrant ssh node1
node1@$consul members 
node1@$consul join 172.20.20.12

note the node can join to any member in the cluster, its not mandatory to join from server node.

** DNS Setup on localhost

localhostmachine@user$sudo apt-get install dnsmasq -y
localhostmachine@user$sudo -s
localhostmachine@user@echo "server=/consul/127.0.0.1#8600" > /etc/dnsmasq.d/10-consul
localhostmachine@user@sudo service dnsmasq restart
 * Restarting DNS forwarder and DHCP server dnsmasq                           [ OK ]

consul-cluster-setup's People

Contributors

Watchers

consul-cluster-setup's Issues

Typical Health Check WebServer Running on Port 80

How to join a third party node to Consul

How to join a third party node to Consul, Example. I want to join my localbox to consul running on local network. I can follow the below steps to acheive it.

siva@user1:~$ consul agent  -data-dir /tmp/consul-tmp -bind 192.168.10.86
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
         Node name: 'user1'
        Datacenter: 'dc1'
            Server: false (bootstrap: false)
       Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
      Cluster Addr: 192.168.10.86 (LAN: 8301, WAN: 8302)
    Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
             Atlas: <disabled>

==> Log data will now stream in as it occurs:

Join to Consul Server running on local network

siva@user1:/etc/consul.d$ consul join 192.168.10.47
Successfully joined cluster by contacting 1 nodes.

siva@usuer1:/etc/consul.d$ echo '{"service": {"name": "web", "tags": ["rails"], "port": 80,"check": {"script": "curl localhost >/dev/null 2>&1", "interval": "10s"}}}' > /etc/consul.d/web.json^C

siva@user1:/etc/consul.d$ consul members 
Node                   Address             Status  Type    Build  Protocol  DC
user1                 192.168.10.86:8301  alive   client  0.6.4  2         dc1
localhost.localdomain  192.168.10.47:8301  alive   server  0.6.4  2         dc1

siva@user1:/etc/consul.d$ consul members ^C
siva@user1:/etc/consul.d$ sudo python -m SimpleHTTPServer 80 
Serving HTTP on 0.0.0.0 port 80 ...
127.0.0.1 - - [26/Aug/2016 18:13:19] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [26/Aug/2016 18:13:29] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [26/Aug/2016 18:13:39] "GET / HTTP/1.1" 200 -

Join multiple Nodes to Cluster

Multiple Health Check Metrics - Download and Install NVM - Nodejs/ Run Express Project on Port 80

Consul Cluster Diagram for one Application Server

Security Concerns with Consul

How to secure the communication channel between nodes within a cluster/service in consul..

Links

consul-security

Route Node Application Via DNS Service

Enable and Disbale Node application running on node 2 - 172.20.20.11

root@node2:/home/vagrant/express# node index.js 
^C
root@node2:/home/vagrant/express# node index.js 
^C
root@node2:/home/vagrant/express#

Ping from LocalBox

As we can see the ping on webservice-poll switches between consul node based on the availability. That is if the health of the node is active it will return that ip else return rest of ipaddres or nothing.

siva@$ ping webservice-poll.service.consul
PING webservice-poll.service.consul (172.20.20.12) 56(84) bytes of data.
64 bytes from 172.20.20.12: icmp_seq=1 ttl=64 time=0.132 ms
64 bytes from 172.20.20.12: icmp_seq=2 ttl=64 time=0.191 ms
64 bytes from 172.20.20.12: icmp_seq=3 ttl=64 time=0.168 ms
^C
--- webservice-poll.service.consul ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.132/0.163/0.191/0.028 ms
siva@$ ping webservice-poll.service.consul
PING webservice-poll.service.consul (172.20.20.12) 56(84) bytes of data.
64 bytes from 172.20.20.12: icmp_seq=1 ttl=64 time=0.150 ms
64 bytes from 172.20.20.12: icmp_seq=2 ttl=64 time=0.215 ms
^C
--- webservice-poll.service.consul ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.150/0.182/0.215/0.035 ms
siva@$ ping webservice-poll.service.consul
PING webservice-poll.service.consul (172.20.20.11) 56(84) bytes of data.
64 bytes from 172.20.20.11: icmp_seq=1 ttl=64 time=0.172 ms
64 bytes from 172.20.20.11: icmp_seq=2 ttl=64 time=0.154 ms
64 bytes from 172.20.20.11: icmp_seq=3 ttl=64 time=0.212 ms
^C
--- webservice-poll.service.consul ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.154/0.179/0.212/0.026 ms

siva@$ ping webservice-poll.service.consul
PING webservice-poll.service.consul (172.20.20.12) 56(84) bytes of data.
64 bytes from 172.20.20.12: icmp_seq=1 ttl=64 time=0.142 ms
64 bytes from 172.20.20.12: icmp_seq=2 ttl=64 time=0.149 ms
64 bytes from 172.20.20.12: icmp_seq=3 ttl=64 time=0.144 ms
^C
--- webservice-poll.service.consul ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.142/0.145/0.149/0.003 ms
siva@$

How to load balance between consul nodes

DNS forwarding from DNSMASQ to Consul DNS:8600

https://www.consul.io/docs/guides/forwarding.html

Install dnsmasq on all nodes.

Make sure to uncomment the below line from /etc/dnsmasq.conf
conf-dir=/etc/dnsmasq.d

cat /etc/dnsmasq.conf  | tail -10
# For debugging purposes, log each DNS query as it passes through
# dnsmasq.
#log-queries

# Log lots of extra information about DHCP transactions.
#log-dhcp

# Include a another lot of configuration options.
#conf-file=/etc/dnsmasq.more.conf
conf-dir=/etc/dnsmasq.d  ## uncomment this line

$vi /etc/resolv.conf
nameserver 127.0.0.1 # add this in resolv.conf

echo "server=/consul/127.0.0.1#8600" > /etc/dnsmasq.d/10-consul

dig @127.0.0.1 -p 8600 appserver.service.consul

dig appserver.service.consul

DNS Forwarding to localbox

How to do DNS forwaring to LocalBox using Vagrant Setup.

You would needs to port forward both UDP/TCP from vagrant to localhost

By Default port 8600 is defined for DNS query. and 8500 for REST - Consul

https://github.com/sivachinnayan/consul-cluster-setup/blob/master/Vagrantfile#L32-L40

UDP_PORTS_LIST={
  "8500" => 8500, # Some service
  "8600" => 8600 # VNC
}

TCP_PORTS_LIST={
  "8600" => 8600, # VNC
  "8500" => 8500 # Some service
}

https://github.com/sivachinnayan/consul-cluster-setup/blob/master/Vagrantfile#L55-L60


      UDP_PORTS_LIST.each do |guest, host|
          node1.vm.network "forwarded_port", guest: "#{guest}", host: "#{host}", protocol: "udp"
      end
      TCP_PORTS_LIST.each do |guest, host|
          node1.vm.network "forwarded_port", guest: "#{guest}", host: "#{host}", protocol: "tcp"
      end

siva@$ dig @127.0.0.1 -p 8600 agent-one.node.consul

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @127.0.0.1 -p 8600 agent-one.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39695
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;agent-one.node.consul.     IN  A

;; ANSWER SECTION:
agent-one.node.consul.  0   IN  A   172.20.20.10

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Fri Aug 26 10:46:15 IST 2016
;; MSG SIZE  rcvd: 76

siva@$ dig @127.0.0.1 -p 8600 agent-two.node.consul

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @127.0.0.1 -p 8600 agent-two.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11696
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;agent-two.node.consul.     IN  A

;; ANSWER SECTION:
agent-two.node.consul.  0   IN  A   172.20.20.11

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Fri Aug 26 10:46:30 IST 2016
;; MSG SIZE  rcvd: 76

siva@$ dig @127.0.0.1 -p 8600 agent-three.node.consul

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @127.0.0.1 -p 8600 agent-three.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42923
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;agent-three.node.consul.   IN  A

;; ANSWER SECTION:
agent-three.node.consul. 0  IN  A   172.20.20.12

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Fri Aug 26 10:46:35 IST 2016
;; MSG SIZE  rcvd: 80

siva@:~/$

DNS Query

Service DNS Query TLD service.consul or node.consul

vagrant@n1:/etc/consul.d$ dig @127.0.0.1 -p 8600 web.service.DC1.consul SRV

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> @127.0.0.1 -p 8600 web.service.DC1.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26552
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;web.service.DC1.consul.        IN  SRV

;; ANSWER SECTION:
web.service.DC1.consul. 0   IN  SRV 1 1 80 agent-two.node.dc1.consul.

;; ADDITIONAL SECTION:
agent-two.node.dc1.consul. 0    IN  A   172.20.20.11

;; Query time: 5 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Aug 25 12:14:19 2016
;; MSG SIZE  rcvd: 148

vagrant@n1:/etc/consul.d$ dig @127.0.0.1 -p 8600 webservice-poll.service.DC1.consul SRV

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> @127.0.0.1 -p 8600 webservice-poll.service.DC1.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 9772
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;webservice-poll.service.DC1.consul. IN SRV

;; AUTHORITY SECTION:
consul.         0   IN  SOA ns.consul. postmaster.consul. 1472127269 3600 600 86400 0

;; Query time: 4 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Aug 25 12:14:29 2016
;; MSG SIZE  rcvd: 120

vagrant@n1:/etc/consul.d$

Node DNS Query


agrant@n1:/etc/consul.d$ 
vagrant@n1:/etc/consul.d$ dig @127.0.0.1 -p 8600 agent-two.node.consul

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> @127.0.0.1 -p 8600 agent-two.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24951
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;agent-two.node.consul.     IN  A

;; ANSWER SECTION:
agent-two.node.consul.  0   IN  A   172.20.20.11

;; Query time: 4 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Aug 25 12:16:07 2016
;; MSG SIZE  rcvd: 76

vagrant@n1:/etc/consul.d$ dig @127.0.0.1 -p 8600 agent-one.node.consul

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> @127.0.0.1 -p 8600 agent-one.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23487
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;agent-one.node.consul.     IN  A

;; ANSWER SECTION:
agent-one.node.consul.  0   IN  A   172.20.20.10

;; Query time: 4 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Aug 25 12:16:11 2016
;; MSG SIZE  rcvd: 76

vagrant@n1:/etc/consul.d$

How to setup DNS Forwarding from LocalBox to Vagrant Machine

How to reduce Consul Logs

Within a week with 4 services running with max of 2 nodes on/off threw log size of more than 110 MB, There needs to be a log rotation to keep in check the log size.

[root@localhost log]# ls -rlth consul.log 
-rw-r--r--. 1 root root 114M Aug 31 14:33 consul.log
[root@localhost log]#