Coder Social home page Coder Social logo

samrocketman / docker-compose-ha-consul-vault-ui Goto Github PK

View Code? Open in Web Editor NEW
204.0 7.0 66.0 136 KB

A docker-compose example of HA Consul + Vault + Vault UI

License: MIT License

Shell 80.61% HTML 12.67% HCL 6.72%
consul vault docker docker-compose high-availability

docker-compose-ha-consul-vault-ui's Introduction

HA Consul + Vault + Vault UI

Docker Logo Consul Logo Vault Logo VaultBoy Logo

This project is an example of using Consul, Vault, and Vault UI in a high availability (HA) configuration. Conveniently packaged as Docker services for provisioning via Docker Compose.

Features:

  • dnsmasq makes Consul DNS available to all containers. A secondary dnsmasq server is provided which grants HA to the DNS available to all containers. This allows consul-template to update DNS with zero DNS downtime. consul-template will create a lock to ensure it is not possible for both primary and secondary DNS servers to be down during DNS configuration updates as part of service discovery.
  • consul-template updates dnsmasq configuration and restarts dnsmasq when the configuration has changed (e.g. consul cluster size is increased on the fly). This makes consul DNS lookups HA.
  • Vault is registered via service discovery which is exposed via Consul DNS.
  • Persists data across restarts as long as the cluster is gracefully shut down. See [Starting and stopping section][#starting-and-stopping].
  • Local docker infrastructure is able to anonymously authenticate with Vault via approle method and its CIDR address.
  • Linux and Mac OS with docker supported.

Prerequisites

Supplemental reading material:

Getting started

Start the cluster

Remove --scale vault=3 if you want to start one instance of Vault. docker compose up -d would bring only Consul up in HA configuration.

./scripts/consul-agent.sh --bootstrap
docker compose up --scale vault=3 -d

Configure your web browser

Configure your browser to use the SOCKS5 proxy listening on localhost:1080. With your browser configured to use the proxy visit http://consul.service.consul:8500/ and wait for the cluster to be ready. After the vault service has all nodes available, it is time to initialize vault.

Initialize Vault

If you wish to secure secret.txt with GPG, then set the recipient_list environment variable. For example, the following.

export recipient_list="<gpg fingerprint to your secret gpg key>"

If you do not use GPG or do not want to, then skip setting recipient_list. Initialize vault witht he following command.

./scripts/initialize-vault.sh

The credentials for vault are located in the file secret.txt which is created when Vault is initialized. Alternately, secret.txt.gpg if using GPG encryption.

Visit the web UI

Configure your browser

Configure your web browser to use the SOCKS5 proxy listening on localhost:1080.

In Firefox, do the following:

  1. Edit connections settings
  2. Set Manual proxy configuration
  3. Set SOCKS host to localhost, set Port to 1080, and check SOCKS v5 boolean.

Alternately install FoxyProxy extension which is an extension for quickly switching proxies on or off.

For other browsers, web search how to configure proxy settings or see what extensions are available for managing proxy settings.

Visit services via Consul DNS

Visit http://portal.service.consul/. It provides links to other web UIs and if you configure additional portal services, then they will also show up automatically.

Alternately, you can visit consul and vault directly at:

To log into Vault UI you must generate for yourself an admin token.

./scripts/get-admin-token.sh

The root user token for Vault is stored in secret.txt at the root of this repository after you initialize Vault.

Other portal services

For playing around with service discovery I have created other docker compose files which will automatically register with this consul cluster. Here's a list of what I have created so far.

Experiment

With HA enabled, container instances of consul and vault can be terminated with minor disruptions.

Consul can be scaled up on the fly. consul-template will automatically update dnsmasq to include new services. dnsmasq will experience zero downtime.

docker compose up --scale vault=3 --scale consul-worker=6 -d

To play with failover for killing consul instances, it is recommended to review fault tolerance for consul HA deployments.

Starting and stopping

Because high availability clusters have to gossip across nodes you can't execute a simple docker compose down without corrupting the clusters. Instead, you have to gracefully shut down all clusters that depend on consul and then gracefully shutdown consul itself. For this, I have provided a script.

Stop consul and vault cluster safely.

./scripts/graceful-shutdown.sh

Start the consul and vault clusters.

docker compose up -d

Troubleshooting

DNS

Currently, output from the dnsmasq and dnsmasq-secondary servers are minimal. Verbosity of output can be increased for troubleshooting. Edit docker compose.yml and add --log-queries to the dnsmasq command.

DNS client troubleshooting using Docker.

docker compose run dns-troubleshoot

Using the dig command inside of the container.

# rely on the internal container DNS
dig consul.service.consul

# specify the dnsmasq hostname as the DNS server
dig @dnsmasq vault.service.consul

# reference vault DNS by tags
dig active.vault.service.consul
dig standby.vault.service.consul

Logs

View vault logs.

docker compose logs vault

User docker exec to log into container names. It allows you to poke around the runtime of the container.

SOCKS5 proxy

Run a SOCKS5 proxy for use with your browser.

docker run --network docker-compose-ha-consul-vault-ui_internal --dns 172.16.238.2 --init -p 127.0.0.1:1080:1080 --rm serjs/go-socks5-proxy

Configure your browser to use SOCKS proxy at 127.0.0.1:1080.

Recovering data

It's possible a cluster was shutdown uncleanly and put into an irrecoverable state with no leader. If you have ever cleanly shut down consul, then it's possible you have a backup in the backups/ directory.

If you're in this leaderless state, then wipe out your old cluster data with the following command (this will permanently delete all old data).

docker compose down -v

Start a new cluster.

docker compose up -d

The latest backup can be restored via the following script.

./scripts/restore-consul.sh

If you have a specific backup you wish to restore, then you can call it as an argument.

./scripts/restore-consul.sh backups/backup.snap

Screenshots

show portal before services are available


show portal after services are available


consul screenshot of all discovered services


consul screenshot of service metadata


License

MIT License

docker-compose-ha-consul-vault-ui's People

Contributors

samrocketman avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

docker-compose-ha-consul-vault-ui's Issues

Script de lancement en défaut

Bonjour c'est un projet TRES COMPLET, j'ai essayé le script avec ou sans les trois instances vaults, mais le script est en défaut, je regarderai tout ça plus tard, je monte actuellement un projet sur la transparence et la confidentialité, j'espère que tout va bien, des compétences comme la tienne pourrait tout à fait être en adéquation avec les ambitions de cette activité pour tous.

Supported OS?

Attempting to use this on MacOS but running into quite a few issues with file encoding and commands etc. Some examples:

  • sh: ./scripts/consul-agent.sh: bad interpreter: /bin/sh^M: no such file or directory
    • Running with sh directly appears to work.(Unsure as I wasn't able to access UI, but it does appear to have brought up the needed dockers instances when running docker-compose up)
  • : command not founde-vault.sh: line 2: : invalid optionlize-vault.sh: line 3: set: - set: usage: set [--abefhkmnptuvxBCHP] [-o option] [arg ...]
    • Not too sure about this one

Not sure if this is a me issue, or just has not been tested on MacOS

Either way, looking forward to testing this on a Linux box if needed

fails to start

Docker version 19.03.13, build 4484c46d9d
CentOS Linux release 8.2.2004 (Core)

$ docker system prune -a 
...
$ ./scripts/consul-agent.sh --bootstrap
...
$ docker-compose up --scale vault=3 -d
Creating network "docker-compose-ha-consul-vault-ui_internal" with driver "bridge"
Pulling dnsmasq (storytel/dnsmasq:)...
latest: Pulling from storytel/dnsmasq
ff3a5c916c92: Pull complete
be8a008d89ab: Pull complete
Digest: sha256:73f265e405ac8e94c43913e1b3e10b759df2b6508b6922a635d5b68b48f907ec
Status: Downloaded newer image for storytel/dnsmasq:latest
Pulling dns-troubleshoot (joffotron/docker-net-tools:)...
latest: Pulling from joffotron/docker-net-tools
3690ec4760f9: Pull complete
0905b79e95dc: Pull complete
Digest: sha256:5752abdc4351a75e9daec681c1a6babfec03b317b273fc56f953592e6218d5b5
Status: Downloaded newer image for joffotron/docker-net-tools:latest
Pulling consul (consul:)...
latest: Pulling from library/consul
df20fa9351a1: Pull complete
c6f53b97be57: Pull complete
cdc3ccb924c0: Pull complete
7d16d05c330f: Pull complete
81ed495875d8: Pull complete
c418676c3be3: Pull complete
Digest: sha256:36fd1f8ca4d702c7dce0d58662893245ff417c863ba38226e73be7f722d0efcb
Status: Downloaded newer image for consul:latest
Pulling vault (vault:)...
latest: Pulling from library/vault
21c83c524219: Pull complete
ef53fb8b17ea: Pull complete
e49dccf6889d: Pull complete
de69c6ffd193: Pull complete
34a6ec7c1ebc: Pull complete
Digest: sha256:121c1eb16a474f5a4c1d92256184dae333ab7284f8c744d4e2754300f84f68f0
Status: Downloaded newer image for vault:latest
Pulling portal (nginx:)...
latest: Pulling from library/nginx
d121f8d1c412: Pull complete
66a200539fd6: Pull complete
e9738820db15: Pull complete
d74ea5811e8a: Pull complete
ffdacbba6928: Pull complete
Digest: sha256:fc66cdef5ca33809823182c9c5d72ea86fd2cef7713cf3363e1a0b12a5d77500
Status: Downloaded newer image for nginx:latest
Pulling socks (serjs/go-socks5-proxy:)...
latest: Pulling from serjs/go-socks5-proxy
8dc4cb174fdf: Pull complete
2d61e0950126: Pull complete
Digest: sha256:bbf794aae0ffa3b3ce6c10db464aa118266bdbd6aacab3441472c8a5924db58c
Status: Downloaded newer image for serjs/go-socks5-proxy:latest
Creating docker-compose-ha-consul-vault-ui_dnsmasq_1           ... done
Creating docker-compose-ha-consul-vault-ui_dnsmasq-secondary_1 ... done
Creating docker-compose-ha-consul-vault-ui_socks_1             ... done
Creating docker-compose-ha-consul-vault-ui_consul_1            ... done

ERROR: for consul-worker  Container "a856e3c25f2b" is unhealthy.
ERROR: Encountered errors while bringing up the project.

$ docker-compose logs
service-log.txt

Troubleshooting:

  • docker-compose restart consul
  • I tried updating the version of vault and consul to 1.5.4 and 1.8.4.
  • Tried manually starting consul

Vault Swarm IP issue

First of all, thank you so much for this repo. I found the Vault documentation to be severely lacking when trying to understand how to deal with Vault in a Swarm scenario.

I borrowed this section of code which was the major breakthrough to get it working for me, but after much debugging, found that the "default" network was not the correct network to be connecting to. I'm not sure exactly why, but my default was pointing to eth2 and it needed to be eth0. After modifying the script to get eth0's IP, clustering the Vault containers worked beautifully.

Mostly putting this here for anyone else to find as well.

if we don't want to use dnsmasq

Nice but can we do the same without dnsmasq and by exposing the current vault service available ? Do you have any idea how to change it ?

missing vaultaddr parameter

Currently running the docker-compose verbatim on an EC2 instance for testing purposes.
After successful unseal, attempting to connect to Vault using the EC2's private ip address (security groups are not an issue here where 10.201.144.8 is the private ip of the EC2 instance).

Getting error:

$ vault login -address=http://10.201.144.8:8000
Token (will be hidden):
Error authenticating: error looking up token: Error making API request.

URL: GET http://10.201.144.8:8000/v1/auth/token/lookup-self
Code: 400. Raw Message:

missing vaultaddr parameter

The same is noted from the browser on http://10.201.144.8:8000/v1/auth/token/lookup-self of course

dnsmasq containers going down

Just as dnsmasq are trying to up, they both go down and nginx too. dnsmasq log is below,

+ checksum -c /tmp/consul-template_0.26.0_linux_amd64.zip.sha256sum
+ type sha256sum
+ sha256sum -c /tmp/consul-template_0.26.0_linux_amd64.zip.sha256sum
sha256sum is /usr/bin/sha256sum
consul-template_0.26.0_linux_amd64.zip: OK
+ unzip consul-template_0.26.0_linux_amd64.zip
Archive:  consul-template_0.26.0_linux_amd64.zip
unzip: 'consul-template' exists but is not a regular file

The problem is with consul-agent,sh script function:

download() {
  # $1 is the application
  # $2 is the version
  # $3 may not exist but if so is the destination path
  if [ ! "$3" = "./" ] && type "$1" || [ -f "$1" ]; then
    return
  fi
  (
    zip_file="$1"_"$2"_linux_amd64.zip
    if [ -z "$3" ]; then
      cd "$bin_path"
    else
      cd "$3"
    fi
    curl -fL https://releases.hashicorp.com/"$1"/"$2"/"$1"_"$2"_SHA256SUMS | \
      grep -- "$zip_file" > /tmp/"$zip_file".sha256sum
    until checksum -c /tmp/"$zip_file".sha256sum; do
      curl -LO https://releases.hashicorp.com/"$1"/"$2"/"$zip_file"
      sleep 3
    done
    unzip "$zip_file"
    chmod 755 "$1"
    rm "$zip_file"
    rm /tmp/"$zip_file".sha256sum
  )
}


Because it does not download the zipped version right file:

https://releases.hashicorp.com/consul-template/0.26.0/

I have set enough permissions on all folders : chmod -R 777 yet that didnt solve

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.