kubernetes-sigs / etcdadm

etcdadm

etcdadm is a command-line tool for operating an etcd cluster. It makes it easy to create a new cluster, add a member to, or remove a member from an existing cluster. Its user experience is inspired by kubeadm.

For major project goals, see the roadmap

Table of Contents

• Getting Started
  • Building
  • Creating a new cluster
  • Adding a member
  • Removing a member
• Advanced Usage
  • Creating a new cluster from a snapshot
• Caveats & Limitations
• Design
• Questions

Getting Started

Building

1. Clone the git repository.
2. Build on the host:

   make etcdadm
   

3. Build in a container, using docker:

   make container-build
   

   If you are in a country/region where Internet access is restricted and you cannot download Go dependencies successfully, make with the GOPROXY environment variable set to a third party you trust, for example:

   GOPROXY=https://goproxy.io make container-build
   

   goproxy.io is not endorsed specifically by the project.

Creating a new cluster

1. Copy etcdadm to each machine that will become a member.
2. Choose one machine and run

etcdadm init

Adding a member

1. Copy the CA certificate and key from any machine in the cluster to the machine being added.

rsync -avR /etc/etcd/pki/ca.* <Member IP address>:/

2. Choose a cluster endpoint (i.e. client URL of some member) and run

etcdadm join <endpoint>

Removing a member

On the machine being removed, run

etcdadm reset

Advanced Usage

Creating a new cluster from a snapshot

If you have an existing etcd snapshot, you can use it to create a new cluster:

etcdadm init --snapshot /path/to/etcd.snapshot

Caveats and Limitations

1. Must run as root. (This is because etcdadm creates a systemd service)
2. Does not support etcd v2.
3. Currently tested on Container Linux, with plans for other platforms.

Design

The goal of etcdadm is to make it easy to operate an etcd cluster. It downloads a specific etcd release, installs the binary, configures a systemd service, generates certificates, calls the etcd API to add (or remove) a member, and verifies that the new member is healthy.

Etcdadm must be run on the machine that is being added or removed. As a consequence, if a member permanently fails, and the operator cannot invoke etcdadm reset on that machine, the operator must use the etcd API to delete the failed member from the list of members.

On its own, etcdadm does not automate cluster operation, but a cluster orchestrator can delegate all the above tasks to etcdadm.

Sequence Diagrams

See docs/diagrams for sequence diagrams of init, join, and reset.

Questions?

For more information reach out to etcdadm slack channel

etcdadm's Issues

Create etcdctl wrapper in install base dir

If the install base dir is /opt/bin and etcd v3.1.12 is installed, create /opt/bin/etcdctl containing:

#!/usr/bin/env sh
source /etc/etcd/etcdctl.env
/opt/bin/etcd-v3.1.12/etcdctl "$@"

Impose deadline on exec calls

https://golang.org/pkg/os/exec/#CommandContext

Make endpoint a required positional argument of etcd join

Additional SANs

There doesn't seem to be a configuration available during init/join to add SANs for the generated member server certificate. The pki code currently infers the private ip address from host default interface.

This becomes mandatory if you run the etcd cluster in some cloud private network and you want to expose it to clients that are

• from a different network interface
• using NATed ips (e.g openstack floating ip)

Perhaps it would make more sense to provide this via a local config file and not with additional CLI parameters.

Only create symlinks for etcd and etcdctl

After running etcdadm init:

# ls -al /opt/bin
total 17076
drwx------. 3 root root     4096 Jun 29 19:23 .
drwx------. 3 root root     4096 Jun 19 03:18 ..
lrwxrwxrwx. 1 root root       26 Jun 29 19:23 Documentation -> etcd-v3.1.12/Documentation
lrwxrwxrwx. 1 root root       30 Jun 29 19:23 README-etcdctl.md -> etcd-v3.1.12/README-etcdctl.md
lrwxrwxrwx. 1 root root       22 Jun 29 19:23 README.md -> etcd-v3.1.12/README.md
lrwxrwxrwx. 1 root root       32 Jun 29 19:23 READMEv2-etcdctl.md -> etcd-v3.1.12/READMEv2-etcdctl.md
lrwxrwxrwx. 1 root root       17 Jun 29 19:23 etcd -> etcd-v3.1.12/etcd
drwx------. 3 root root     4096 Jun 29 19:23 etcd-v3.1.12
-rwxr-xr-x. 1 root root 17425874 Jun 29 19:15 etcdadm
lrwxrwxrwx. 1 root root       20 Jun 29 19:23 etcdctl -> etcd-v3.1.12/etcdctl

Support etcd certificate rotation

The Peer, Client and Server certs we generate today is valid for one year. We will need to run etcdadm reset followed by etcdadm join/init to create new certs within the given year for the etcd cluster to continue functioning.

Cert rollover needs to be implemented.

Update etcd repo

• constants.DefaultReleaseURL should use https://github.com/etcd-io/etcd
• All go import paths should use go.etcd.io/etcd

etcd-io/etcd#9965

Add option to set log verbosity

These tools are very quiet at the moment making it hard to debug.

etcdadm reset must respect certificate dir

This is configurable in init/join, and reset must also respect it.

Add flag to only download and extract the etcd release to the install base dir

This can be used to generate the files etcdadm will expect to find in the install base dir on a host without internet connectivity.

For example:

etcdadm init --only-binaries

Will download the etcd release and extract it, but skip all subsequent steps.

Alternatively, we can add support for phases a la kubeadm, e.g.,

etcdadm phase binaries

Make operations repeatable and idempotent

init

• Make idempotent - Install binaries
• Make idempotent - Generate PKI
• Make idempotent - Generate configuration
• Make idempotent - Start service
• Implement - Verify local endpoint health (with retries up to a timeout) #54

join

• Make idempotent - Add member to cluster
• Make idempotent - Install binaries
• Make idempotent - Generate PKI
• Make idempotent - Generate configuration
• Make idempotent - Start service
• Implement - Verify local endpoint health (with retries up to a timeout) #54

reset

• Make idempotent - Remove member from cluster
• Make idempotent - Stop service
• Make idempotent - Remove configuration
• Make idempotent - Remove PKI
• Make idempotent - Remove binaries

Do not download etcd release if it is already in the local cache

etcd and etcdctl environment files should not be readable by anyone but root

The files can contain sensitive information that would otherwise be available only via the etcd API, which is not accessible without an etcd client cert and key, the latter of which is readable by root only.

Client certificate mode

It would be great if etcdadm could allow nodes that are not going to be etcd members, but instead clients of etcd to be able to create their own certificates. I think it would add value so that one binary can handle all necessary etcd setup tasks.

I understand that distributing ca certs to non-etcd nodes is a "bad idea", but configuration management tools could benefit here.

Change import paths to sigs.k8s.io/etcdadm

Verify that symlinks are correct as part of verifying installation

This is not verified, hence when the binaries are pre-downloaded, but the symlinks are not present (or incorrect), the symlinks do not get created.

Separate etcd configuration and environment file templates for init and join

The join configuration should not contain an initial cluster token, and the token should not be in the join environment file template.

Make download connect timeout configurable

etcdctl.sh fails when default shell is not bash

root@daniel-ubuntu16:~# /opt/bin/etcdctl.sh member list                                                                                                                                                                               
/opt/bin/etcdctl.sh: 6: /opt/bin/etcdctl.sh: source: not found
client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused                                                                                                               
; error #1: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

Implement in-place upgrade

Remove log calls from util.etcdclient

Return errors instead

Create a SECURITY_CONTACTS file.

As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.

The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".

Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)

Thanks so much, let me know if you have any questions.

(This issue was generated from a tool, apologies for any weirdness.)

[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md

Create symlink to etcd from the install base dir

If the install base dir is /opt/bin, and etcd v3.1.12 is installed, then create symlink /op/bin/etcd -> /opt/bin/etcd-v3.1.12/etcd.

Use token only for init

The token is used to generate a unique cluster ID.

We generate the clusterID based on hash(configuration, cluster-name)
-- etcd-io/etcd#1437 (comment)

If a multi-member cluster is statically bootstrapped, the members must use the same token. Providing a unique cluster name helps avoid the case that members of two clusters with identical static configurations accidentally join the wrong cluster.

Once a cluster is formed, members must be added using the API. The token is ignored.

Therefore the token should not be accepted as a flag to the etcdadm join command.

--
History:

This was first reported in etcd-io/etcd#1430, and the cluster token documentation was added in etcd-io/etcd#1437.

Support tuning configuration

The configuration is hard-coded in the systemd unit file:

LimitNOFILE=65536
Nice=-10
IOSchedulingClass=best-effort
IOSchedulingPriority=2

Keep these values as defaults, but make them user-configurable.

See https://coreos.com/etcd/docs/latest/tuning.html

Implement etcdadm reset

This will be used to remove the local member from the cluster and clean up any/all state and configuration files.

Usage:

etcdadm reset

Specific steps to be performed:

• Call "MemberRemove" for the local member
• Remove etcd certs
• Remove etcd environment file
• Remove etcdctl environment file
• Remove systemd unit file

This reset call must be idempotent.

Reset fails if init/join did not complete

etcdadm join failed because I did not have the CA cert on the host, and a etcdadm created a different CA cert; etcdadm could not present a valid client cert to the endpoint:

# etcdadm join https://172.16.0.84:2379 --token blah
2018/06/29 18:58:39 [install] verifying etcd 3.1.12 is installed in /opt/bin/etcd-v3.1.12
2018/06/29 18:58:40 [certificates] creating PKI assets
2018/06/29 18:58:40 creating a self signed etcd CA certificate and key files
[certificates] Generated ca certificate and key.
2018/06/29 18:58:40 creating a new server certificate and key files for etcd
[certificates] Generated server certificate and key.
[certificates] server serving cert is signed for DNS names [coreos-10-4-252-159platform9.sys] and IPs [127.0.0.1 172.16.0.49]
2018/06/29 18:58:40 creating a new certificate and key files for etcd peering
[certificates] Generated peer certificate and key.
[certificates] peer serving cert is signed for DNS names [coreos-10-4-252-159platform9.sys] and IPs [172.16.0.49]
2018/06/29 18:58:40 creating a new client certificate for the etcdctl
[certificates] Generated etcdctl-etcd-client certificate and key.
2018/06/29 18:58:41 creating a new client certificate for the apiserver calling etcd
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/etcd/pki"
2018/06/29 18:58:46 context deadline exceeded

I then ran etcdadm reset, which failed, presumably because it could not reach the local etcd endpoint:

# etcdadm reset
2018/06/29 18:59:51 dial tcp 127.0.0.1:2379: getsockopt: connection refused

Handle single node etcdadm reset gracefully (Special case)

When calling etcdadm reset on a single node etcd cluster, etcdadm tries to delete the local member from the etcd cluster. Etcd complains when etcdadm tries to make this call, since this member is the only member.

single node etcd cluster needs special handling.

cache and install directory should be executable and readable by all

$ stat /var/cache/etcdadm/etcd-v3.3.8/
  File: '/var/cache/etcdadm/etcd-v3.3.8/'
  Size: 4096            Blocks: 16         IO Block: 4096   directory
Device: fd09h/64777d    Inode: 201         Links: 2
Access: (0700/drwx------)  Uid: (  755/ UNKNOWN)   Gid: (    0/    root)
Context: system_u:object_r:var_t:s0
Access: 2018-07-25 18:38:22.000000000 +0000
Modify: 2018-07-25 18:31:04.000000000 +0000
Change: 2018-07-25 18:38:22.000000000 +0000
 Birth: -

$ stat /opt/bin/etcd-v3.3.8
  File: '/opt/bin/etcd-v3.3.8'
  Size: 4096            Blocks: 16         IO Block: 4096   directory
Device: fd09h/64777d    Inode: 1563654     Links: 3
Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:unlabeled_t:s0
Access: 2018-07-29 02:27:30.000000000 +0000
Modify: 2018-07-29 02:27:30.000000000 +0000
Change: 2018-07-29 02:27:30.000000000 +0000
 Birth: -

etcdadm join must require existing CA certs

Join generates CA certs, but cannot work, and the error is mysterious:

# etcdadm join https://10.105.16.48:2379
2018/08/02 20:04:54 [install] extracting etcd archive /var/cache/etcdadm/etcd-v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz to /opt/bin/etcd-v3.3.8
2018/08/02 20:04:55 [install] verifying etcd 3.3.8 is installed in /opt/bin/etcd-v3.3.8
2018/08/02 20:04:55 [certificates] creating PKI assets
2018/08/02 20:04:55 creating a self signed etcd CA certificate and key files
[certificates] Generated ca certificate and key.
2018/08/02 20:04:55 creating a new server certificate and key files for etcd
[certificates] Generated server certificate and key.
[certificates] server serving cert is signed for DNS names [coreos-daniel-157-10-105-16-46platform9.sys] and IPs [10.105.16.46 127.0.0.1]
2018/08/02 20:04:56 creating a new certificate and key files for etcd peering
[certificates] Generated peer certificate and key.
[certificates] peer serving cert is signed for DNS names [coreos-daniel-157-10-105-16-46platform9.sys] and IPs [10.105.16.46]
2018/08/02 20:04:56 creating a new client certificate for the etcdctl
[certificates] Generated etcdctl-etcd-client certificate and key.
2018/08/02 20:04:57 creating a new client certificate for the apiserver calling etcd
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/etcd/pki"
ls -l /etc/etcd/2018/08/02 20:05:08 [cluster] Error: failed to list cluster members: [cluster] Error: etcdclient failed to connect: context deadline exceeded

Update etcd version

How to update etcd form 3.3.8 to 3.3.10 using etcdadm?
Tried updating version in /constants/constants.go and /vendor/github.com/coreos/etcd/version/version.go. Still installing 3.3.8.

Refactor Reset command implementation

Code can be moved out into new functions:

• certs.DestroyPKIAssets
• service.DeleteUnitFile
• service.DeleteEnvironmentFile
• service.DeleteEtcdctlEnvFile

These complement existing functions.

etdctl init/join fails if the install directory does not exist

root@daniel-ubuntu16:~# ./etcdadm init
2018-09-27 23:56:12.277010 I | [install] Removing existing data dir "/var/lib/etcd"
2018-09-27 23:56:12.277462 I | [install] Artifact not found in cache. Trying to fetch from upstream: %!s(<nil>)
2018-09-27 23:56:12.277558 I | [install] Downloading & installing etcd https://github.com/coreos/etcd/releases/download from 3.3.8 to /var/cache/etcdadm/etcd/v3.3.8                                                                  
2018-09-27 23:56:12.277822 I | [install] downloading etcd from https://github.com/coreos/etcd/releases/download/v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz to /var/cache/etcdadm/etcd/v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz                
######################################################################## 100.0% 
2018-09-27 23:56:43.166607 I | [install] extracting etcd archive /var/cache/etcdadm/etcd/v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz to /tmp/etcd029406230                                                                                  
2018-09-27 23:56:43.557369 I | [install] Artifact could not be installed from cache: unable to copy binaries: exit status 1   

Reset must set dynamic defaults

download should abandon if the throughput drops for a sufficient period

The download should detect a "hung" transfer and abandon more quickly than it does:

[19:07:02][Step 3/4] #####################################################################     96.6%
[19:07:02][Step 3/4] #####################################################################     96.8%
[19:10:12][Step 3/4] #####################################################################     96.9%
[19:10:12][Step 3/4] curl: (56) TCP connection reset by peer
[19:10:12][Step 3/4] 2018-08-30 02:10:12.558539 I | [download] Error: unable to download etcd: exit status 56

`etcdadm init` does not set the initial cluster

This is a bug, but benign, because the initial cluster etcd sets by default happens to be the same as the one etcdadm init would set.

Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys systemd[1]: Starting etcd...
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://10.105.16.6:2379
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_CERT_FILE=/etc/etcd/pki/server.crt
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.105.16.6:2380
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=69c9e5d2
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_KEY_FILE=/etc/etcd/pki/server.key
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://10.105.16.6:2379,https://127.0.0.1:2379
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://10.105.16.6:2380
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_NAME=coreos-daniel-157-10-105-16-6platform9.sys
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/etcd/pki/peer.crt
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/etcd/pki/peer.key
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_STRICT_RECONFIG_CHECK=true
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: unrecognized environment variable ETCD_INITIAL_CLUSTER=
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: etcd Version: 3.3.8
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: Git SHA: 33245c6b5
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: Go Version: go1.9.7
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: Go OS/Arch: linux/amd64
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: setting maximum number of CPUs to 2, total number of available CPUs is 2
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: peerTLS: cert = /etc/etcd/pki/peer.crt, key = /etc/etcd/pki/peer.key, ca = , trusted-ca = /etc/etcd/pki/ca.crt, client-cert-auth = true, crl-file =
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: listening for peers on https://10.105.16.6:2380
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: listening for client requests on 10.105.16.6:2379
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: listening for client requests on 127.0.0.1:2379
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: name = coreos-daniel-157-10-105-16-6platform9.sys
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: data dir = /var/lib/etcd
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: member dir = /var/lib/etcd/member
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: heartbeat = 100ms
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: election = 1000ms
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: snapshot count = 100000
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: advertise client URLs = https://10.105.16.6:2379
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: initial advertise peer URLs = https://10.105.16.6:2380
Aug 03 20:24:53 coreos-daniel-157-10-105-16-6platform9.sys etcd[15217]: initial cluster = coreos-daniel-157-10-105-16-6platform9.sys=https://10.105.16.6:2380

Add etcd info command

Print the endpoints (client URLs) of the local etcd service

$ etcdadm info
{
  "ID": "6cf60d45a7315a44",
  "name":"coreos-10-4-252-52platform9.sys",
  "peerURLs": [ "https://172.16.0.84:2380" ],
  "clientURLs": [ "https://172.16.0.84:2379" ],
  "status": "started",
  "health": "healthy"
}

Print the etcdadm join command to run to join the cluster

$ etcdadm info --print-join-command
Copy the CA (/etc/etcd/pki/ca.crt, /etc/etcd/pki/ca.key) to the joining host. Then run:
    etcdadm join https://172.16.0.84:2379

Remove unwanted directory footprint

etcdadm seems to leave behind a few directories under /root/ for some reason

# ls
etcd.env  etcd.service  etcdctl.env

I believe these are directories that are used as staging when downloading the binaries and needs to be removed.

Impose timeout on curl

Use --connect-timeout=10s

Make cache directory configurable

Document use without internet egress

etcdadm init fails if `ETCDCTL_API=3` is defined

isEtcdctlInstalled calls etcdctl --version, but etcdctl version must be used for the v3 API.

Add flag to specify listening address.

This is necessary for environments like Vagrant that have multiple network interfaces.

if I could specify the ip that is used to resolve the external address this would solve this.

Reduce cache size by storing artifacts as tar's instead of inflated contents

etcdadm download command downloads binaries from upstream as a tarball. This tarball is unzipped and stored in the cache directory /var/cache/etcdadm/etcd-<version>/. The cache can be made efficient by storing things as tarballs rather than expanded binaries & files.

Move InitialClusterTokenDeclaration out of config

PR #21 creates the declaration as part of the config, but the declaration is really specific to writing the environment file.

Use template action instead?

{{if .InitialClusterToken}}
ETCD_INITIAL_CLUSTER_TOKEN={{ .InitialClusterToken }}
{{end}}

Apply connect timeout to etcd as well as download

If etcd cluster has lost quorum, `etcdadm join` segfaults

2018/08/16 20:55:21 running "/opt/bin/etcdadm join https://10.105.16.46:2379" command. This might take some time..
2018/08/16 20:55:24 [certificates] Using the existing ca certificate and key.
[certificates] Generated server certificate and key.
[certificates] server serving cert is signed for DNS names [coreos-daniel-268-10-105-16-69platform9.sys] and IPs [127.0.0.1 10.105.16.69]
[certificates] Generated peer certificate and key.
[certificates] peer serving cert is signed for DNS names [coreos-daniel-268-10-105-16-69platform9.sys] and IPs [10.105.16.69]
[certificates] Generated etcdctl-etcd-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/etcd/pki"

2018/08/16 20:55:24 2018-08-16 20:56:41.825736 I | [install] extracting etcd archive /var/cache/etcdadm/etcd/v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz to /tmp/etcd425626077
2018-08-16 20:56:42.512163 I | [install] verifying etcd 3.3.8 is installed in /opt/bin/
2018-08-16 20:56:42.556884 I | [certificates] creating PKI assets
2018-08-16 20:56:42.556933 I | creating a self signed etcd CA certificate and key files
2018-08-16 20:56:43.308801 I | creating a new server certificate and key files for etcd
2018-08-16 20:56:43.690654 I | creating a new certificate and key files for etcd peering
2018-08-16 20:56:44.711776 I | creating a new client certificate for the etcdctl
2018-08-16 20:56:44.935711 I | creating a new client certificate for the apiserver calling etcd
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xb9544e]

goroutine 1 [running]:
github.com/platform9/etcdadm/cmd.glob..func5(0x12e86a0, 0xc42021ed20, 0x1, 0x1)
        /go/src/github.com/platform9/etcdadm/cmd/join.go:83 +0x57e
github.com/platform9/etcdadm/vendor/github.com/spf13/cobra.(*Command).execute(0x12e86a0, 0xc42021ecf0, 0x1, 0x1, 0x12e86a0, 0xc42021ecf0)
        /go/src/github.com/platform9/etcdadm/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/platform9/etcdadm/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x12e8b60, 0xc420233f58, 0xb93faf, 0xc42012e900)
        /go/src/github.com/platform9/etcdadm/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/platform9/etcdadm/vendor/github.com/spf13/cobra.(*Command).Execute(0x12e8b60, 0xc42008c058, 0x0)
        /go/src/github.com/platform9/etcdadm/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/platform9/etcdadm/cmd.Execute()
        /go/src/github.com/platform9/etcdadm/cmd/root.go:21 +0x2d
main.main()
        /go/src/github.com/platform9/etcdadm/main.go:10 +0x20

Use local file system cache for all the downloaded artifacts.

Update etcdadm to download all the artifacts in a local filesystem cache e.g. /var/cache/etcdadm/<version.
This cache would act as a representative state of any disk footprint that would be created by etcdadm.
This would allow reset to clean all the footprint created by init while still allowing calling init again later, even in an airgaped env.

Reset fails on one-member cluster

# etcdadm reset
2018/06/29 20:55:00 [cluster] Error: failed to remove self from etcd cluster: etcdserver: re-configuration failed due to not enough started members

Placeholder for README.md work

Add a README document