saltstack-formulas / etcd-formula

etcd

Formula to install and configure etcd on Linux and MacOS.

Note

See the full Salt Formulas installation and usage instructions. Refer to pillar.example and defaults.yaml for configurable values.

Available states

• etcd
• etcd.install
• etcd.linuxenv
• etcd.service
• etcd.service.stopped
• etcd.docker.running
• etcd.docker.stopped

etcd

Metastate for all the software and service deployment states.

etcd.install

Installs etcd version by downloading the archive from github and extracting it to the {{ etcd.prefix }} directory. Includes etcd.service by default (unless etcd.docker.enabled is True).

etcd.linuxenv

Configure launchd, systemd, or upstart configuration, this will copy. Setup linux alternatives if enabled and supported.

Note

Enable linux alternatives by setting nonzero 'altpriority' pillar value; otherwise feature is disabled.

etcd.service

Ensure the etcd service is registered and (re)started on minion.

etcd.service.stopped

Ensure the etcd service is stopped on minion.

etcd.docker.running

Ensure that an etcd container with a specific configuration is present and running. Requires Docker to be running - you could run docker state from docker-formula to ensure this requirement.

State top file:

base:
  '*':
    - docker                  #saltstack-formulas/docker-formula
    - etcd.docker.running     #saltstack-formulas/etcd-formula

etcd.docker.stopped

Ensure that etcd container is stopped

etcd-formula's Issues

docker container start iptables error (Centos)

Getting this error on Centos7 for docker.running state - not sure why.

ID: run-etcd-dockerized-service
    Function: docker_container.running
        Name: osdsdb-etcd
      Result: False
     Comment: Created container 'osdsdb-etcd'. Failed to start container 'osdsdb-etcd': 'Error 500: driver failed programming external connectivity on endpoint osdsdb-etcd (db78c9f109a92fd93291c6e1788f7d582fe9509ad1e2f73909fc69b0be87865d):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.3 --dport 2380 -j ACCEPT: iptables: No chain/target/match by that name.
               (exit status 1))'.
     Started: 22:51:45.463321
    Duration: 17193.464 ms
     Changes:   

Centos package manager issue (formula rerun)

Noting some weird package manager issue on Centos 7 on formula rerun.

----------
          ID: etcd-docker-python2-docker-package
    Function: pkg.installed
        Name: python2-docker
      Result: False
     Comment: Error occurred installing package(s). Additional info follows:
              
              errors:
                  - Running scope as unit run-6142.scope.
                    Loaded plugins: fastestmirror, versionlock
                    Loading mirror speeds from cached hostfile
                     * base: mirror.strencom.net
                     * epel: ftp.nluug.nl
                     * extras: mirror.strencom.net
                     * updates: mirror.strencom.net
                    Resolving Dependencies
                    --> Running transaction check
                    ---> Package python2-docker.noarch 0:2.4.2-1.2.el7 will be installed
                    --> Processing Dependency: python2-six >= 1.4.0 for package: python2-docker-2.4.2-1.2.el7.noarch
                    --> Processing Dependency: python-websocket-client >= 0.32.0 for package: python2-docker-2.4.2-1.2.el7.noarch
                    --> Processing Dependency: python2-requests for package: python2-docker-2.4.2-1.2.el7.noarch
                    --> Processing Dependency: python-docker-pycreds for package: python2-docker-2.4.2-1.2.el7.noarch
                    --> Running transaction check
                    ---> Package python-requests.noarch 0:2.6.0-1.el7_1 will be obsoleted
                    ---> Package python-six.noarch 0:1.9.0-2.el7 will be obsoleted
                    ---> Package python-websocket-client.noarch 0:0.34.0-3.el7 will be installed
                    ---> Package python2-docker-pycreds.noarch 0:0.2.1-4.el7 will be installed
                    ---> Package python2-requests.noarch 0:2.14.2-1.el7 will be obsoleting
                    --> Processing Dependency: python2-urllib3 = 1.21.1 for package: python2-requests-2.14.2-1.el7.noarch
                    ---> Package python2-six.noarch 0:1.10.0-9.el7 will be obsoleting
                    --> Running transaction check
                    ---> Package python-urllib3.noarch 0:1.10.2-5.el7 will be obsoleted
                    ---> Package python2-urllib3.noarch 0:1.21.1-1.el7 will be obsoleting
                    --> Processing Dependency: python-pysocks for package: python2-urllib3-1.21.1-1.el7.noarch
                    --> Running transaction check
                    ---> Package python2-pysocks.noarch 0:1.6.8-5.el7 will be installed
                    --> Finished Dependency Resolution
                    
                    Dependencies Resolved
                    
                    ================================================================================
                     Package                   Arch     Version            Repository          Size
                    ================================================================================
                    Installing:
                     python2-docker            noarch   2.4.2-1.2.el7      openstack-queens   168 k
                     python2-requests          noarch   2.14.2-1.el7       openstack-queens   116 k
                         replacing  python-requests.noarch 2.6.0-1.el7_1
                     python2-six               noarch   1.10.0-9.el7       openstack-queens    31 k
                         replacing  python-six.noarch 1.9.0-2.el7
                     python2-urllib3           noarch   1.21.1-1.el7       openstack-queens   173 k
                         replacing  python-urllib3.noarch 1.10.2-5.el7
                    Installing for dependencies:
                     python-websocket-client   noarch   0.34.0-3.el7       openstack-queens    57 k
                     python2-docker-pycreds    noarch   0.2.1-4.el7        openstack-queens    15 k
                     python2-pysocks           noarch   1.6.8-5.el7        epel                30 k
                    
                    Transaction Summary
                    ================================================================================
                    Install  4 Packages (+3 Dependent packages)
                    
                    Total download size: 589 k
                    Downloading packages:
                    --------------------------------------------------------------------------------
                    Total                                              414 kB/s | 589 kB  00:01     
                    Running transaction check
                    Running transaction test
                    Transaction test succeeded
                    Running transaction
                      Installing : python2-six-1.10.0-9.el7.noarch                             1/10 
                      Installing : python-websocket-client-0.34.0-3.el7.noarch                 2/10 
                      Installing : python2-docker-pycreds-0.2.1-4.el7.noarch                   3/10 
                      Installing : python2-pysocks-1.6.8-5.el7.noarch                          4/10 
                      Installing : python2-urllib3-1.21.1-1.el7.noarch                         5/10Error unpacking rpm package python2-urllib3-1.21.1-1.el7.noarch
                     
                    error: unpacking of archive failed on file /usr/lib/python2.7/site-packages/urllib3/packages/ssl_match_hostname: cpio: rename
                      Installing : python2-requests-2.14.2-1.el7.noarch                        6/10 
                    error: python2-urllib3-1.21.1-1.el7.noarch: install failed
                      Installing : python2-docker-2.4.2-1.2.el7.noarch                         7/10 
                      Erasing    : python-requests-2.6.0-1.el7_1.noarch                        8/10 
                    warning: file /usr/lib/python2.7/site-packages/requests-2.6.0-py2.7.egg-info: remove failed: No such file or directory
                      Erasing    : python-six-1.9.0-2.el7.noarch                               9/10 
                    error: python-urllib3-1.10.2-5.el7.noarch: erase skipped
                    warning: file /usr/lib/python2.7/site-packages/six-1.9.0-py2.7.egg-info/top_level.txt: remove failed: No such file or directory
                    warning: file /usr/lib/python2.7/site-packages/six-1.9.0-py2.7.egg-info/dependency_links.txt: remove failed: No such file or directory
                    warning: file /usr/lib/python2.7/site-packages/six-1.9.0-py2.7.egg-info/SOURCES.txt: remove failed: No such file or directory
                    warning: file /usr/lib/python2.7/site-packages/six-1.9.0-py2.7.egg-info/PKG-INFO: remove failed: No such file or directory
                    warning: file /usr/lib/python2.7/site-packages/six-1.9.0-py2.7.egg-info: remove failed: No such file or directory
                      Verifying  : python2-pysocks-1.6.8-5.el7.noarch                          1/10 
                      Verifying  : python-websocket-client-0.34.0-3.el7.noarch                 2/10 
                      Verifying  : python2-requests-2.14.2-1.el7.noarch                        3/10 
                      Verifying  : python2-six-1.10.0-9.el7.noarch                             4/10 
                      Verifying  : python2-docker-2.4.2-1.2.el7.noarch                         5/10 
                      Verifying  : python2-docker-pycreds-0.2.1-4.el7.noarch                   6/10python-urllib3-1.10.2-5.el7.noarch was supposed to be removed but is not!
                     
                      Verifying  : python-urllib3-1.10.2-5.el7.noarch                          7/10 
                      Verifying  : python2-urllib3-1.21.1-1.el7.noarch                         8/10 
                      Verifying  : python-requests-2.6.0-1.el7_1.noarch                        9/10 
                      Verifying  : python-six-1.9.0-2.el7.noarch                              10/10 
                    
                    Installed:
                      python2-docker.noarch 0:2.4.2-1.2.el7  python2-requests.noarch 0:2.14.2-1.el7 
                      python2-six.noarch 0:1.10.0-9.el7     
                    
                    Dependency Installed:
                      python-websocket-client.noarch 0:0.34.0-3.el7                                 
                      python2-docker-pycreds.noarch 0:0.2.1-4.el7                                   
                      python2-pysocks.noarch 0:1.6.8-5.el7                                          
                    
                    Replaced:
                      python-requests.noarch 0:2.6.0-1.el7_1     python-six.noarch 0:1.9.0-2.el7    
                    
                    Failed:
                      python-urllib3.noarch 0:1.10.2-5.el7   python2-urllib3.noarch 0:1.21.1-1.el7  
                    
                    Complete!
     Started: 21:09:13.903760
    Duration: 9201.153 ms
     Changes:   
              ----------
              python-requests:
                  ----------
                  new:
                  old:
                      2.6.0-1.el7_1
              python-six:
                  ----------
                  new:
                  old:
                      1.9.0-2.el7
              python-websocket-client:
                  ----------
                  new:
                      0.34.0-3.el7
                  old:
              python2-docker:
                  ----------
                  new:
                      2.4.2-1.2.el7
                  old:
              python2-docker-pycreds:
                  ----------
                  new:
                      0.2.1-4.el7
                  old:
              python2-pysocks:
                  ----------
                  new:
                      1.6.8-5.el7
                  old:
              python2-requests:
                  ----------
                  new:
                      2.14.2-1.el7
                  old:
              python2-six:
                  ----------
                  new:
                      1.10.0-9.el7
                  old:
----------
          ID: etcd-docker-python-pip-install
    Function: pip.installed
        Name: docker
      Result: True
     Comment: Python package docker was already installed
              All specified packages are already installed
     Started: 21:09:24.055855
    Duration: 579.527 ms
     Changes:   
----------
          ID: etcd-ensure-docker-service
    Function: service.running
        Name: docker
      Result: True
     Comment: The service docker is already running
     Started: 21:09:24.637415
    Duration: 30.214 ms
     Changes:   
----------
          ID: run-etcd-dockerized-service
    Function: docker_container.running
      Result: False
     Comment: One or more requisite failed: etcd.docker.running.etcd-docker-python2-docker-package
     Started: 21:09:24.673146
    Duration: 0.032 ms
    

pip install python-urllib3 fails on Cent7

• pipconflictchecker also reports conflict

[root@selenium-node opensds-linux-amd64-devstack]# pipconflictchecker 
--------------------------------------------------
 Conflicts Detected
--------------------------------------------------
 - requests(2.19.1) docker-compose(<2.19,!=2.18.0,!=2.11.0,!=2.12.2,>=2.6.1)
 - pyudev(0.15) rtslib-fb(>=0.16.1)

• installing python2-docker on Cent7 fails (https://bugzilla.redhat.com/show_bug.cgi?id=1187057)

[ERROR   ] Command '[u'systemd-run', u'--scope', u'yum', u'-y', u'install', u'python2-docker']' failed with return code: 1
[ERROR   ] stdout: Running scope as unit run-8416.scope.
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
 * base: repo.uk.bigstepcloud.com
 * epel: mirror.bytemark.co.uk
 * extras: mirror.strencom.net
 * updates: mirror.strencom.net
Resolving Dependencies
--> Running transaction check
---> Package python2-docker.noarch 0:2.4.2-1.2.el7 will be installed
--> Processing Dependency: python-websocket-client >= 0.32.0 for package: python2-docker-2.4.2-1.2.el7.noarch
--> Processing Dependency: python2-requests for package: python2-docker-2.4.2-1.2.el7.noarch
--> Processing Dependency: python-docker-pycreds for package: python2-docker-2.4.2-1.2.el7.noarch
--> Running transaction check
---> Package python-requests.noarch 0:2.6.0-1.el7_1 will be obsoleted
---> Package python-websocket-client.noarch 0:0.34.0-3.el7 will be installed
---> Package python2-docker-pycreds.noarch 0:0.2.1-4.el7 will be installed
---> Package python2-requests.noarch 0:2.14.2-1.el7 will be obsoleting
--> Processing Dependency: python2-urllib3 = 1.21.1 for package: python2-requests-2.14.2-1.el7.noarch
--> Running transaction check
---> Package python-urllib3.noarch 0:1.10.2-5.el7 will be obsoleted
---> Package python2-urllib3.noarch 0:1.21.1-1.el7 will be obsoleting
--> Processing Dependency: python-pysocks for package: python2-urllib3-1.21.1-1.el7.noarch
--> Running transaction check
---> Package python2-pysocks.noarch 0:1.6.7-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                   Arch     Version            Repository          Size
================================================================================
Installing:
 python2-docker            noarch   2.4.2-1.2.el7      openstack-queens   168 k
 python2-requests          noarch   2.14.2-1.el7       openstack-queens   116 k
     replacing  python-requests.noarch 2.6.0-1.el7_1
 python2-urllib3           noarch   1.21.1-1.el7       openstack-queens   173 k
     replacing  python-urllib3.noarch 1.10.2-5.el7
Installing for dependencies:
 python-websocket-client   noarch   0.34.0-3.el7       openstack-queens    57 k
 python2-docker-pycreds    noarch   0.2.1-4.el7        openstack-queens    15 k
 python2-pysocks           noarch   1.6.7-1.el7        epel                29 k

Transaction Summary
================================================================================
Install  3 Packages (+3 Dependent packages)

Total download size: 557 k
Downloading packages:
--------------------------------------------------------------------------------
Total                                              391 kB/s | 557 kB  00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : python2-docker-pycreds-0.2.1-4.el7.noarch                    1/8 
  Installing : python-websocket-client-0.34.0-3.el7.noarch                  2/8 
  Installing : python2-pysocks-1.6.7-1.el7.noarch                           3/8 
  Installing : python2-urllib3-1.21.1-1.el7.noarch                          4/8Error unpacking rpm package python2-urllib3-1.21.1-1.el7.noarch
 
error: unpacking of archive failed on file /usr/lib/python2.7/site-packages/urllib3/packages/ssl_match_hostname: cpio: rename
  Installing : python2-requests-2.14.2-1.el7.noarch                         5/8 
error: python2-urllib3-1.21.1-1.el7.noarch: install failed
  Installing : python2-docker-2.4.2-1.2.el7.noarch                          6/8 
  Erasing    : python-requests-2.6.0-1.el7_1.noarch                         7/8 
warning: file /usr/lib/python2.7/site-packages/requests-2.6.0-py2.7.egg-info: remove failed: No such file or directory
  Verifying  : python2-pysocks-1.6.7-1.el7.noarch                           1/8 
  Verifying  : python-websocket-client-0.34.0-3.el7.noarch                  2/8 
  Verifying  : python2-requests-2.14.2-1.el7.noarch                         3/8 
  Verifying  : python2-docker-2.4.2-1.2.el7.noarch                          4/8 
  Verifying  : python2-docker-pycreds-0.2.1-4.el7.noarch                    5/8python-urllib3-1.10.2-5.el7.noarch was supposed to be removed but is not!
 
  Verifying  : python-urllib3-1.10.2-5.el7.noarch                           6/8 
  Verifying  : python2-urllib3-1.21.1-1.el7.noarch                          7/8 
  Verifying  : python-requests-2.6.0-1.el7_1.noarch                         8/8 

Installed:
  python2-docker.noarch 0:2.4.2-1.2.el7  python2-requests.noarch 0:2.14.2-1.el7 

Dependency Installed:
  python-websocket-client.noarch 0:0.34.0-3.el7                                 
  python2-docker-pycreds.noarch 0:0.2.1-4.el7                                   
  python2-pysocks.noarch 0:1.6.7-1.el7                                          

Replaced:
  python-requests.noarch 0:2.6.0-1.el7_1                                        

Failed:
  python-urllib3.noarch 0:1.10.2-5.el7   python2-urllib3.noarch 0:1.21.1-1.el7  

....

[BUG] 'gid_from_name' is an invalid keyword argument for 'user.present'

Your setup

Formula commit hash / release tag

Using the latest commit at time of writing: d4c849a

Versions reports (master & minion)

The Salt master is version 3000.3 and the minion is 3001. From the master:

Salt Version:
           Salt: 3000.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Apr 18 2020, 01:56:04)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 5.4.0-40-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Pillar / config used

I'm basically using the example pillar data. I'm just starting out and don't know what I want yet. :-)

etcd:
  version: 3.2.24
  arch: {{ grains["osarch"] }}
  prefix: /usr/local
  realhome: /usr/local/etcd-v3.2.24-linux-{{ grains["osarch"] }}
  user: etcd
  use_upstream_repo: true
  service:
    name: etcd0
    data_dir: /var/lib/etcd/etcd0
    initial_cluster: 'etcd0=https://0.0.0.0:2380'
    initial_cluster_state: new
    initial_cluster_token: etcd-cluster-1
    initial_advertise_peer_urls: 'https://0.0.0.0:2380'
    listen_peer_urls: 'https://0.0.0.0:2380'
    listen_client_urls: 'https://0.0.0.0:2379'
    advertise_client_urls: 'https://0.0.0.0:2379'
    cert_file: /etc/ssl/etcd/ca.pem
    key_file: /etc/ssl/etcd/client.pem
    ca_file: /etc/ssl/etcd/client-key.pem
    cmd_args: '--auto-tls --peer-auto-tls'
    auto_tls: True
    peer_auto_tls: True

---

Bug details

Describe the bug

It seems that the use of "gid_from_name" in the context of "user.present" has been replaced with "usergroup" in the latest (3001) version of SaltStack. The state provided here in "etcd/install.sls" uses the older "gid_from_name" (line 19). This causes the state to fail to apply. The resulting error is:

ERROR executing 'state.apply': 'gid_from_name' is an invalid keyword argument for 'user.present'

Steps to reproduce the bug

Expected behaviour

Applying the state should work.

Attempts to fix the bug

I have forked this repo and changed line 19 in "etcd/install.sls" to read:

    - usergroup: True

I then removed the "group.present" stanza. usergroup: True automatically creates the group but if the group already exists, it fails to create the user. Then I removed a few requires that were looking for the "group.present" that I removed.

My version can be found here: https://github.com/ShaunMaher/etcd-formula/blob/master/etcd/install.sls

This resolves the issue but, I'm not sure if this will break older versions of Salt.

Additional context

support etcd.install only (without service handling)

It is not possible to install binaries only (for etcd client only).

Theinstall.sls SLS references service.sls via includes and watch statements. The init.sls also references services.sls via extend.

Cannot bind etcd on a specific IP

Hello :)

Summary

Etcd cannot be bind to a specific IP. Default value (127.0.0.1) cannot be overloaded by my pillars.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      19470/etcd
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      19470/etcd

Details

Running on ubuntu 16.04

Pillars examples

(Copy of pillar.example)

etcd:
  lookup:
    version: 3.2.18

  service:
    name: etcd0
    data_dir: /var/lib/etcd/etcd0
    initial_cluster: etcd0=https://0.0.0.0:2380
    initial_cluster_state: new
    initial_cluster_token: etcd-cluster-1
    initial_advertise_peer_urls: https://0.0.0.0:2380
    listen_peer_urls: https://0.0.0.0:2380
    listen_client_urls: https://0.0.0.0:2379
    advertise_client_urls: https://0.0.0.0:2379
    auto_tls: '"false"'
    peer_auto_tls: '"false"'
    proxy: OFF

Expected behaviour

ETCD should be listening to *:2379 and *2380

Possible fix

Issue seems to be related to this line. etcd.lookup doesn't contain etcd listening configuration.

Thanks :)

docker_container __virtual__ returned False [Ubuntu]

          ID: run-etcd-dockerized-service
    Function: docker_container.running
        Name: osdsdb
      Result: False
     Comment: State 'docker_container.running' was not found in SLS 'etcd.docker.running'
              Reason: 'docker_container' __virtual__ returned False: 'docker.version' is not available.
     Changes:


vagrant@ubuntu1804:~$ salt-call --versions
Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 8.9.1
       dateutil: 2.6.1
      docker-py: 2.5.1
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.7 (default, Oct 22 2018, 11:32:17)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ISO-8859-1
        machine: x86_64
        release: 4.15.0-45-generic
         system: Linux
        version: Ubuntu 18.04 bionic