sous-chefs / etcd

Development repository for the etcd cookbook

Etcd Cookbook

The Etcd Cookbook is a library cookbook that provides custom resources for use in recipes.

Scope

This cookbook is concerned with the Etcd distributed key/value store as distributed by CoreOS, Inc.

Maintainers

This cookbook is maintained by the Sous Chefs. The Sous Chefs are a community of Chef cookbook maintainers working together to maintain important cookbooks. If you’d like to know more please visit sous-chefs.org or come chat with us on the Chef Community Slack in #sous-chefs.

Requirements

• Chef Infra Client 16+
• Network accessible web server hosting the etcd binary.

Platform Support

The following platforms have been tested with Test Kitchen. It will most likely work on other platforms as well

|---------------+--------+
|               | 3.2.15 |
|---------------+--------+
| amazonlinux 2 |   X    |
|---------------+--------+
| centos-7      |   X    |
|---------------+--------+
| debian-9      |   X    |
|---------------+--------+
| debian-10     |   X    |
|---------------+--------+
| fedora        |   X    |
|---------------+--------+
| ubuntu-16.04  |   X    |
|---------------+--------+
| ubuntu-18.04  |   X    |
|---------------+--------+
| ubuntu-20.04  |   X    |
|---------------+--------+
| opensuse-leap |   X    |
|---------------+--------+

Cookbook Dependencies

• docker

Usage

• Add depends 'etcd' to your cookbook's metadata.rb
• Use the resources shipped in cookbook in a recipe, the same way you'd use core Chef resources (file, template, directory, package, etc).

etcd_service 'etcd0' do
  advertise_client_urls 'http://127.0.0.1:2379'
  listen_client_urls 'http://0.0.0.0:2379'
  initial_advertise_peer_urls 'http://127.0.0.1:2380'
  listen_peer_urls 'http://0.0.0.0:2380'
  initial_cluster_token 'etcd-cluster-1'
  initial_cluster 'etcd0=http://127.0.0.1:2380,etcd1=http://127.0.0.1:3380,etcd2=http://127.0.0.1:4380'
  initial_cluster_state 'new'
  action :start
  ignore_failure true # required for the first cluster build
end

etcd_service 'etcd1' do
  advertise_client_urls 'http://127.0.0.1:3379'
  listen_client_urls 'http://0.0.0.0:3379'
  initial_advertise_peer_urls 'http://127.0.0.1:3380'
  listen_peer_urls 'http://0.0.0.0:3380'
  initial_cluster_token 'etcd-cluster-1'
  initial_cluster 'etcd0=http://127.0.0.1:2380,etcd1=http://127.0.0.1:3380,etcd2=http://127.0.0.1:4380'
  initial_cluster_state 'new'
  action :start
  ignore_failure true
end

etcd_service 'etcd2' do
  advertise_client_urls 'http://127.0.0.1:4379'
  listen_client_urls 'http://0.0.0.0:4379'
  initial_advertise_peer_urls 'http://127.0.0.1:4380'
  listen_peer_urls 'http://0.0.0.0:4380'
  initial_cluster_token 'etcd-cluster-1'
  initial_cluster 'etcd0=http://127.0.0.1:2380,etcd1=http://127.0.0.1:3380,etcd2=http://127.0.0.1:4380'
  initial_cluster_state 'new'
  action :start
  ignore_failure true
end

By default reosource creates etcd-NODE_NAME unit file name. Sometimes it's not comfortable.

If you don't run multi etcd service in node, you can change this action to default service name.

etcd_service 'etcd' do
  action :start
  default_service_name true
...
end

Test Cookbooks as Examples

The cookbooks ran under test-kitchen make excellent usage examples.

The test recipes are found at:

test/cookbooks/etcd_test/

Resources Overview

• etcd_service: composite resource that uses etcd_installation and etcd_service_manager
• etcd_installation: automatically selects an installation method
• etcd_service_manager: automatically selects a service manager
• etcd_key: manages keys in etcd
• etcd_installation_binary: copies a pre-compiled etcd binary onto disk
• etcd_installation_docker: pulls a docker image to the DOCKER_HOST
• etcd_service_manager_systemd: manage etcd daemon with systemd unit files
• etcd_service_manager_docker: starts a docker process on the DOCKER_HOST

Resources Details

etcd_installation

The etcd_installation resource auto-selects one of the below resources with the provider resolution system. Defaults to binary installation.

Example

etcd_installation 'default' do
  action :create
end

etcd_installation_binary

The etcd_installation_binary resource copies the precompiled Go binary onto the disk.

Example

etcd_installation_binary 'default' do
  version '3.2.6'
  source 'https://my.computers.biz/dist/etcd'
  checksum '90aff7364caa43932fd46974825af20e0ecb70fe7e01981e2d3a496106f147e7'
  action :create
end

etcd_installation_docker

The etcd_installation_docker resource uses the docker_image resource to pull an image to the DOCKER_HOST.

Properties

• repo - The image name to pull. Defaults to 'quay.io/coreos/etcd'
• tag - The image tag to pull.
• version - String used to calculate tag string when tag is omitted. Defaults to '2.3.7'

etcd_service_manager

The etcd_service_manager resource auto-selects one of the below resources with the provider resolution system. The etcd_service family all share a common set of properties, which are listed under the etcd_service composite resource.

Warning

etcd startup behavior is a bit quirky. etcd loops indefinitely on startup until quorum can be established. Due to this the first nodes service start will fail unless all nodes come up at the same time. Due to this there is an ignore_failure property for the systemd service managers which allows you to continue on in the chef run if the service fails to start. systemd will automatically keep restarting the service until all nodes are up and the cluster is healthy. For sys-v init you're on your own.

Example

etcd_service_manager 'default' do
  action :start
end

properties

• ignore_failure - Ignore failures starting the etcd service. Before quorum is established nodes will loop indefinitely and never successfully start. This can help ensure all instances are up when init systems can handle restart on failure. Default: false

etcd_service_manager_systemd

Example

etcd_service_manager_systemd 'default' do
  action :start
end

properties

• service_timeout - The time in seconds before the service start fails. Default: 120
• ignore_failure - Ignore failures starting the etcd service. Before quorum is established nodes will loop indefinitely and never successfully start. This can help ensure all instances are up when init systems can handle restart on failure. Default: false

etcd_service_manager_docker

Example

etcd_service_manager_docker 'default' do
  action :start
end

properties

• repo - defaults to 'quay.io/coreos/etcd'
• tag - default calculated from version
• version - defaults to '3.2.15',
• container_name - defaults to resource name
• port - defaults to ['2379/tcp4:2379', '2380/tcp4:2380']
• host_data_path - Path to store data locally on the host, which will be mounted into the container

etcd_service

The etcd_service: resource is a composite resource that uses etcd_installation and etcd_service_manager resources to install and manage the etcd service.

• The :create action uses an etcd_installation
• The :delete action uses anetcd_installation
• The :start action uses an etcd_service_manager
• The :stop action uses an etcd_service_manager

The service management strategy for the host platform is dynamically chosen based on platform, but can be overridden.

Properties

The etcd_service resource property list corresponds to the options found in

Etcd Configuration Flags documentation

Member properties

• source
• node_name
• data_dir
• wal_dir
• snapshot_count snapshot to disk.
• heartbeat_interval
• election_timeout
• listen_peer_urls
• listen_client_urls
• max_snapshots
• max_wals
• cors
• quota_backend_bytes

Clustering properties

• initial
• initial_advertise_peer_urls
• initial_cluster
• initial_cluster_state
• initial_cluster_token
• advertise_client_urls
• discovery
• discovery_srv
• discovery_fallback
• discovery_proxy
• strict_reconfig_check
• auto_compaction_retention
• enable_v2

Proxy properties

• proxy
• proxy_failure_wait
• proxy_refresh_interval
• proxy_dial_timeout
• proxy_write_timeout
• proxy_read_timeout

Security properties

• cert_file
• key_file
• client_cert_auth
• trusted_ca_file
• auto_tls
• peer_cert_file
• peer_key_file
• peer_client_cert_auth
• peer_trusted_ca_file
• peer_cert_allowed_cn
• peer_auto_tls
• etcdctl_client_cert_file
• etcdctl_client_key_file
• experimental_peer_skip_client_san_verification

Logging properties

• debug
• log_package_levels

Profiling properties

• enable_pprof
• metrics
• listen-metrics-urls

Auth Flpropertiesags

• auth_token

Unsafe properties

• force_new_cluster

Misc properties

• http_proxy
• https_proxy
• no_proxy
• auto_restart

etcd_key

The etcd_key resource sets, watches and deletes keys in etcd.

Actions

• The :set action sets a key
• The :delete action deletes a key
• The :watch action waits for a key to update

Properties

• key - The key name
• value - The desired value
• ttl - The ttl for the key (optional)
• host - The hostname of the etcd server, defaults to 127.0.0.1
• port - The port that etcd is listening on, defaults to 2379

Examples

etcd_key "/test" do
  value "a_test_value"
  action :set
end

Contributors

This project exists thanks to all the people who contribute.

Additional Contributors

• Jesse Nelson
• Soulou
• Aaron O'Mullan
• Anthony Scalisi
• Robert Coleman
• James Gregory
• Sean OMeara

Backers

Thank you to all our backers!

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website.

etcd's Issues

etcd_service_manager_docker: undefined method new_resource

Cookbook version

etcd 5.0.0 (latest available in https://supermarket.chef.io)

Chef-client version

Chef: 12.16.42

Platform Details

Host

Arch Linux x86_64, latest
linux kernel: 4.12.10
Vagrant 1.9.7
libvirtd (libvirt) 3.7.0
vagrant-libvirt (0.0.40)

Guest

Centos 7
linux kernel: 3.10.0-514.26.2.el7.x86_64
Chef: 12.16.42

Scenario

When trying to converge my recipe containing:

docker_service 'default'

etcd_installation_docker 'default' do
  action :create
end

etcd_service_manager_docker 'default' do
  action :start
end

I receive the following exception

================================================================================
Error executing action `start` on resource 'etcd_service_manager_docker[default]'
================================================================================

NoMethodError
-------------
undefined method `new_resource' for EtcdCookbook::EtcdServiceManagerDocker

Cookbook Trace:
---------------
/var/chef/cache/cookbooks/etcd/libraries/helpers_service.rb:65:in `etcd_name'
/var/chef/cache/cookbooks/etcd/libraries/etcd_service_manager_docker.rb:10:in `block in <class:EtcdServiceManagerDocker>'
/var/chef/cache/cookbooks/etcd/libraries/etcd_service_manager_docker.rb:15:in `block in <class:EtcdServiceManagerDocker>'

Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/etcd_node/recipes/service.rb

 13: etcd_service_manager_docker 'default' do
 14:   action :start
 15: end

Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/etcd_node/recipes/service.rb:13:in `from_file'

etcd_service_manager_docker("default") do
  action [:start]
  retries 0
  retry_delay 2
  default_guard_interpreter :default
  declared_type :etcd_service_manager_docker
  cookbook_name "etcd_node"
  recipe_name "service"
end

Platform:
---------
x86_64-linux

Steps to Reproduce

Create a cookbook that depends on etcd cookbook version 5.0.0 with the recipe mentioned above and try to do a kitchen converge.

Expected Result:

It should converge without errors and have etcd setup to run on docker.

Actual Result:

Fails with the above exception right after pulling the etcd image.

Update Source install to use a cook for handling installing golang on multiple platforms

etcd_service errors out when using docker for install method

Cookbook version

4.1.0

Chef-client version

12.15.19

Platform Details

Debian 8.5

Scenario:

I'm trying to install the etcd service using Docker and the etcd_service resource.

Steps to Reproduce:

In a recipe, all you have to do it is invoke the resource like so:

etcd_service 'etcd' do
  install_method 'docker'
end

Expected Result:

I was expecting this generic resource to correctly delegate the install operation to a implementation-specific resource.

Actual Result:

This errored out because the underlying logic does not support the Docker install method.

Here's the stacktrace:

The properties method is being called on NilClass because there's no switch case for the docker option (https://github.com/chef-cookbooks/etcd/blob/v4.1.0/libraries/etcd_service.rb#L33-L46).

I can try to patch and submit a PR for this is that's desirable. I just want to make sure that you are trying to support this use case and if the design is correct.

Start action on service resoruce shouldn't work w/o binary installed

We should validate that the binary exists. Otherwise things fail with very cryptic messaging.

Document Source Install

Tests for source install?

I've been trying to use the source installation, but i'm encountering a lot of issues.

Is this expected? Is the source install tested?

Add kitchen bats/serverspec tests for source installl

systemd service_timeout property is unused

Cookbook version

5.3.1

Chef-client version

13.8.5

Platform Details

Ubuntu 16.04 AWS

Scenario:

etcd_service_manager_systemd "default" do
  service_timeout 3600
  ...
end

Steps to Reproduce:

Use a resource as above.

Expected Result:

Systemd service unit should use the property value for TimeoutStartSec.

Actual Result:

Resource does not use this property so TimeoutStartSec in service unit remains unchanged.

The manually-specified cluster members are not actually configured

When I specify etcd.nodes = ["node1", "node2"], all I get is a file - /etc/etcd_members.

That file is not loaded as -peers-file, so the attribute has no effect.

I temporarily solved this by setting etcd.args = "-peers-file=/etc/etcd_members", but it looks like a bug to me, as the Readme doesn't mention any other actions while setting up the cluster manually.

Should I send a patch that fixes this ?

no etcd user

etcd should start as the etcd user ideally?
I am going to make a pull request.

refactor cookbook for etcd 2.0

I made some changes on my own fork to run etcd 2.0 and realised that the vast amount of config changes might warrant a refactor of the cookbook.

Im wondering if we move to LWRP's or just move the cluster cookbook logic into the library?

I feel like LWRP's would be a cleaner implementation however.

Thoughts?

doc & practice a release

I'd like to make sure we can still release this cookbook if we need to add a new maintainer. Could someone work to show me how to do a release?

Bump version, probably to 3.0.0

My latest patch changes the user and group that runs etcd from root to etcd, this breaks etcd, it starts but cannot access the state dir. It errors in the logs. You need to chown -R etcd:etcd /var/cache/etcd

I can't do chown -R in chef, as i dont think the recursive attribute works like that from memory.

As per semver, this should be 3.0.0 as it is a non backwards compatible breaking change.

Install from tar

As github releases are tars, it'd be good if node[:etcd][:url] could be a tar.

e.g. node[:etcd][:url] = 'https://github.com/coreos/etcd/releases/download/v0.1.0/etcd-v0.1.0-Linux.tar.gz'

Support versions 3.3.19 and 3.4.6

Cookbook version

5.6.0

Chef-client version

14.10.9

Platform Details

Debian 4.9

Scenario:

Trying to install newer versions of etcd, them being 3.3.x and 3.4.x

Steps to Reproduce:

etcd_installation 'default' do
  action :create
end

etcd_service_manager 'default' do
  advertise_client_urls "http://#{node['ipaddress']}:2379"
  listen_client_urls 'http://0.0.0.0:2379'
  initial_advertise_peer_urls "http://#{node['ipaddress']}:2380"
  listen_peer_urls 'http://0.0.0.0:2380'
  initial_cluster "default=http://#{node['ipaddress']}:2380"
  action :start
end

content in Berksfile

cookbook 'etcd', '~> 5.6.0'

Expected Result:

Newer versions of etcd should get installed when we call the cookbook

Actual Result:

The default etcd which get's installed as of now is 3.2.15

Make `etcd_bin_prefix` configurable

🗣️ Foreword

Thank for taking the time to fill this feature request fully. Without it we may not be able to , and the issue may be closed without resolution.

🙍 Problem Statement

I would love to be able to configure the folder where the etcd binaries are installed. We want to install two different versions on a single server.

❔ Possible Solution

I can see that the etcd_bin_prefix is currently hard coded:

etcd/resources/etcd_installation_binary.rb

Lines 66 to 68 in 9b26b2c

	def etcd_bin_prefix
	'/usr/bin'
	end

We would like to make it a property, just like the version for instance.

This prefix should also be configurabe in etcd_service_manager_systemd.

If this is something interesting for you, we developed a PR (Scalingo#1) which can be backported here.

⤴️ Describe alternatives you've considered

We considered using etcd_installation_docker (rather than etcd_installation_binary currently) but it would be a lot of work. Moreover it seems impossible to change the mounted volumes but we need it to mount the TLS files inside the etcd container.

➕ Additional context

n/a

docker service manager does NOT map host volume

Cookbook version (4.1.0), Chef-client version (12.15.19)

etcd_service_manager_docker does not provide an option to map the data_dir to a local volume on the host. Doing so will allow users to restart/update their containers without blasting all of their etcd data, and this is very important in order to maintain cluster stability in production.

the setgid/uid options are not supported in the upstart in centos6

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

github-actions

.github/workflows/ci.yml

• sous-chefs/.github 3.1.1
• actions/checkout v4@a5ac7e51b41094c92402da3b24376905380afc29
• actionshub/chef-install 3.0.0
• actionshub/test-kitchen 3.0.0
• actions/checkout v4@a5ac7e51b41094c92402da3b24376905380afc29
• actionshub/chef-install 3.0.0
• actionshub/test-kitchen 3.0.0
• actions/checkout v4@a5ac7e51b41094c92402da3b24376905380afc29
• actionshub/chef-install 3.0.0
• actionshub/test-kitchen 3.0.0
• ubuntu 22.04

.github/workflows/stale.yml

• actions/stale v9

---

• Check this box to trigger a request for Renovate to run again on this repository

init script should not fail when one node starts

I'm working on etcd cluster with DNS discovery. It's a cluster of three nodes.
According to etcd developers DNS must be ready when first node starts. (See etcd-io/etcd#5909 for context).

When there are three nodes in DNS and the first node starts it expects quorum. However nodes are starting one by one and second doesn't start until first successfully starts.
Even though etcd starts as expected and behaves as expected in this situation the init script exits with non-zero it breaks knife bootstrap.

The init script should exit with zero so second node can start and the cluster becomes operational.

Otherwise it's not possible to start the cluster. Chef will panic on a non-zero code.
The only workaround I see is to start three nodes in parallel which is not appropriate/possible in some cases.

update docker service manager to install etcd v3

Cookbook version (4.1.0), Chef-client version (12.15.19)

The etcd_service_manager_docker resource maps ports 2379 and 4001 by default. Etcd v3 now uses port 2380 for peer traffic. The version and port properties should both be updated so that we can use the latest version of etcd by default.

Here is the port mapping

Use strings in preference to symbols to access node attributes

As per foodcritic FC001 rule, strings should be used instead of symbols.
someone run something like this, but better
echo "node[:etcd][:upstart][:start_on]" | sed "s/]/']/g" | sed "s/:/'/g"

7.0.0+ No longer works

🗣️ Foreword

Thank for taking the time to fill this bug report fully. Without it we may not be able to fix the bug, and the issue may be closed without resolution.

👻 Brief Description

When using etcd cookbook 7.0.0 or 7.0.1, the start service is not working properly.

etcd_service node['hostname'] do
  advertise_client_urls "http://#{node['ipaddress']}:2379"
  listen_client_urls "http://#{node['ipaddress']}:2379,http://127.0.0.1:2379"
  initial_advertise_peer_urls "http://#{node['ipaddress']}:2380"
  listen_peer_urls "http://#{node['ipaddress']}:2380"
  initial_cluster_token 'etcd-cluster-1'
  initial_cluster cluster_members.join(',')
  initial_cluster_state 'existing'
  default_service_name true
  data_dir '/var/lib/etcd'
  enable_v2 true
  action :start
end

After adding additional logging, the resource is not picking up any of those resource attributes. I have narrowed down the issue to the unified_mode true line in etcd_service.rb resource. When I set that to be false, everything works fine. Alternatively, if I use etcd_service_manager_systemd resource directly it also works fine.

🥞 Cookbook version

7.0.0/7.0.1

👩‍🍳 Chef-Infra Version

Tested on both 15.17.4 & 17.2.29

🎩 Platform details

RHEL

Steps To Reproduce

Steps to reproduce the behavior:

1. Use resource above
2. Run chef-client
3. Note that the etcd service file created is not using the default name and has none of the attributes passed in

🚓 Expected behavior

Resource attributes should properly pass to the systemd resource

➕ Additional context

Add any other context about the problem here. e.g. related issues or existing pull requests.

etcd options use single dash instead of double dash

etcd documentation specifies options as having double dash such as --data-dir not a single dash like -data-dir as used in this cookbook.
https://coreos.com/etcd/docs/latest/op-guide/configuration.html

etcd does not seem to care one way or the other but this can cause unexpected issues.

For example, this InSpec profile does not work because it is looking for --data-dir, not -data-dir:
https://github.com/dev-sec/cis-kubernetes-benchmark

While one might argue that this should be fixed in the InSpec profile it seems most appropriate that this cookbook should be updated to use syntax as defined by etcd.

Cookbook no longer works with etcd 3.5.x

🗣️ Foreword

Thank for taking the time to fill this bug report fully. Without it we may not be able to fix the bug, and the issue may be closed without resolution.

👻 Brief Description

This cookbook no longer works with 3.5.x of etcd. This is due to the -debug flag being deprecated. This flag should either be removed or conditionally removed based on the version.

🥞 Cookbook version

7.0.3

👩‍🍳 Chef-Infra Version

17.7.29

🎩 Platform details

RHEL7

Steps To Reproduce

Steps to reproduce the behavior:

1. Attempt to setup systemd for 3.5.1 of etcd

🚓 Expected behavior

Should start properly

➕ Additional context

N/A

etcd_service_manager_docker launches container with incorrect command

Cookbook version (4.1.0), Chef-client version (12.15.19)

The etcd_service_manager_docker resource has a bug inside of its :start action. It's passing the value of etcd_daemon_opts.join(' ').strip to the docker container command when, I believe, it should be the value of etcd_cmd.

Busted stuff: https://github.com/chef-cookbooks/etcd/blob/v4.1.0/libraries/etcd_service_manager_docker.rb#L16

Better option: https://github.com/chef-cookbooks/etcd/blob/master/libraries/helpers_service.rb#L8-L10

how can chef-brigade help?

We're working on a number of ways to help cookbooks, and the existing stuff is in https://github.com/chef-brigade/meta

But is there anything in particular etcd needs help with? Common things:

• testing
• release process
• handling open PRs
• usage doc
• contributing doc

cookbook installs outdated version

Cookbook version

3.2.5

Chef-client version

# etcdctl -v
etcdctl version 2.2.2

Platform Details

# cat /etc/*-release
CentOS Linux release 7.2.1511 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.2.1511 (Core)
CentOS Linux release 7.2.1511 (Core)

Scenario:

I'm installing an etcd cluster with DNS discovery.
Due to a bug in 2.2.2 etcdctl connects to wrong port.

Steps to Reproduce:

See discussion on https://groups.google.com/forum/?pli=1#!topic/etcd-dev/R4x9l7pi6fc

Expected Result:

etcdctl is updated to 2.3.7

Actual Result:

etcdctl version 2.2.2

Missing peer_cert_allowed_cn security property

v3.3.0 adds etcd --peer-cert-allowed-cn flag to support CN(Common Name)-based auth for inter-peer connections. Kubernetes TLS bootstrapping involves generating dynamic certificates for etcd members and other system components (e.g. API server, kubelet, etc.). Maintaining different CAs for each component provides tighter access control to etcd cluster but often tedious. When --peer-cert-allowed-cn flag is specified, node can only join with matching common name even with shared CAs.