elastic / cloud-on-k8s

Elastic Cloud on Kubernetes

Elastic Cloud on Kubernetes (ECK)

Elastic Cloud on Kubernetes automates the deployment, provisioning, management, and orchestration of Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, Elastic Maps Server, and Logstash on Kubernetes based on the operator pattern.

Current features:

• Elasticsearch, Kibana, APM Server, Enterprise Search, and Beats deployments
• TLS Certificates management
• Safe Elasticsearch cluster configuration & topology changes
• Persistent volumes usage
• Custom node configuration and attributes
• Secure settings keystore updates

Supported versions:

• Kubernetes 1.25-1.29
• OpenShift 4.11-4.14
• Elasticsearch, Kibana, APM Server: 6.8+, 7.1+, 8+
• Enterprise Search: 7.7+, 8+
• Beats: 7.0+, 8+
• Elastic Agent: 7.10+ (standalone), 7.14+, 8+ (Fleet)
• Elastic Maps Server: 7.11+, 8+
• Logstash 8.7+

Check the Quickstart to deploy your first cluster with ECK.

If you want to contribute to the project, check our contributing guide and see how to setup a local development environment.

For general questions, please see the Elastic forums.

cloud-on-k8s's Issues

Support for taking and getting heap dumps and other analytics

• How should initiate and how should we be able to get to it?
  • E.g: store it inside a mounted volume? Upload to a s3/repository bucket? Same as snapshot bucket? Lifecycle then becomes an issue to GC eventually
• Other JVM/debugging tools that would be relevant?

Migrate data being triggered right away

Problem

It seems that we're issuing the migrate calls before the ES pod is ready.

Causes the migrate check to pollute the logs

Raw logs

make run
go generate ./pkg/... ./cmd/...
goimports -w pkg cmd
go vet ./pkg/... ./cmd/...
go run ./cmd/manager/main.go
{"level":"info","ts":1541597261.359024,"logger":"entrypoint","caller":"manager/main.go:21","msg":"setting up client for manager"}
{"level":"info","ts":1541597261.360708,"logger":"entrypoint","caller":"manager/main.go:29","msg":"setting up manager"}
{"level":"info","ts":1541597262.9435961,"logger":"entrypoint","caller":"manager/main.go:36","msg":"Registering Components."}
{"level":"info","ts":1541597262.9436312,"logger":"entrypoint","caller":"manager/main.go:39","msg":"setting up scheme"}
{"level":"info","ts":1541597262.943677,"logger":"entrypoint","caller":"manager/main.go:46","msg":"Setting up controller"}
{"level":"info","ts":1541597262.943923,"logger":"kubebuilder.controller","caller":"controller/controller.go:120","msg":"Starting EventSource","Controller":"stack-controller","Source":"kind source: /, Kind="}
{"level":"info","ts":1541597262.9442098,"logger":"kubebuilder.controller","caller":"controller/controller.go:120","msg":"Starting EventSource","Controller":"stack-controller","Source":"kind source: /, Kind="}
{"level":"info","ts":1541597262.944353,"logger":"kubebuilder.controller","caller":"controller/controller.go:120","msg":"Starting EventSource","Controller":"stack-controller","Source":"kind source: /, Kind="}
{"level":"info","ts":1541597262.94458,"logger":"kubebuilder.controller","caller":"controller/controller.go:120","msg":"Starting EventSource","Controller":"stack-controller","Source":"kind source: /, Kind="}
{"level":"info","ts":1541597262.944698,"logger":"entrypoint","caller":"manager/main.go:52","msg":"setting up webhooks"}
{"level":"info","ts":1541597262.9447172,"logger":"entrypoint","caller":"manager/main.go:59","msg":"Starting the Cmd."}
{"level":"info","ts":1541597263.1489651,"logger":"kubebuilder.controller","caller":"controller/controller.go:134","msg":"Starting Controller","Controller":"stack-controller"}
{"level":"info","ts":1541597263.253657,"logger":"kubebuilder.controller","caller":"controller/controller.go:153","msg":"Starting workers","Controller":"stack-controller","WorkerCount":1}
{"level":"info","ts":1541597290.767325,"logger":"stack-controller","caller":"stack/stack_controller.go:274","msg":"Created Pod stack-sample-es-mzgmvh9t6f","iteration":1}
{"level":"info","ts":1541597290.8207848,"logger":"stack-controller","caller":"stack/stack_controller.go:274","msg":"Created Pod stack-sample-es-vlb8kq58pf","iteration":1}
{"level":"info","ts":1541597290.873826,"logger":"stack-controller","caller":"stack/stack_controller.go:274","msg":"Created Pod stack-sample-es-dntfxxqr6j","iteration":1}
{"level":"info","ts":1541597290.873926,"logger":"stack-controller","caller":"stack/service_control.go:26","msg":"Creating service default/stack-sample-es-discovery","iteration":1}
{"level":"info","ts":1541597290.965258,"logger":"stack-controller","caller":"stack/service_control.go:26","msg":"Creating service default/stack-sample-es-public","iteration":1}
{"level":"info","ts":1541597291.0271301,"logger":"stack-controller","caller":"stack/deployment_control.go:77","msg":"Creating Deployment default/stack-sample-kibana","iteration":1}
{"level":"info","ts":1541597291.1224709,"logger":"stack-controller","caller":"stack/service_control.go:26","msg":"Creating service default/stack-sample-kb","iteration":1}
{"level":"error","ts":1541597291.25213,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597292.255729,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597293.261707,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597294.2684531,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597295.275768,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597296.2815711,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597297.2881322,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597298.295207,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597299.301722,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597300.588762,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597303.1528802,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597308.27569,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597317.761508,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597318.768116,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1541597319.774601,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"stack-controller","Request":"default/stack-sample","error":"Error during migrate data: Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host","errorVerbose":"Put http://stack-sample-es-public:9200/_cluster/settings: dial tcp: lookup stack-sample-es-public: no such host\nError during migrate data\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).DeleteElasticsearchPods\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:327\ngithub.com/elastic/stack-operators/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/Users/marc/go/src/github.com/elastic/stack-operators/pkg/controller/stack/stack_controller.go:147\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/marc/go/src/github.com/elastic/stack-operators/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

Discuss: RBAC on K8s resources and operator service account

Operator needs a service account with a specific set of responsibilities.

It would be great to limit this to just the concrete resources etc we need within a namespace, but depending on the granularity we can express, we might have to accept some namespace defining boundaries in some cases.

Discuss: Stack versions as a resource to match stack packs

Conceptually one or maps that translates a stack version to images for our stack.

Things to decide on:

• Should this include a version requirement/description for the stack operator?
• Should this be a single resource or one resource per stack version?
• Overrides and defaults to ES settings?

Stack versions are similar to stack packs providing concrete stack image names, available plugins, default plugins, node type support, platform/controller/operator version requirements etc.

Metering support: determining resource usage and policy enforcement

We should enable capturing resource utilization and enable support for metering as a way to handle pluggable and auditable show/charge-backs as well as potential billing.

Support for APM server configuration

• apm server controller
• full support for APM configuration (see #53 (comment))

Support for temporarily pausing the stack operator

It would be great to be able to pause the stack operator from doing anything, which could be useful in everything from "hey, I think we're having a cascading failure situation here, and things need to calm down before we continue" to "ok, for testing purposes, let's purposely put the system in this specific state", then start reconciling.

Snapshot repository settings for a cluster

We want to provide the ability for clusters to perform snapshots:

• Part of the ES spec directly?
• But could also reference another snapshot-resource (or snapshotresourcerequest that can be fulfilled by another controller)?

Custom YAML Settings for Elasticsearch

Support YAML settings in the Elasticsearch Spec similar to what we currently have with user_settings_override

Support different cluster topology change policies.

• In-line changes / restart (might not be possible with emptyDir?)
  • Some caveats exist here when it comes to what can change in a pod spec without it being considered a new node.
• Rolling Blue/Green (how many at a time etc). Does this relate to some pod disruption budget (https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) as well?
• A/B-style changes. (e.g complete set of new instances, all migration happens simultaneously).

Change ES readiness to request /_cat/master

For now we check for green health before we consider an ES pod to be ready.
This is a problem during topology changes.

Instead we should consider the pod to be ready when it sees a master ES node (call to _cat/master).

Licensing support (per deployment first)

At some point, we'd like to be able to support licensing ECE-style where the install contains a pool of licenses applied to clusters (basically copy the implementation / behavioral details from ECE, roughly).

Shorter term perhaps a dedicated resource for this could make just as much sense? E.g user has to update a placeholder resource we create as part of the cluster with the license? Or reference a secret containing the license?

Setup a service for cluster discovery

Support IP filtering on a cluster level

This might have more involved implications when it comes to accepting connections from remote clusters, possibly accessed through some form of ingress-like proxy (here we'd have to enforce the ip filtering in the proxy and allow the proxy to connect to the ES instances).

Perhaps network policies would make more sense than using the xpack ip filtering support directly in some cases? Or would both make sense still?

Migrate Data Before Deleting Pods

Similar to existing constructor logic:

• set allocation filters
• observe shard allocation and/or recovery to detect progress and completion

Reorganize code to multiple packages

We have a rather flat structure in the elasticsearch package, that could be broken down into multiple more scoped packages.
Also we should probably lighten a bit the main controller file.

Support for declaring a remote monitoring cluster

Auth between clusters needs to be configured in this case. Should not require the target cluster to be within the same installation even. Some manual steps may then have to be taken, but we can probably provide some more automation if it's within the same installation, and also some (other) automation if it's in a remote installation where we have some (stack-operators level) federation support available.

Discuss: Events and us, what should we emit as events?

• ES instance exit codes are important to get visibility into

• Bootlooping detection

  • Usually happens during reconfigs or upgrades, what would we do in a reconciliation loop? Not entirely clear what we can / should roll back to?

Expose Kibana as a service

Implement rolling grow and shrink

For now we:

1. grow
2. shrink

During step 1 we might double the actual node count, which is not great.

Support for custom realms for Elasticsearch authentication.

By default we will use the file realm for our internal uses and let consumers of our deployment use the ES native realm. For larger deployments, custom realms become more important:

These resources might have to include a versioned component as well (not all of these are as important as the others):

• SAML
• LDAP
• Oauth / OpenID connect

In some cases, this would also entail installing a certificate (e.g ldaps) so we can support encryption.

Phone home style support for telemetry

Not quite clear where this would fall in terms of responsibilities. Could be as simple as Kibana for the single-operator case.

Might become clearer what we need / want once we start working on a multiple-stacks operator?

Reschedule pods when a node is tainted / drained.

Drain

Reference links

• https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
• https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

Set default node type to master+data+ingest

I tried tweaking quickly with UnmarshalJSON on the NodeTypesSpec struct but it never got called by kubebuilder... Not sure why.

For now in the stack we default to false on every type, meaning we deploy a coordinator node.
ML type defaults to false in Elasticsearch, unless X-Pack is activated. In such case it defaults to true. Not sure how to handle that.

Fail-over and snapshot recovery

When and how should we make the decision to restore from a snapshot?

• Restores are not always the same:

  • Partial restores required sometimes?
  • Full restores at other times?

• Possibility to tie into cluster lifecycle state (as part of the controller state machine system?)

• Queueing of one-off operations as a general feature?

Support different cluster topologies

Create various topology for Elasticsearch / Kibana

• I can create master-only nodes
• I can create data-only nodes
• I can create ml nodes
• I can create ingest nodes
• I can create coordinator nodes

Discuss: Dry-running or validation of configuration changes

It would be nice to have a concept of dry-running changes without affecting the stability or actual state of the system.

E.g realm configuration changes: validation, connectivity checks etc to external systems.

Define and Implement Client Timeouts when Interacting with Elasticsearch

Follow up to #24

See https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/

We want probably fairly short client timeouts in the controller, to avoid blocking the reconcile loop for longer than necessary. Instead we should retry the request in the next iteration of the loop

Alternative: pull the request out of the reconciliation loop and allow longer timeouts.

User-provided node affinities for stack component nodes

How should we expose these to our users?

Custom plugin and bundles support.

We don't want to store binary data in K8s apis, so perhaps the ECE way of declaring URLs are acceptable? Does tie the availability of nodes to the availability of the URLs.

Perhaps a new CRD for these would make sense as well? They are potentially something we'd like shared between clusters, and they need to be inherently versioned. (Plugins more-so than bundles, so perhaps we split this into a subtask for each?)

Discuss: One-time actions vs state reconciliation

Decouple “one-time” action (eg. cluster restart) from state change reconciliation (eg. ES deployment or version upgrade). But make sure both cannot run at the same time.?

Figure out how to handle this kind of “one-time” request, probably with sub-resources CRD webhooks. Related: https://groups.google.com/forum/#!msg/kubernetes-sig-api-machinery/wMblxpOSoiA/T96qSXwZBQAJ and https://kubernetes.io/blog/2018/07/27/kubevirt-extending-kubernetes-with-crds-for-virtualized-workloads/

Support for diagnostics bundle / dumping

Perhaps not something to directly bundle into our resources, but bake some supporting infra / labels etc into this?

A script or something otherwise runnable could suffice, and what we'd want is a final artifact of a reasonable size that will encapsulate all information (without containing secrets etc) we'd need when remotely debugging weird or unexpected behavior on a system we can't or won't directly access.

This is something we'd like to have working on both a installation-wide basis as well as on a deployment-wide basis.

K8s installation and deployment tooling

Integrations includes a large section around configuration management where customers want to integrate with tools such as terraform / ansible. As Kubernetes already has a ton of community integrations with these configuration management tools we can build on top of what already exists. Note: That while we gain some benefit with pre-existing configuration code we would essentially trade those benefits for additional testing. Customer tooling would become a regression that we would need to rewrite in Kubernetes. As mentioned above with Kubernetes we can use functions such as the internal describe and logging functionality, but these would need to be properly combined to match fidelity already found in ECE diagnostics today.

Volumes for the ES pods

• emptyDir (This would be the lowest common denominator (quotas though?)
  • sizeLimit on emptyDir
• PVCs:
  • Local a viable option?
  • Which ones would we like to add support for first?

Use testify instead of Gomega

Install s3 plugin with a script in an init container

This is as a "simple" hacky solution. See discussion on #34.

Migration should take into consideration auto-expand indices

Explore local persistent volumes feasibility

Local persistent volume?

Concepts

Local persistent volume

See the official doc: https://kubernetes.io/docs/concepts/storage/volumes/#local
The idea behind persistent volumes is to create:

• a persistent volume resource, bound to a specific node with a selector, that has a particular capacity
• a persistent volume claim to use that resource, that a pod spec will reference as volume

The problem here is we need to create these "PersistentVolume", but we don't know in advance what are going to be the pod's storage requirements. That's why some persistent volumes have "dynamic" provisioner, in the sense that the PV will be created automatically by a controller to match a given PVC. There is no dynamic provisioner yet for local persistent volumes.

It is marked as beta in Kubernetes v1.10.

CSI

Stands for Container Storage Interface (similar to CNI: Container Networking Interface). A project to standardize the way vendors implement their k8s storage API.

Spec: https://github.com/container-storage-interface/spec
K8S doc: https://kubernetes-csi.github.io/docs/
A short read on how it works: https://medium.com/google-cloud/understanding-the-container-storage-interface-csi-ddbeb966a3b
A more complete read on how to implement a CSI: https://arslan.io/2018/06/21/how-to-write-a-container-storage-interface-csi-plugin/
CSI community sync agenda: https://docs.google.com/document/d/1-oiNg5V_GtS_JBAEViVBhZ3BYVFlbSz70hreyaD7c5Y/edit#heading=h.h3flg2md1zg

Components:

K8S-internal (not vendor specific, maintained by K8S team):

• driver-registrar: Sidecar container that registers the CSI driver with kubelet, and 2) adds the drivers custom NodeId to a label on the Kubernetes Node API Object.
• external-provisioner: Sidecar container that watches Kubernetes PersistentVolumeClaim objects and triggers CreateVolume/DeleteVolume against a driver endpoint.
• external-attacher: Sidecar container that watches Kubernetes VolumeAttachment objects and triggers ControllerPublish/Unpublish against a driver endpoint

CSI driver - vendor specific (all these should implement the gRPC CSI standard interface), composed of 3 components:

• identity: gRPC server identifying the plugin service, making sure it’s healthy, and returning basic information about the plugin itself. Can be implemented as part of the Node or Controller plugins.
• Node plugin: deployed on each node, gRPC server responsible for mounting volumes. Usually a daemonset.
• controller: usually deployed only once. gRPC server responsible for controlling and managing all volumes in the cluster.

FlexVolumes

FlexVolumes can be seen as the old, unclean version of CSI. It also allows vendors to write their own storage plugins. The plugin driver needs to be installed to a specific path on each node (/usr/libexec/kubernetes/kubelet-plugins/volume/exec/). It's basically a binary exec file that needs to support a few subcommands (init, attach, detach, mount, unmount etc.).

Our need

We need dynamically provisioned local persistent volumes. Which means a controller should take care of mapping existing PVC to a new PV of the expected size.
Also, we expect the size to act as a quota: when reached, the user should not be able to write on disk anymore. This is a strong requirement: for instance using ext4 behind our persistent volumes would probably not guarantee this.

Interesting resources

1. kubernetes-incubator local-volume static provisioner

Links: https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume, https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume/provisioner

A static local volume provisioner, running as a daemon set on all nodes of the cluster. It monitors mount points on the system, and maps it to the creation of a PV of the corresponding size.
Mount points are discovered in the configured discovery dir (eg. /mnt/disks).
To work with directory-based volumes instead of device-based volumes, we can simply symlink dirs to the directory we want into the discovery dir.
It does not handle any quota; but the backing FS could (eg. XFS or LVM).
Code is open source, quite small and simple to understand.

Dynamic provisioner WIP

Dynamic provisioning seems to be WIP according to this issue. There is a PR open for design proposal and a PR open for implementation.
Based on the design doc:

• LVM seems to be the target filesystem to support first
• The driver running on all nodes should be a CSI provisioner

Based on this comment, the dynamic CSI provisioner is still at the level of "internal discussions".

lichuqiang seems to be pretty involved in that. Interestingly, he created a Github repo for a CSI driver which is mostly based on mesosphere's csilvm.

Overall, this looks very promising and close to what we need. Patience is required :)

2. Mesosphere csilvm

Link: https://github.com/mesosphere/csilvm

A CSI for LVM2. It lives as a single csilvm binary that implements both Node and Controller plugins, to be installed on every node. The names of the volume group (VG) and the physical volumes (PVs) it consists of are passed to the plugin at launch time as command-line parameters.

It is originally intended to work on Mesos, not on Kubernetes. But the CSI standard is supposed to work for both.

The code is quite clean and easy to understand :)

This issue contains some interesting comments (from july) on how the project does not exactly comply with the expected k8s interface.

I could not find any reference of someone using it as a daemonset on a k8s cluster.

3. wavezhang/k8s-csi-lvm

Link: https://github.com/wavezhang/k8s-csi-lvm

Seems a bit less clean than csilvm, but explicitely targets Kubernetes.
Not much doc, and only a few commits, but the code looks quite good.
Based on the code and examples, can be deployed as a DaemonSet (along with the required kubernetes CSI stuff and apiserver configuration).
It relies on having lvmd installed on he host, with a LVM volume group pre-created. See this bash script, supposed to be run on each node.

4. akrog/ember-csi

Link: https://github.com/akrog/ember-csi

5. scality/metalk8s LVM usage

Link: https://github.com/scality/metalk8s

6. Dynamic provisioner FlexVolume implementation

Link: https://github.com/monostream/k8s-localflex-provisioner

Potential solutions for dynamic provisioning

Best solution seems to be:

• A CSI implementation for dynamic local storage
  • Runs as a daemon set on all nodes (or a subset of labeled nodes)
  • With access to the host disks (devices or partitions)
  • Probably needs to be privileged (depending on the FS choice)
  • Dynamically provision a PV based on pods PVC
• It needs to "discover" local disks
  • Can we assume it can use whatever local disk is attached to the node? I think not, since they might be used already for other pods
  • Most solutions (including the official static local volume provisioner) seem to rely on a "discovery directory", where all disks or partitions should be mounted (or symlinked) to be discovered. Eg. /mnt/disks
  • Another option could be to label the node resource with paths to the disks that can be used as local volumes. This does not seem to be very standard though.

Potential solutions for quota-aware filesystems

Best solution seems to rely on LVM + XFS:
* Create one volume group from the available physical disks or partitions
* Create one logical volume per pod, with the requested size: the volume size acts as the quota
* Use thin volume if we need to overcommit on disk space
* Then we need to choose over XFS or ext4 filesystems for the logical volumes:
* XFS is supposed to have better performance, but cannot be shrinked without being unmounted first
* ext4 does support hot grow/shrink

XFS

XFS is an I/O optimized file system (compared to eg. ext4).
It supports quotas per directory (xfs_quota command), allowing us to create a directory per user, and associate a quota to it.
This is the solution we use on Elastic Cloud.

LVM

LVM allows to gather multiple disks or partitions to form a logical Volume Group (VG) (vgcreate command), where physical disks are abstracted. In this volume group, multiple Logical Volumes (LV) can be created with the chosen size (pvcreate command), and formatted with any FS we want (mkfs command). A logical volume may span over multiple physical disks.

LVM thin provisioning allows to create a thin pool on which we can allocate multiple thin volumes with a given size. That size will appear as the volume size, but the occupied underlying disk space will not be reserved for the volume. For instance, we could have a 10GB thin pool with 3x5GB thin volumes. Each volume would see 5GB, and everything is fine as long as the entire underlying 10GB is not fully occupied. It allows us to overcommit on disk space.

The "quota" in LVM would simply be the size of the created logical volume.

Volume group and logical volumes can be resized without unmounting them. However the logical volumes FS also needs to support that. Ext4 does support hot resize (grow and shrink), but XFS does only support hot grow (not shrink).

I/Os can be limited through Linux cgroups per logical volume (see https://serverfault.com/questions/563129/i-o-priority-per-lvm-volume-cgroups).

Set up Jenkins or similar build automation for testing etc

fmt, vet, compile, test, build docker, push.

Index curation support

For hot-warm and similar architectures it would be great to have some built-in index curation support to match ECE.

Security: Generate CSR in pods so private keys never leave a pod

Currently the operator generates private keys and pushes them to the pod through a Secret. This isn't optimal

Deploy a Kibana instance

Use a deployment for now to do that.

Periodic snapshotting

Depends on: #36

We'd like to have snapshots taken at specified intervals for DR and backup purposes.

Authenticated Probes

Once we have authenticated access to Elasticsearch/Kibana we need to provide those secrets to liveliness and readiness probes

• using environment variables has the problem that a change in credentials would require a container restart to update those
• @\nkvoll mentioned that the responsibility for those probes should maybe not be instance local but in the controller (support for that is rumored for 1.12)

Declare improved Conditions for our Status section of the StackDeployment CRD

Design TBD, but we can start with something obvious.

Internal TLS between nodes

Perhaps using https://github.com/cloudflare/cfssl

Set Zen discovery settings in control loop

discovery.zen.minimum_master_nodes needs to be adjusted based on the desired number of master eligible nodes in the cluster to make sure the cluster does not desintegrate on downscales because the originally configured value cannot be satisfied anymore.

This should be done via the cluster settings API as we otherwise would have to restart all the pods to update this setting which is currently configured in an environment variable

Security: Kibana HTTP-side certificates

We are already mirroring the http attribute that we have in the Elasticsearch CRD in the Kibana CRD but no behaviour is currently associated with that structure.

We should therefore:

• configure client/Kibana server TLS based on the http.tls attribute
• default to off if not defined
• generate self-signed certificate if SAN is specified
• allow users to specify their own certificate

CPU boosting support for smaller pods

Battery style? Could port the old found-based one to be a generic K8s-style daemonset-feature?

Add support for file realm based users and roles

• analog to current Cloud found_users
• store user/passwords/roles in secrets

Options:

• aggregate user created password with internal passwords