pires / kubernetes-elk-cluster

ELK (Elasticsearch + Logstash + Kibana) cluster on top of Kubernetes made easy.

This project is no longer maintained

As of November 7th, 2018, I've decided to end my commitment to maintaining this repo and related.

It's been more than 3 years since I last used ELK, so I no longer have the motivation it takes to maintain and evolve this project. Also, other projects need all the attention I can give.

It was a great run, thank you all.

kubernetes-elk-cluster

ELK (Elasticsearch + Logstash + Kibana) cluster on top of Kubernetes, made easy.

Here you will find:

• Kubernetes pod descriptor that joins Elasticsearch client-node container with Logstash container (for localhost communication)
• Kubernetes pod descriptor that joins Elasticsearch client-node container with Kibana container (for localhost communication)
• Kubernetes service descriptor that publishes Logstash
• Kubernetes service descriptor that publishes Kibana

Pre-requisites

• Kubernetes 1.1.x cluster (tested with 4 nodes Vagrant + CoreOS)
• kubectl configured to access your cluster master API Server
• Elasticsearch cluster deployed - you can skip deploying client-nodes provisioning, since those will be paired with Logstash and Kibana containers, and automatically join the cluster you've assembled with my Elasticsearch cluster instructions).

Deploy

The current Logstash configuration is expecting logstash-forwarder (Lumberjack secure protocol) to be its log input and the certificates provided are valid only for logstash.default.svc.cluster.local. I highly recommend you to rebuild your Logstash images with your own configuration and keys, if any.

Attention:

• If you're looking for details on how quay.io/pires/docker-elasticsearch-kubernetes images are built, take a look at my other repository.
• If you're looking for details on how quay.io/pires/docker-logstash image is built, take a look at my Logstash repository.
• If you're looking for details on how quay.io/pires/docker-logstash-forwarder image is built, take a look at my docker-logstash-forwarder repository.
• If you're looking for details on how quay.io/pires/docker-kibana image is built, take a look at my Kibana repository.

Let's go, then!

kubectl create -f service-account.yaml
kubectl create -f logstash-service.yaml
kubectl create -f logstash-controller.yaml
kubectl create -f kibana-service.yaml
kubectl create -f kibana-controller.yaml

Wait for provisioning to happen and then check the status:

$ kubectl get pods
NAME              READY     STATUS    RESTARTS   AGE
es-client-s1qnq   1/1       Running   0          57m
es-data-khoit     1/1       Running   0          56m
es-master-cfa6g   1/1       Running   0          1h
kibana-w0h9e      1/1       Running   0          2m
kube-dns-pgqft    3/3       Running   0          1h
logstash-9v8ro    1/1       Running   0          4m

As you can assert, the cluster is up and running. Easy, wasn't it?

Access the service

Don't forget that services in Kubernetes are only acessible from containers within the cluster by default, unless you have provided a LoadBalancer-enabled service.

$ kubectl get service kibana
NAME      LABELS                      SELECTOR                    IP(S)           PORT(S)
kibana    component=elk,role=kibana   component=elk,role=kibana   10.100.187.62   80/TCP

You should know what to do from here.

kubernetes-elk-cluster's Issues

Kibana unable to connect to elasticsearch

I'm getting these errors when running kibana

{"name":"Kibana","hostname":"kibana-z2aab","pid":7,"level":30,"msg":"Unable to connect to elasticsearch at http://elasticsearch.default.svc.cluster.local:9200. Retrying in 2.5 seconds.","time":"2016-02-11T19:12:16.168Z","v":0}

I tried nslookup and it cannot resolve the service name

kubectl exec kibana-z2aab -- nslookup elasticsearch.default.svc.cluster.local

Server: (null)
Address 1: ::1 localhost
Address 2: 127.0.0.1 localhost

nslookup: can't resolve 'elasticsearch.default.svc.cluster.local': Name does not resolve
error: error executing remote command: Error executing command in container: Error executing in Docker Container: 1

Looks like kubernetes DNS issue. Any ideas / pointers ?

thanks

Support Kubernetes 1.1

error validating data: found invalid field source for v1.Volume; if you choose to ignore these errors, turn validation off with --validate=false

    volumeMounts:
    - mountPath: /certs
      name: certs
  volumes:
  - name: storage
    source:
      emptyDir: {}
  - name: certs
    source:
      hostPath:
        path: /tmp

I am at tag 108f879 Fixes #5

I tried switching certs to source: emptyDir: {} but that didn't affect the error message

Sync with pires/kubernetes-elasticsearch-cluster

Latest changes to how Elasticsearch cluster is provisioned has broken this example.

Upgrade entire stack

Including:

• Elasticsearch 2.1.0
• Logstash 2.1.0
• Kibana 4.3.0

kubernetes-elk-cluster with coreos-kubernetes

Hi Paulo,

Hope u are doing well ! And thanks for your great repos !

I am looking to use the following set of tools with Kubernetes but having some problems with the authentication system on several levels. So I was wondering if you could help to solve how to integrate https://github.com/pires/kubernetes-elk-cluster into the https://github.com/coreos/coreos-kubernetes/ deployment in an easy way.

Here are the components that I would like to use:

Kubernetes Cluster setup: Coreos/AWS/TLS
https://github.com/coreos/coreos-kubernetes/blob/v0.1.0/Documentation/kubernetes-on-aws.md

Reverse Proxying:

• https://github.com/mailgun/vulcand

External Service Discovery:

• https://github.com/timelinelabs/romulus
  Pb: Not dealing with client certificate, so I cannot register new external IPs

Cluster-Monitoring/Addons

• https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-monitoring/influxdb (for the nodes)
• https://github.com/pires/kubernetes-elk-cluster (for specific services)

UIs visible from a public IP with a basic authentication (having a problem to get the IP public with using the Kube Proxy):

• Kube-UI (with the graph module)
• Graphana (cluster metrics)
• Kibana (custom logs processing)

How could I re-use the ssl certificate deployment of the coreos kurbenetes cluster with your elk replica controllers and services ?

Cheers,
Luc Michalski

Change label component

Right now we're reusing Kubernetes the label present in https://github.com/pires/kubernetes-elasticsearch-cluster:

{"component":"elasticsearch"}

We should change it to:

{"component":"elk"}

In order to do so, we need to replicate Elasticsearch cluster Kubernetes descriptors and made them available in this repository as well.

Replace logstash-forwarder with FileBeat

See the Migration Guide

Move to v1

• Move to API v1
• Replace JSON with YAML
• Upgrade to latest versions

How to connect co Kibana

Hi!

I'm unsure how to connect to Kibana once running. I have started the Vagrant cluster example and I can access the service from the inside the vbox image but not outside it.

Use Secrets to provision Logstash SSL certificates

http://kubernetes.io/v1.0/docs/user-guide/secrets.html

TLS IP SAN issues with logstash-forwarder

I've been seeing an issue that appears to be related to the move to Go 1.3 in logstash-forwarder with non-hostname based SSL certificates. To use IP addresses, there must be a subject with the appropriate IP,even if ssl strict verify is set to false.

See issues elastic/logstash-forwarder#230 and elastic/logstash-forwarder#221 (comment) for details.

If I'm reading these right, I would need to create the SSL cert for the logstash container after Kubernetes assigns the logstash service an IP address, then make sure the logstash and the logstash-forwarder get the new SSL cert.

Clarification about service-account

I'm a little confused about why you're creating the elk service-account. It doesn't look like it has any specific secrets associated with it. Why not just use the default service account?

SSL certificates file name

What kind (and names) of certificates is the container expecting?

Curious as to why this isn't supported/updated anymore?

This isn't a compliant but just more curiosity. I've used your elasticsearch in kube which has worked great. I've also gotten logstash setup in the k8s cluster and is sending logs. But just wanted to see if there was reasoning behind it (i.e. since only HTTP ingress is supported unless in GCE, not a good option for logstash, etc). Or just too much to maintain?

I just want to make sure if there are any pitfalls to running logstash in kube so I don't stumble across them myself.