inovex / kubernetes-logging

Demonstration of Kubernetes Logging with Fluentd and ElasticStack

Kubernetes Logging into Elasticsearch

This repository contains a docker images for a fluentd logshipper, kubernetes configs to deploy a basic elasticsearch cluster with kibana frontend, and documentation. These files should show how to setup a fluentd logshipper as kubernetes daemonset and pipe all container logs into an elasticsearch cluster. The logs are enriched with Metadata like pod_name, pod_id, docker_id. If your kubernetes application logs structured JSON log to STDOUT or STDERR the JSON is interpreted and each field of the JSON is a field in the elasticsearch index.

The content of this repository is based on the example given in the kubernetes repository. https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

Changes need for production

The file kubernetes_config/fluentd-daemonset.yaml contains the configuration for the fluentd logshipper. See the comments in this files for changes in production.

The fluentd_image/Makefile builds the docker container. Change the variables according to your docker registry setup.

The file kubernetes_config/fluentd_config/td-agent.conf contains the fluentd config. You might change it to consume also journald logs or further logs of the kubelet.

Demo

The demo relies on minikube. You have to setup minikube and then run:

minikube start
eval $(minikube docker-env)

The above will start a minikube cluster and setup the docker client in your current shell to the minikube cluster docker daemon.

To run this demo you have to create the image of the fluentd shipper in the minikube VM. Run the following commands in the shell where you ran the commands above:

cd fluentd_image
make build
cd ../es-image
make build
cd ../kibana-image
make build

To setup the Elasticsearch Cluster and a kibana frontend run:

kubectl create -f kubernetes_config/es-controller.yaml
kubectl create -f kubernetes_config/kibana-controller.yaml
kubectl create -f kubernetes_config/es-service.yaml
kubectl create -f kubernetes_config/kibana-service.yaml

To start the fluentd daemonset run:

kubectl create configmap fluentd-config --from-file=kubernetes_config/fluentd_config/td-agent.conf --namespace=kube-system
kubectl create -f kubernetes_config/fluentd-daemonset.yaml

After this setup you can check the pods you have deployed. The command is:

kubectl --namespace=kube-system get pods

The output should look like this:

NAME                             READY     STATUS    RESTARTS   AGE
elasticsearch-logging-v1-15qsf   1/1       Running   0          2m
elasticsearch-logging-v1-278v2   1/1       Running   0          2m
fluentd-logging-mcegp            1/1       Running   0          10s
kibana-logging-v1-mmfv4          1/1       Running   0          2m
kube-addon-manager-minikubevm    1/1       Running   0          4m
kubernetes-dashboard-ms0el       1/1       Running   0          4m

To look at your logs visit the kibana dashboard. You can get the URL to the kibana dashboard via:

minikube service -n kube-system --url kibana-logging

kubernetes-logging's Issues

Fluentd log level field

I read your blog, and a picture in your article that show kibana table. There is a "level" field.
Could you tell me how to add the "level" field?
My fluentd and es did not have this field.

DaemonSet fluentd-logging can't read container logs - Minikube

Versions

Hi !
I am running Minikube version: v0.22.3

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-09-29T05:56:06Z", GoVersion:"go1.9", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-10-06T20:53:14Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

What I have done

I executed run_demo.sh and everything was deployed (pods were running and configmap ok). 👌

What happened

However, I was't able to see logs onto Kibana.

When I execute the command kubectl --namespace kube-system logs -f fluentd-logging-z6lcd I get the following (one line for each container):

...
2017-10-23 19:30:16 +0000 [warn]: /var/log/containers/<container>.log unreadable. It is excluded and would be examined next time.
...

After googling a bit, I found out that the same issue was faced in other fluentd images (e.g. fabric8io/docker-fluentd-kubernetes#14).

So I compared your fluentd-daemonset.yaml with the solution proposed here.

It turns out that your fluentd-daemonset.yaml definition missed an entry in volumeMounts. In particular the following:

        volumeMounts:
        ...
        - name: var-lib-docker
          mountPath: /var/lib/docker/
          readOnly: true
        ...

And then in volumes:

      terminationGracePeriodSeconds: 30
      volumes:
      ...
      - name: var-lib-docker
        hostPath:
          path: /var/lib/docker/

Once I have added them and re-deploy the DaemonSet I got the following (as expected):

...
2017-10-23 19:20:07 +0000 [info]: following tail of /var/log/containers/<container>.log
...

And now I am able to see logs onto Kibana. Hope it helps. 😃
PS: Let me know if you want a PR.

Daemon not working

I am getting following error while running fluentd on minikube:

kubectl logs fluentd-24n22
chown: changing ownership of '/fluentd/etc/..2018_07_25_00_44_28.652122639/fluentd-configmap.yaml': Read-only file system
chown: changing ownership of '/fluentd/etc/..2018_07_25_00_44_28.652122639': Read-only file system
chown: changing ownership of '/fluentd/etc/..data': Read-only file system
chown: changing ownership of '/fluentd/etc/fluentd-configmap.yaml': Read-only file system
chown: changing ownership of '/fluentd/etc': Read-only file system
/var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/supervisor.rb:735:in `initialize': No such file or directory @ rb_sysopen - /fluentd/etc/fluent.conf (Errno::ENOENT)
	from /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/supervisor.rb:735:in `open'
	from /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/supervisor.rb:735:in `read_config'
	from /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/supervisor.rb:451:in `run_supervisor'
	from /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/command/fluentd.rb:310:in `<top (required)>'
	from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /var/lib/gems/2.3.0/gems/fluentd-0.14.21/bin/fluentd:5:in `<top (required)>'
	from /usr/local/bin/fluentd:22:in `load'
	from /usr/local/bin/fluentd:22:in `<main>'