Coder Social home page Coder Social logo

fluent-bit-kubernetes-daemonset's Introduction

Help to Test!

Fluent Bit v0.11 was just released at the end of March 2017 and it comes with support for Kubernetes logging needs: it can consume logs and enrich them with proper Kubernetes metadata, retrieved from the API server of course.

Our goal is to make logging cheaper in terms of memory consumption. In order to accomplish this we need to deal with different factors, not only processing incoming logs but also formatting the data for our backends like Elasticsearch, which can be a little expensive as it requires a JSON representation.

Note: Fluent Bit uses a binary representation for logs, converting this to JSON requires enough memory for the process.

Getting started

Fluent Bit must be deployed as a DaemonSet, on that way it will be available on every node of your Kubernetes cluster.

This repository contains two Yaml DaemonSet files:

Yaml file Description
fluent-bit-daemonset-elasticsearch deploys a stable release of Fluent Bit.

The current DaemonSet points to this specific Docker Hub image:

0.11 fluent/fluent-bit-kubernetes-daemonset:0.11

Steps

  1. Make sure your Elasticsearch backend is running and can be reach through the hostname elasticsearch-logging. This value can be changed in the Yaml file

  2. Deploy the daemonset file from this repository:

$ kubectl apply -f fluent-bit-daemonset-elasticsearch.yaml

Details

The default configuration of Fluent Bit makes sure of the following:

  • Consume all containers logs from the running Node.
  • The Tail input plugin will not append more than 5MB into the engine until they are flushed to the Elasticsearch backend. This limit aims to provide a workaround for backpressure scenarios.
  • The Kubernetes filter will enrigh the logs with Kubernetes metadata, specifically labels and annotations. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.
  • The default backend in the configuration is Elasticsearch set by the Elasticsearch Ouput Plugin. It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.
  • There is an option called Retry_Limit set to False, that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeed.

Get back to us!

Your contribution on testing is highly appreciated, we aim to make logging cheaper for everybody so your feedback is fundamental, try to get back to us on:

fluent-bit-kubernetes-daemonset's People

Contributors

edsiper avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.