sherifabdlnaby / rubban

Kibana Automatic Index Pattern Discovery and Other Elastic Stack Curating Tasks

Rubban - Kibana Automatic Index Pattern Discovery and Other Curating Tasks.

Introduction

Rubban for Elastic Stack is a curating tool that will automate tasks to make using Kibana a more pleasant experience.

Features

Automatic Index Pattern Discovery & Creation

In a dynamic environment where web services are created and deployed rapidly with all their logging infra set up, It is always annoying having to create Index Pattern for each service manually while everything else in the setup is automated. Rubban uses Kibana's client API's and Automate Index Pattern Creation and Discovery using General Patterns.

Automatic Refreshing for Index Pattern Fields

Kibana Index Pattern caches field names and types; when a new field is indexed after Index Pattern creation you won't be able to interact with it unless you refresh Index Pattern field list. Rubban can automate Index Pattern field list refreshing every set interval.

Refreshing resets the popularity counter of each field.

Automatic Creation for Dashboards

Still under development.

Currently tested on Kibana 7.0 and greater versions.

Examples for Automatic Index Pattern Discovery

Say your indices has the following convention for your apache access logs: logs-apache-access-<service-name>-<date> where <service-name> and <date> are dynamic based on the service and time. You can have logs-apache-access-serviceX-2020-02-02 and logs-apache-access-serviceY-2020-02-02 and you'll need to create index patterns logs-apache-access-serviceX-* and logs-apache-access-serviceX-* respectively to have them appear nicely in Kibana for developers.

If a new service is deployed and shipping its logs to Elasticsearch, you will need to still manually create the index pattern substituting just the service name. With Rubban you can configure a general pattern like logs-apache-access-?-* (notice ? is in the service-name place), and then Rubban will query Kibana and Elasticsearch periodically to look for indices that match this pattern *that's not covered under an Index Pattern.

Installation

Build from Source

1. Install Go (for macOS brew install go)
2. make build
3. ./bin/rubban

Docker

via Docker CLI

docker run  --env='RUBBAN_KIBANA_HOST=https://kibana:5601' \
            --env='RUBBAN_KIBANA_USER=elastic' \
            --env='RUBBAN_KIBANA_PASSWORD=changeme' \
            --env='RUBBAN_AUTOINDEXPATTERN_ENABLED=true' \
            --env='RUBBAN_AUTOINDEXPATTERN_SCHEDULE=*/5 * * * *' \
            --env='RUBBAN_AUTOINDEXPATTERN_GENERALPATTERNS=[{"pattern":"logs-apache-access-*-?","timeFieldName":"@timestamp"}]' \
             sherifabdlnaby/rubban:latest

via Docker Compose

docker-compose up -d

Configuration

• Configuration is in ./rubban.yml and file path can be overridden by the RUBBAN_CONFIG_DIR environment variable. (Configuration can be JSON, YAML, or TOML)
• Any configuration can be overridden with environment variables. ex: kibana.user: elastic can be overridden with RUBBAN_KIBANA_USER=elastic2.
  • Prefix key with RUBBAN_, ALL CAP all key, and replace . with _ and any _ to __.
  • Arrays can be declared in environment variables using 1. comma separated list, 2. json encoded array in a string.
  • Maps and objects can be declared in environment using a json encoded object in a string.

Kibana

kibana.host: Kibana Host (with Port). if HTTPS is enabled make sure to add https:// in the host. (default: http://localhost:5601)

kibana.user: Kibana User. Make sure user has privilege for Kibana configuration and setup.

kibana.password: Kibana User's Password. (It's advised to use RUBBAN_KIBANA_PASSWORD Env variable instead of adding it to config in plaintext)

Example:

kibana:
    host: kibana:5601
    user: elastic
    password: changeme

Automatic Index Pattern Discovery & Creation

autoIndexPattern.enabled: Enable/Disable Auto Index Discovery & Creation

autoIndexPattern.schedule: A Cron Expression that specify fixed schedule to run Auto Index Discovery & Creation. (default: */5 * * * * every 5 minutes)

autoIndexPattern.concurrency: Control How many Requests are made to Kibana API concurrently. (default: 20)

autoIndexPattern.generalPatterns: An array of General Pattern Objects, where pattern is the general pattern used to discover indices and timeFieldName is the time field that will be used for the created index pattern.

How do General Pattern works ?

A general pattern should be general for both indices names and index patterns (applies to them both). Unlike Kibana index pattern that can only contain wildcard *, general pattern has the ? wildcard. It will be used to find indices that doesn't belong to any index pattern.

If Kibana has currently logs-apache-access-serviceX-* index pattern and logs-apache-access-serviceX-2020-02-01 index. after a while logs-apache-access-serviceX-2020-02-02 and logs-apache-access-serviceY-2020-02-02 got created. the next time Rubban run with general pattern logs-apache-access-?-*, it will automatically create logs-apache-access-serviceY-* index pattern that covers the newly created index.

Example:

autoIndexPattern:
    enabled: true
    schedule: "* * * * *"
    concurrency: 20
    generalPatterns:
        -   pattern: logs-apache-access-*-?
            timeFieldName: "@timestamp"

Automatic Refreshing for Index Pattern Field

refreshIndexPattern.enabled: Enable/Disable Auto Refreshing for Index Pattern Field

refreshIndexPattern.schedule: A Cron Expression that specify fixed schedule to run Auto Index Discovery & Creation. (default: */5 * * * * every 5 minutes)

refreshIndexPattern.concurrency: Control How many Requests are made to Kibana API concurrently. (default: 20)

refreshIndexPattern.patterns: An array of Patterns, where each pattern can match multiple index patterns. Similar to General Patterns explained above but without ? matcher instead all uses *.

Example:

refreshIndexPattern:
    enabled: true
    schedule: "* * * * *"
    concurrency: 10
    patterns:
        - logstash-apache-*-*-*

Logging

logging:
    level: info       # any of (debug|info|warn|fatal)
    debug: false      # enable/disable debug config
    color: true       # show color in output (not for json format)
    format: console   # any of (console|logfmt|json)

License

MIT License Copyright (c) 2020 Sherif Abdel-Naby

Contribution

PR(s) are Open and Welcomed.

rubban's Issues

Consider renaming the project

Hello! First of all, thanks for the project! We have considered making something similar at $MY_CURRENT_COMPANY. However, there is another popular project with the same name at https://bosun.org/. Please consider renaming this to something else so that it wouldn't clash :P

Not an Issue, this is to host README.md assets ¯\_(ツ)_/¯

I can't get Rubban to create index patterns

Hi, I was wondering if you could help me with where I'm going wrong. I'm using elastdocker 3.2.0 with version 8.5.2. I thought Rubban sounded really useful and have been trying to set it up but it's not creating index patterns for me. I have indexes like:

logstash-dev-2022-12-09 
logstash-dev-2022-12-12 
logstash-test-2022-12-09 

My docker-compose.tools.yml is:

version: '3.5'

services:
  rubban:
    image: sherifabdlnaby/rubban:latest
    restart: unless-stopped
    environment:
      RUBBAN_KIBANA_HOST: "https://${KIBANA_HOST}:${KIBANA_PORT}"
      RUBBAN_KIBANA_USER: ${ELASTIC_USERNAME}
      RUBBAN_KIBANA_PASSWORD: ${ELASTIC_PASSWORD}
      RUBBAN_REFRESHINDEXPATTERN_ENABLED: 'true'
      RUBBAN_REFRESHINDEXPATTERN_SCHEDULE: '*/5 * * * *'
      RUBBAN_REFRESHINDEXPATTERN_PATTERNS: '*'
      RUBBAN_AUTOINDEXPATTERN_ENABLED: 'true'
      RUBBAN_AUTOINDEXPATTERN_SCHEDULE: '*/1 * * * *'
      RUBBAN_AUTOINDEXPATTERN_GENERALPATTERNS: '[{"pattern":"logstash-*","timeFieldName":"@timestamp"}]'

I've tried various patterns logstash-*, logstash-?-* and logstash-*-?. From the logs it looks like Rubban is running successfully, but it always just says "Successfully created 0 Index Patterns.".

elastic-rubban-1  | {"level":"info","ts":1670839006.0726402,"logger":"Rubban","caller":"rubban/rubban.go:52","msg":"Successfully Loaded Configuration"}
elastic-rubban-1  | {"level":"info","ts":1670839006.0726922,"logger":"Rubban","caller":"rubban/rubban.go:135","msg":"Initializing Kibana API client..."}
elastic-rubban-1  | {"level":"info","ts":1670839006.0727115,"logger":"Rubban.api","caller":"kibana/client.go:111","msg":"Testing connection to Kibana API at ***"}
elastic-rubban-1  | {"level":"info","ts":1670839006.089116,"logger":"Rubban.api","caller":"kibana/client.go:128","msg":"Successfully connected to Kibana API ***"}
elastic-rubban-1  | {"level":"info","ts":1670839006.0891774,"logger":"Rubban","caller":"rubban/rubban.go:147","msg":"Validated Initial Connection to Kibana API"}
elastic-rubban-1  | {"level":"info","ts":1670839006.1385114,"logger":"Rubban","caller":"rubban/rubban.go:155","msg":"Determined Kibana Version: 8.5.2"}
elastic-rubban-1  | {"level":"info","ts":1670839006.1386297,"logger":"Rubban","caller":"rubban/rubban.go:102","msg":"Enabled Auto Index Pattern, Loaded 1 General Pattern(s)"}
elastic-rubban-1  | {"level":"info","ts":1670839006.1386523,"logger":"Rubban","caller":"rubban/rubban.go:107","msg":"Enabled Refresh Indices Patterns, Refreshing 1 Pattern(s)"}
elastic-rubban-1  | {"level":"info","ts":1670839006.1386912,"logger":"Rubban.scheduler","caller":"rubban/scheduler.go:90","msg":"Registered Auto Index Pattern"}
elastic-rubban-1  | {"level":"info","ts":1670839006.138715,"logger":"Rubban.scheduler","caller":"rubban/scheduler.go:90","msg":"Registered Refresh Indices Patterns"}
elastic-rubban-1  | {"level":"info","ts":1670839006.1387486,"logger":"Rubban","caller":"rubban/rubban.go:78","msg":"Starting Rubban..."}
elastic-rubban-1  | {"level":"info","ts":1670839006.1387775,"logger":"Rubban.scheduler","caller":"rubban/scheduler.go:36","msg":"Starting scheduler..."}
elastic-rubban-1  | {"level":"info","ts":1670839006.138953,"logger":"Rubban.scheduler","caller":"rubban/scheduler.go:43","msg":"Next Auto Index Pattern run at 2022-12-12 09:57:00 +0000 UTC (13 seconds from now)"}
elastic-rubban-1  | {"level":"info","ts":1670839006.1389782,"logger":"Rubban.scheduler","caller":"rubban/scheduler.go:43","msg":"Next Refresh Indices Patterns run at 2022-12-12 10:00:00 +0000 UTC (3 minutes from now)"}
elastic-rubban-1  | {"level":"info","ts":1670839020.0003912,"logger":"Rubban.scheduler","caller":"rubban/scheduler.go:75","msg":"Running Auto Index Pattern..."}
elastic-rubban-1  | {"level":"info","ts":1670839020.054825,"logger":"Rubban.autoIndexPattern","caller":"autoindexpattern/task.go:55","msg":"Successfully created 0 Index Patterns.","Index Patterns":{}}
elastic-rubban-1  | {"level":"info","ts":1670839020.0549786,"logger":"Rubban.scheduler","caller":"rubban/scheduler.go:81","msg":"Finished Auto Index Pattern. (took ≈ 54ms)"}
elastic-rubban-1  | {"level":"info","ts":1670839020.0550067,"logger":"Rubban.scheduler","caller":"rubban/scheduler.go:82","msg":"Next Auto Index Pattern run at 2022-12-12 09:58:00 +0000 UTC (59 seconds from now)"}