anchorfree / fluentbit-ip-cleanup-filter

Fluent-bit filter plugin written in Lua to anonymize IPv4 and IPv6 addresses from log records

Provides a Lua filter for fluent-bit to anonymize IPv4 and IPv6 addresses from log records.

Motivation

Most filters and plugins from fluent-bit/fluentd ecosystem assume a prior knowledge of the structure of incoming log records. However, in environment with multiple sources of logs which are not under your control, there is no way to know upfront the key names or spectrum of possible values (i.e. how IP address is enclosed in the value: simple <ipaddr> or <ipaddr>:<port> or even <some-data><ipaddr><some-data>). Thus, there is no way to apply a default fluent-bit's Modify filter or any of the myriad of fluentd plugins to solve given problem.

fluentbit-ip-cleanup-filter implements fluent-bit's Lua filter plugin that finds all IPv4 and IPv6 entries in the record (no matter of it's structure) using LPeg and replaces them with values provided in configuration.

Benchmarks

Section in progress

Usage

Docker

• Clone this repo and cd into it
• Run fluent-bit with the sample configuration (fluent-bit.conf) mounting current directory inside container:

# Contents of fluent-bit.conf
[INPUT]
    Name   dummy
    Dummy {"ipPort":"127.0.0.1:3233", "email":"[email protected]"}
    Tag    dummy.log

[FILTER]
    Name            lua
    Match           *
    Protected_mode  false
    script          cleanup_ip.lua
    call            clean

[OUTPUT]
    Name   stdout
    Match  *

$ docker run -ti --rm -v $PWD:/fluent-bit/etc -e VENDOR_PATH="/fluent-bit/etc/" fluent/fluent-bit
Fluent Bit v1.4.6
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/06/30 16:05:19] [ info] [storage] version=1.0.3, initializing...
[2020/06/30 16:05:19] [ info] [storage] in-memory
[2020/06/30 16:05:19] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/06/30 16:05:19] [ info] [engine] started (pid=1)
[2020/06/30 16:05:19] [ info] [sp] stream processor started
[0] dummy.log: [1593533120.216797700, {"ipPort"=>"0.0.0.0:3233", "email"=>"[email protected]"}]
[1] dummy.log: [1593533121.216744100, {"ipPort"=>"0.0.0.0:3233", "email"=>"[email protected]"}]
[2] dummy.log: [1593533122.217766100, {"ipPort"=>"0.0.0.0:3233", "email"=>"[email protected]"}]
[3] dummy.log: [1593533123.219193500, {"ipPort"=>"0.0.0.0:3233", "email"=>"[email protected]"}]

Kubernetes

Starting from v2.10.0, the official fluent-bit chart supports init containers via initContainers value. Below is the example of values.yaml that installs fluentbit-ip-cleanup-filter before starting fluent-bit:

env:
  # https://github.com/AnchorFree/fluentbit-ip-cleanup-filter#configuration
  - name: VENDOR_PATH
    value: &plugin_path /fluent-bit/plugins

extraVolumes:
  - name: &vol_name plugins
    emptyDir:
      medium: Memory
      sizeLimit: 5Mi

extraVolumeMounts:
  - name: *vol_name
    mountPath: *plugin_path

# https://docs.fluentbit.io/manual/pipeline/filters/lua
extraEntries:
  filter: |-
    [FILTER]
        Name            lua
        Match           *
        script          /fluent-bit/plugins/cleanup_ip.lua
        call            clean

initContainers:
  load-plugin:
    image: "appropriate/curl:latest"
    imagePullPolicy: "IfNotPresent"
    volumeMounts:
      - name: *vol_name
        mountPath: *plugin_path
    command:
      - "/bin/sh"
      - "-c"
      - |
        curl -sS https://codeload.github.com/AnchorFree/fluentbit-ip-cleanup-filter/zip/master -o /plugin.zip
        unzip /plugin.zip
        cp -av /fluentbit-ip-cleanup-filter-master/* /fluent-bit/plugins/

Configuration

It is possible to configure script parameters via environment variables:

VENDOR_PATH - path to ./vendor directory (relies on how volumes are mounted in container)
IPV4_REPL   - replacement for IPv4 addresses (default: 0.0.0.0)
IPV6_REPL   - replacement for IPv6 addresses (default: 0000:0000:0000:0000:0000:0000:0000:0000)

Compatibility

Requires at least fluent-bit v1.4.3 (supports code 2 for plugin return values).

Development

Update ./vendor

Currently containers from fluent/fluent-bit are running Lua 5.1 under Linux. Thus, ./vendor should be built under linux, using Lua 5.1 and luarocks:

$ luarocks install --tree vendor lpeg
$ luarocks install --tree vendor lpeg_patterns
$ luarocks install --tree vendor lunajson

Local development

Install dev dependencies:

$ make install-dev

Lint code (requires luacheck):

$ make lint

Format code (requires lua-format):

$ make format

Run tests:

$ make test

fluentbit-ip-cleanup-filter's Issues

Add benchmarks

Need to have figures on how many records/s it can handle. Especially it would be useful to compare performance by vendoring different Lua JSON libs (like here: https://github.com/grafi-tt/lunajson#benchmark).

Config param to enable/disable IPv4 / IPv6 filtering

Currently both IPv4 and IPv6 records are filtered and this is not configurable.