redborder / events-counter

Simple utility for counting messages on a Kafka topic.

events-counter

• Overview
• Installing
• Usage
• Configuration

Overview

events-counter is a simple utility that can be used for accounting messages in a Kafka topic. It supports accounting by number of bytes (regardless the numer of messages).

Messages are expected to be on JSON format when counting messages.

events-counter has two independent modules:

• UUIDCounter
  • Reads messages from a set of input topics.
  • Count the number of bytes of these messages.
  • Sends a message periodically to an output topic with the account of bytes on the inputs topic.
• CountersMonitor
  • The input topic of the counter monitor is the output topic of the UUIDCounter.
  • Reads messages with the account of bytes. Discards old messages.
  • If the number of total bytes exceeds the limit sends an alert.

events-counter ignores Teldat sensors traffic, so these kind of sensors can send an unlimited amount of bytes.

License

events-counter limits can't be configured. These limits are loaded from a LICENSE file. These licenses must be signed to be considered a valid license. Also, a license has an expiration time and it won't be considered valid after the expiration time.

The public key used to verify the license should be specified at compile time and can't be modified once the binary is compiled. To specify the public key, the application should be compiled like this:

export PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\nMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCqGKukO1De7zhZj6+H0qtjTkVxwTCpvKe4eCZ0\nFPqri0cb2JZfXJ/DgYSF6vUpwmJG8wVQZKjeGcjDOL5UlsuusFncCzWBQ7RKNUSesmQRMSGkVb1/\n3j+skZ6UtW+5u09lHNsj6tQ51s1SPrCBkedbNf0Tp0GbMJDyR4e9T04ZZwIDAQAB\n-----END PUBLIC KEY-----"
make

Note that every line break should be replaced by "\n"

Installing

To install this application ensure you have the GOPATH environment variable set and glide installed.

curl https://glide.sh/get | sh

And then:

1. Clone this repo and cd to the project

   git clone https://github.com/redBorder/events-counter && cd events-counter

2. Install dependencies and compile

   make

3. Install on desired directory

   prefix=/opt/events-counter/ make install

Usage

Usage of events-counter:

--version
    Show version info
--config string
    Config file
--debug
    Print debug info

Configuration

counters:
  batch_timeout_s: 5                     # Max time to wait before send a count message
  batch_max_messages: 1000               # Max number of messages to hold before send a count message
  uuid_key: "sensor_uuid"                # JSON key for the UUID
  kafka:                                 # Kafka configuration
    write_topic: "rb_monitor"            # Topic to send the count
    read_topics:                         # Topics to read messages for accounting
      - "rb_flow"
    attributes:                          # Custom internal rdkafka attributes
      bootstrap.servers: "kafka:9092"
      group.id: "counters"

monitor:
  timer:
    period: 86400                        # Width in seconds of the interval between counters reset (86400 -> 24h)
    offset: 0                            # Offset in seconds to change the start of the interval (0 -> 00:00h)
  kafka:                                 # Kafka configuration
    write_topic: "rb_limits"             # Topic to send the alerts
    read_topics:                         # Topics to read messages with accounting info
      - "rb_monitor"
    attributes:                          # Custom internal rdkafka attributes
      bootstrap.servers: "kafka:9092"
      enable.auto.commit: "false"        # IMPORTANT: Should be set to false
      group.id: "monitor"

licenses_directory: /etc/licenses       # Path to the directory where licenses are stored
organization_mode: false                # Ignore organization_uuid. Count every message as belonging to org "*"

events-counter's Issues

Count traffic for organizations

Configuration for choosing between organization mode and normal mode

Handle SIGHUP signal to reload licenses

events-counter should handle a SIGHUP signal and do the following:

• Reload licenses
• Send a counters_reset notification

Traffic from Teldat sensors should be ignored

Identify Teldat sensors and ignore the bytes coming from them since these kind of sensors should not have any limit.

Reset signal should not be sent when no licenses are available

Make events-counter specify the organization_uuid when sending a signal to reset counters. The reset signal should only be sent when there is at least one license for the organization.

The organization "*" is the "global organization". A reset signal for the global organization unblocks all sensors.

---

#redmine-9701

Rename "uuid" field

Change "uuid" to "organization_uuid" in counters monitor messages

Count messages by license UUID instead orgaization UUID

Send an alert to inform that a license has expired

When a license has exired, a signal should be sent. The format should be the following:

{
  "monitor": "alert",
  "type": "license_expired",
  "uuid": "<license uuid for the expired license>",
  "timestamp": 1497955816
}

---

#redmine-9895

Use license file

Count message per organization

• Organization mode
  • Messages from org A are accounted to org A if there is a license for this org.
  • Messages from org B are not accounted if there is no license for org B.
  • Messages with no organization are not accounted.
• Normal mode:
  • All messages are accounted for the generic organization * regardless the real organization (which may be null).

Sprint 3

Counter component

Functionality for count the number of bytes on a message.

Read limit configuration from license file instead config file

Read organization licenses

Monitor messages

Create a component for monitor the count of messages and send an alert to notify that the max number of allowed messages has been reached.

Main application

Basic functionality

Create a service that reads JSON messages on a Kafka topic and when a specified limit has been reached it should send an alert message on another Kafka topic so another service can perform the proper action to block the data coming from the sensor.

Kafka producer component