Flume syslog source 2

This project defines a plugin for Flume containing an improved Syslog source.

It parses RFC 3164 (BSD Syslog) as well as the newer RFC 5424 (Syslog.) The source goes to a greater length to create a Flume event than the built-in version. All fields possible are parsed, and the body is just the body of the message. Even the Tag and PID fields are parsed. This should leave clean and normalized log messages, at the expense of more CPU cycles.

The data is interpreted as ASCII octets where possible. The notable exception is the hostname, which Flume stores as a String, and must thus be decoded. Where applicable, decoding to Unicode assumes the octet stream is encoded in UTF-8.

Usage

Build with mvn package. The produced Jar should be installed like any Flume plugin, and the plugin class to be added to flume.plugin.classes in the configuration file is called com.spotify.flume.syslog2.SyslogSource2Plugin.

The source is accessed using the name syslog2, and allows both TCP and UDP connections.

For TCP, the syntax is

syslog2(tcp[, host[, port]])

and for UDP

syslog2(udp[, host[, port[, bufferSize]]])

The default port is 514, the well-known Syslog port. The buffer size (which is the maximum size of an incoming UDP packet) is 64 kB unless overridden.

Caveats

Note that the structured data of RFC 5424 is parsed, but not split into multiple fields. This is for space effiency. If you need it split, add a sink decorator for it.

The code hasn’t been tested in Flume yet.

Implementation

The SyslogParser class is the main class and implements a recursive decent parser based on an InputStream. Most bytes are handled without concern for encoding, for speed and robustness. The philosophy is; it’s better to get the data into Flume, than rejecting it. This might not be suitable for you, but then you have the option of writing a decorator to handle that.

License

Copyright 2011 Spotify Ltd

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

jveiga / flume-syslog-source2 Goto Github PK