A Logback appender that encodes logs to GELF and transports them to Graylog servers.
Disclaimer: This repo is no longer maintained as of v0.3 as I've stopped using Graylog (different job). Please reach out at [email protected] if you wish to take over the repo. Some alternative GELF appenders for logback are listed below
- osiegmar/logback-gelf
- rkcpi/logback-gelf-appender
- potentially more on the graylog marketplace
Latest version:
<dependency>
<groupId>me.moocar</groupId>
<artifactId>logback-gelf</artifactId>
<version>0.3</version>
</dependency>- Append via TCP or UDP (with chunking) to a remote graylog server
- MDC k/v converted to fields
- Fields may have types
- Auto include logger_name
- Auto include Markers
- Auto include Thread name
- Static fields (E.g facility)
- Very Few dependencies (Logback and GSON)
Note, 0.2 is a breaking version. It is NOT compatible with 0.12 and previous versions. Read about the changes here
The minimal possible logback.xml you can write is something like.
<configuration>
<appender name="GELF UDP APPENDER" class="me.moocar.logbackgelf.GelfUDPAppender">
<encoder class="me.moocar.logbackgelf.GZIPEncoder">
<layout class="me.moocar.logbackgelf.GelfLayout"/>
</encoder>
</appender>
<root level="debug">
<appender-ref ref="GELF UDP APPENDER" />
</root>
</configuration>A more complete example that shows how you would overwrite many default values:
<configuration>
<!--Use TCP instead of UDP-->
<appender name="GELF TCP APPENDER" class="me.moocar.logback.net.SocketEncoderAppender">
<remoteHost>somehost.com</remoteHost>
<port>12201</port>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="me.moocar.logbackgelf.GelfLayout">
<!--An example of overwriting the short message pattern-->
<shortMessageLayout class="ch.qos.logback.classic.PatternLayout">
<pattern>%ex{short}%.100m</pattern>
</shortMessageLayout>
<!-- Use HTML output of the full message. Yes, any layout can be used (please don't actually do this)-->
<fullMessageLayout class="ch.qos.logback.classic.html.HTMLLayout">
<pattern>%relative%thread%mdc%level%logger%msg</pattern>
</fullMessageLayout>
<useLoggerName>true</useLoggerName>
<useThreadName>true</useThreadName>
<useMarker>true</useMarker>
<host>Test</host>
<additionalField>ipAddress:_ip_address</additionalField>
<additionalField>requestId:_request_id</additionalField>
<includeFullMDC>true</includeFullMDC>
<fieldType>requestId:long</fieldType>
<!--Facility is not officially supported in GELF anymore, but you can use staticFields to do the same thing-->
<staticField class="me.moocar.logbackgelf.Field">
<key>_facility</key>
<value>GELF</value>
</staticField>
</layout>
</encoder>
</appender>
<root level="debug">
<appender-ref ref="GELF TCP APPENDER" />
</root>
</configuration>me.moocar.logbackgelf.GelfLayout
This is where most configuration resides, since it's the part that actually converts a log event into a GELF compatible JSON string.
- useLoggerName: If true, an additional field call "_loggerName"
will be added to each gelf message. Its contents will be the fully
qualified name of the logger. e.g:
com.company.Thingo. Default:false - useThreadName: If true, an additional field call "_threadName"
will be added to each gelf message. Its contents will be the name of
the thread. Default:
false - host The hostname of the host from which the log is being sent.
Displayed under
sourceon web interface. Default:getLocalHostName() - useMarker: If true, and the user has set a
slf4j Marker on their
log, then the marker.toString() will be added to the gelf message
as the field "_marker". Default:
false - shortMessageLayout: The
Layout used to create
the gelf
short_messagefield. Shows up in the message column of the log summary in the web interface. Default:"%ex{short}%.100m"(PatternLayout) - fullMessageLayout: The
Layout used to create
the gelf
full_messagefield. Shows up in the message field of the log details in the web interface. Default:"%rEx%m"(PatternLayout) - additionalFields: See additional fields below. Default: empty
- fieldType: See field type conversion below. Default: empty (fields sent as string)
- staticFields: See static fields below. Note, now that facility is deprecated, use this to set a facility Default: empty
- staticAdditionalFields: deprecated. Use staticFields. Default: empty
- includeFullMDC: See additional fields below. Default:
false
Both UDP and TCP transports are supported. UDP is the recommended graylog transport.
UDP can be configured using the
me.moocar.logbackgelf.GelfUDPAppender appender. Once messages reach
a certain size, they will be chunked according to the
gelf spec. A maximum of
128 chunks can be sent per log. If the encoded log is bigger than
that, the log will be dropped. Assuming the default 512 max packet
size, this allows for 65536 bytes (64kb) total per log message
(unzipped).
- remoteHost: The remote graylog server host to send log messages
to (DNS or IP). Default:
"localhost" - port: The remote graylog server port. Default:
12201 - maxPacketSize: The maximum number of bytes per datagram packet.
Once the limit is reached, packets will be chunked. Default:
512
GZIP
For UDP, you have the option of Gzipping the Gelf JSON before sending
over UDP. To do this, replace the
ch.qos.logback.core.encoder.LayoutWrappingEncoder encoder with the
me.moocar.logbackgelf.GZIPEncoder encoder. E.g
<appender name="GELF UDP APPENDER" class="me.moocar.logbackgelf.GelfUDPAppender">
<encoder class="me.moocar.logbackgelf.GZIPEncoder">
<layout class="me.moocar.logbackgelf.GelfLayout"/>
</encoder>
</appender>Remember, The GZIP encoder should NOT be used with TCP
TCP transport can be configured using the
me.moocar.logback.net.SocketEncoderAppender appender. Unfortunately,
the built in Logback Socket
Appender
doesn't give you control of how logs are encoded before being sent
over TCP, which is why you have to use this appender. To make the
system as flexible as possible, I moved this new appender into its
own library. Note
that due to an unresolved Graylog
issue, GZIP
is not supported when using TCP.
<appender name="GELF TCP APPENDER" class="me.moocar.logback.net.SocketEncoderAppender">
<port>12201</port>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="me.moocar.logbackgelf.GelfLayout">
....
</layout>
</encoder>
</appender>- remoteHost: The remote graylog server host to send log messages
to (DNS or IP). Default:
"localhost" - port: The remote graylog server port. Required.
- maxPacketSize: The number of logs to keep in memory while the
graylog server can't be reached. Default:
128 - acceptConnectionTimeout: Milliseconds to wait for a connection
to be established to the server before failing. Default:
1000
Additional Fields are extra k/v pairs that can be added to the GELF json, and thus searched as structured data using graylog. In the slf4j world, MDC (Mapped Diagnostic Context) is an excellent way of programmatically adding fields to your GELF messages.
Let's take an example of adding the ip address of the client to every logged message. To do this we add the ip address as a key/value to the MDC so that the information persists for the length of the request, and then we inform logback-gelf to look out for this mapping every time a message is logged.
- Store IP address in MDC
// Somewhere in server code that wraps every request
...
org.slf4j.MDC.put("ipAddress", getClientIpAddress());
...- Inform logback-gelf of MDC mapping
<layout class="me.moocar.logbackgelf.GelfLayout">
<additionalField>ipAddress:_ip_address</additionalField>
</layout>If the property includeFullMDC is set to true, all fields from the
MDC will be added to the gelf message. Any key, which is not listed as
additionalField will be prefixed with an underscore. Otherwise the
field name will be obtained from the corresponding additionalField
mapping.
If the property includeFullMDC is set to false (default value) then
only the keys listed as additionalField will be added to a gelf
message.
Use static additional fields when you want to add a static key value pair to every GELF message. Key is the additional field key (and should thus begin with an underscore). The value is a static string.
Now that the GELF facility is deprecated, this is how you add a
static facility. StaticFields replace staticAdditionalFields
E.g in the appender configuration:
<layout class="me.moocar.logbackgelf.GelfLayout">
<staticField class="me.moocar.logbackgelf.Field">
<key>_facility</key>
<value>GELF</value>
</staticField>
<staticField class="me.moocar.logbackgelf.Field">
<key>_node_name</key>
<value>www013</value>
</staticField>
</layout>Static Additional fields have been deprecated and superceded by staticFields. While they offered a more concise way of expressing the key/value pair, it was impossible to include a colon in the value. staticFields are fully structured and don't have this problem.
You can configure a specific field to be converted to a numeric type.
Key is the additional field key as inserted into the MDC, value is the
type to convert to. Currently supported types are int, long, float and double.
<layout class="me.moocar.logbackgelf.GelfLayout">
<additionalField>requestId:_request_id</additionalField>
<fieldType>requestId:long</fieldType>
</layout>If the conversion fails, logback-gelf will leave the field value alone (i.e.: send it as String) and print the stacktrace
Logback-gelf tests are written in Clojure. I could try and justify this by saying that the tests are better than they were in java, which is true, but mostly it's because I love Clojure and have been using it in my day job for over 3 years. I'm aware this will make it harder for others to submit PRs, and that pains me a little, but look, treat it as an exercise. Clojure will make you a better programmer.
The good news is that running the tests is just as easy as before.
Simply run mvn test.
The other good news, is that we're now using generative testing via test.check. Tests are generated based on specs and then properties are asserted. It means that much more of the possible value space is tested.
If you want to test interactively, you can start a repl using mvn clojure:repl, or if you want to connect via nrepl, use mvn clojure:nrepl
In logback-gelf v0.2, configuration has been reworked to fit better into the logback ecosystem. The primary driver was adding TCP transport. Under 0.12 configuration, a transport option would have been added to the main appender, but then there would be no logical place to put TCP specific configuration such as connectTimeout. UDP also has its own quirks, requiring chunking and the option of GZIP.
So the new configuration follows the logback way and provides both UDP and TCP appenders, and the GELF serialization logic is now in a GelfLayout. This required a significant refactor but will provide more flexibility going forward. For example, adding a Kafka or AMPQ appender should now be trivial, and you won't even need to modify this library.
Because it's such a big change, I deliberately broke backwards compatibility. Here's a list of all the changes:
me.moocar.logbackgelf.GelfAppenderhas been removed and replaced withme.moocar.logbackgelf.GelfUDPAppenderandme.moocar.logback.net.SocketEncoderAppender(all non transport configuration is now under theGelfLayoutLayout)- graylog2ServerHost is now
GelfUDPAppender.remoteHostorSocketEncoderAppender.remoteHost - graylog2ServerPort is now
GelfUDPAppender.portorSocketEncoderAppender.port - facility is deprecated in GELF 1.1, so I've removed it. It can be added using Static Additional Fields.
- hostName is now host (to be inline with Gelf spec)
- graylog2ServerVersion no longer exists since it's assumed that you are using graylog 1.0 or above.
- chunkThreshold is now
GelfUDPAppender.maxPacketSizeand the default is 512 bytes. - messagePattern is now fullMessageLayout and is no longer assumed to be a PatternLayout, rather it is a generic Layout, giving you full flexibility over how a message string is transformed. For example, if you really wanted, you could use a XML or HTML layout.
- shortMessagePattern same as above. Note that auto truncation no longer occurs. You must now describe how to format the log message using Layouts.
- Testing is now in Clojure
- Development version 0.4-SNAPSHOT (current Git
master) - Release [0.3] on 2016-Jan-19
- Add Structured Static Fields #57
- Release [0.2beta3] on 2015-May-31
- Handle IO errors in UDP appender #54
- Release [0.2beta2] on 2015-Apr-06
- added clojars repository
- added TCP socket encoder to dependencies
- Release [0.2beta1] on 2015-Apr-05
- Breaking change version
- Refactor to support multiple appenders
- UDP config now under GelfUDPAppender
- View complete changes
- Release [0.12] on 2014-Nov-04
- Explicitly set Zipper string encoding to UTF-8 #41
- Release [0.11] on 2014-May-18
- Release [0.10p2] on 2014-Jan-12
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent