skjolber / json-log-domain

Library supporting JSON-logging with Logback and Stackdriver

json-log-domain

Library supporting JSON-logging. Currently working with Logback and logstash-logback-encoder and native Google Stackdriver.

Users will benefit from

• JSON-logging with domain-specific subtrees
• Simple YAML-based definition format
• User-friendly helper-classes generated via Maven or Gradle plugin
• Markdown documentation generator
• Elasticsearch configuration generator
• JAX-RS log-annotation for automatic MDC population

Multiple domains can be combined in the same log statement.

Bugs, feature suggestions and help requests can be filed with the issue-tracker.

License

Apache 2.0

Obtain

The project is based on Maven and is available from central Maven repository. See further down for dependencies.

Usage

Using static imports, log expressions can be simplified to (usually) a single line.

The generated sources allow for writing statements like

logger.info(system("fedora").tags(LINUX), "Hello world");

for SLF4J, or

LogEntry entry = DomainLogEntry.newBuilder(system("fedora").tags(LINUX).message("Hello world"))
    .setSeverity(Severity.INFO)
    .setLogName(logName)
    .setResource(MonitoredResource.newBuilder("global").build())
    .build();

for Stackdriver. Resulting in

{
  "message": "Hello world",
  "system": "fedora",
  "tags": ["linux"]
}

Multiple domains

Combine multiple domains in a single log statement via and(..):

logger.info(name("java").version(1.7).tags(JIT) // programming language
        .and(host("127.0.0.1").port(8080)) // network
        .and(system("Fedora").tags(LINUX)), // global
        "Hello world"); 

for SLF4J or

LogEntry entry = DomainLogEntry.newBuilder(
            name("java").version(1.7).tags(JIT)  // programming language
            .and(host("127.0.0.1").port(8080)) // network
            .and(system("Fedora").tags(LINUX).message("Hello world")) // global
        .setSeverity(Severity.INFO)
        .setLogName(logName)
        .setResource(MonitoredResource.newBuilder("global").build())
        .build();

for Stackdriver. This outputs domain-specific subtrees:

{
  "message": "Hello world",
  "language": {
    "name": "java",
    "version": 1.7,
    "tags": ["JIT"]
  },
  "network": {
    "port": 8080,
    "host": "127.0.0.1"
  },
  "system": "fedora",
  "tags": ["linux"]
}

where the global fields are at the root of the message.

MDC-style logging

Create AutoClosable scopes using

try (AutoCloseable a =  mdc(host("localhost").port(8080))) { // network
    logger.info().name("java").version(1.7).tags(JIT)  // programming language
        .and(system("Fedora").tags(LINUX)) // global
        .message("Hello world");
}

or the equivalent using try-finally;

Closeable mdc = mdc(host("localhost").port(8080); // network
try {
    ...
} finally {
    mdc.close();
}

Unlike the built-in SLF4J MDC, the JSON MDC works like a stack. For Logback, see Logback support artifact for configuration.

YAML definition format

The relevant fields and tags are defined in a YAML file, from which Java, Markdown and Elastic sources are generated.

Example definition:

version: '1.0'
name: Global
package: com.example.global
description: Global values
keys:
  - system:
      name: operating system name
      type: string
      description: The system name
      example: Ubuntu, Windows 10 or Fedora
  - memory:
      name: physical memory
      type: integer
      format: int32
      description: Physical memory in megabytes
      example: 1024, 2048, 16384
tags:
 - linux: Linux operating system
 - mac: Apple operating system
 - windows: Microsoft windows operating system

The definition format consists of the fields

• version - file version
• name - domain name (will prefix generated java sources)
• package - package of generated sources
• qualifier - name of domain subtree in logged JSON output (optional)
• description - textual description of domain
• keys - list of key-value definitions (see below).
• tags - list of tag definitions (see below)

In the above JSON example output, the optional qualifier corresponds to network and language while keys include system, port, host, version.

Keys

Each key is defined by:

• name - name of field (Operting System etc)
• type - datatype (string, number, integer etc)
• format - datatype subformat (int32, int64 etc)
• description - textual description of key
• example - example of legal value

The list item itself is the key in the logged key-value. The type/format datatype definition is borrowed from Swagger Code Generator. The intention is that log statements and REST services use the exact same definition for the same data type. Furthermore, framework-level interceptors should be able to pick up interesting fields in JSON objects and/or paths and automatically add those as context to a service invocation, saving the developer valuable time.

Tags

Each tag is defined by:

• name - a valid Java Enum name
• description - textual description of the tag

Generating useful outputs

Files in the above YAML format can be used to generate Java helper classes, Elastic message configuration and/or Markdown documents.

Maven plugin

YAML-files are converted to helper classes using log-domain-maven-plugin.

<plugin>
    <groupId>com.github.skjolber.log-domain</groupId>
    <artifactId>log-domain-maven-plugin</artifactId>
    <version>1.0.3</version>
    <executions>
        <execution>
            <id>generate</id>
            <goals>
                <goal>generate</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <outputDirectory>${project.build.directory}/generated-sources/domain-log-codegen</outputDirectory>
        <types>
            <markdown>true</markdown>
            <java>
                <logback>true</logback>
                <!-- OR -->
                <stackDriver>true</stackDriver>
            </java>
        </types>    
        <domains>
            <domain>
                <path>${basedir}/src/main/resources/yaml/network.yaml</path>logger
            </domain>
        </domains>
    </configuration>
</plugin>

Gradle plugin

Add Gradle plugin using

plugins {
  id "com.github.skjolber.json-log-domain" version "1.0.3"
}

and configure a jsonLogDomain task

jsonLogDomain { 
	definitions = files('src/main/resources/network.yaml')
	
	logback {
	}
}

sourceSets {
    main.java.srcDirs += [jsonLogDomain.logback.outputDirectory]
}

with the addition of markdown, elastic and stackDriver configurations for additional generation.

Multi-domain

In a multi-domain setup, the recommended approach is to generate per-domain artifacts, so that each project only generates helper classes for its own application-specific YAML file and accesses the helper classes for the other domains via a Gradle/Maven dependency.

Support-library

A few common classes are not part of the generated sources:

<dependency>
    <groupId>com.github.skjolber.log-domain</groupId>
    <artifactId>log-domain-support-logback</artifactId>
    <version>1.0.3</version>
</dependency>

or

<dependency>
    <groupId>com.github.skjolber.log-domain</groupId>
    <artifactId>log-domain-support-stackdriver</artifactId>
    <version>1.0.3</version>
</dependency>

Markdown documentation

A markdown file can also be generated for online documentation.

Elasticsearch configuration

Elasticsearch properties can be generated. One or more of these files can be combined into an application-specific message field mapping, typically at deploy time. See Elastic example.

Testing

Logging is an essential part of any application, verify that logging is performed during unit testing using the test libraries.

Alternatives

If you do not like this prosject, maybe you'll like

• godaddy-logger
• slf4jtesting
• slf4j-json-logger
• logback-more-appenders

See also

• Logbook - configurable request/response-logging

History

• 1.0.3: Stackdriver and Gradle support, minor improvements.
• 1.0.2: JAX-RS helper library, various improvements.
• 1.0.1: Added MDC support, various improvements.
• 1.0.0: Initial version