Logbook noun, /lɑɡ bʊk/: A book in which measurements from the ship's log are recorded, along with other salient details of the voyage.
Logbook is an extensible Java library to enable complete request and response logging for different client- and server-side technologies. It satisfies a special need by a) allowing web application developers to log any HTTP traffic that an application receives or sends b) in a way that makes it easy to persist and analyze it later. This can be useful for traditional log analysis, meeting audit requirements or investigating individual historic traffic issues.
Logbook is ready to use out of the box for most common setups. Even for uncommon applications and technologies, it should be simple to implement the necessary interfaces to connect a library/framework/etc. to it.
- Logging: of HTTP requests and responses, including the body; partial logging (no body) for unauthorized requests
- Customization: of logging format, logging destination, and conditions that request to log
- Support: for Servlet containers, Apache’s HTTP client, Square's OkHttp, and (via its elegant API) other frameworks
- Optional obfuscation of sensitive data
- Spring Boot Auto Configuration
- Scalyr compatible
- Sensible defaults
- Java 8
- Any build tool using Maven Central, or direct download
- Servlet Container (optional)
- Apache HTTP Client (optional)
- JAX-RS 2.x Client and Server (optional)
- Netty 4.x (optional)
- OkHttp 2.x or 3.x (optional)
- Spring 4.x or 5.x (optional)
- Spring Boot 1.x or 2.x (optional)
- logstash-logback-encoder 5.x (optional)
Add the following dependency to your project:
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-core</artifactId>
<version>${logbook.version}</version>
</dependency>
Additional modules/artifacts of Logbook always share the same version number.
Alternatively, you can import our bill of materials...
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-bom</artifactId>
<version>${logbook.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
... which allows you to omit versions:
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-core</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-httpclient</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-jaxrs</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-json</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-netty</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-okhttp</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-okhttp2</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-servlet</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-logstash</artifactId>
</dependency>
The logbook logger must be configured to trace level in order to log the requests and responses. With Spring Boot 2 (using Logback) this can be accomplised by adding the following line to your application.properties
logging.level.org.zalando.logbook: TRACE
All integrations require an instance of Logbook
which holds all configuration and wires all necessary parts together.
You can either create one using all the defaults:
Logbook logbook = Logbook.create();
or create a customized version using the LogbookBuilder
:
Logbook logbook = Logbook.builder()
.condition(new CustomCondition())
.queryFilter(new CustomQueryFilter())
.pathFilter(new CustomPathFilter())
.headerFilter(new CustomHeaderFilter())
.bodyFilter(new CustomBodyFilter())
.requestFilter(new CustomRequestFilter())
.responseFilter(new CustomResponseFilter())
.sink(new DefaultSink(
new CustomHttpLogFormatter(),
new CustomHttpLogWriter()
))
.build();
Logbook used to have a very rigid strategy how to do request/response logging:
- Requests/responses are logged separately
- Requests/responses are logged soon as possible
- Requests/responses are logged as a pair or not logged at all
(i.e. no partial logging of traffic)
Some of those restrictions could be mitigated with custom HttpLogWriter
implementations, but they were never ideal.
Starting with version 2.0 Logbook now comes with a Strategy pattern
at its core. Make sure you read the documentation of the Strategy
interface to understand the implications.
Logbook comes with some built-in strategies:
Logbook works in several different phases:
Each phase is represented by one or more interfaces that can be used for customization. Every phase has a sensible default.
Logging HTTP messages and including their bodies is a rather expensive task, so it makes a lot of sense to disable logging for certain requests. A common use case would be to ignore health check requests from a load balancer, or any request to management endpoints typically issued by developers.
Defining a condition is as easy as writing a special Predicate
that decides whether a request (and its corresponding response) should be logged or not. Alternatively you can use and combine predefined predicates:
Logbook logbook = Logbook.builder()
.condition(exclude(
requestTo("/health"),
requestTo("/admin/**"),
contentType("application/octet-stream"),
header("X-Secret", newHashSet("1", "true")::contains)))
.build();
Exclusion patterns, e.g. /admin/**
, are loosely following Ant's style of path patterns
without taking the the query string of the URL into consideration.
The goal of Filtering is to prevent the logging of certain sensitive parts of HTTP requests and responses. This usually includes the Authorization header, but could also apply to certain plaintext query or form parameters — e.g. password.
Logbook supports different types of filters:
Type | Operates on | Applies to | Default |
---|---|---|---|
QueryFilter |
Query string | request | access_token |
PathFilter |
Path | request | n/a |
HeaderFilter |
Header (single key-value pair) | both | Authorization |
BodyFilter |
Content-Type and body | both | json: access_token and refresh_token form: client_secret and password |
RequestFilter |
HttpRequest |
request | Replace binary, multipart and stream bodies. |
ResponseFilter |
HttpResponse |
response | Replace binary, multipart and stream bodies. |
QueryFilter
, PathFilter
, HeaderFilter
and BodyFilter
are relatively high-level and should cover all needs in ~90% of all
cases. For more complicated setups one should fallback to the low-level variants, i.e. RequestFilter
and ResponseFilter
respectively (in conjunction with ForwardingHttpRequest
/ForwardingHttpResponse
).
You can configure filters like this:
Logbook logbook = Logbook.builder()
.requestFilter(replaceBody(contentType("audio/*"), "mmh mmh mmh mmh"))
.responseFilter(replaceBody(contentType("*/*-stream"), "It just keeps going and going..."))
.queryFilter(accessToken())
.queryFilter(replaceQuery("password", "<secret>"))
.headerFilter(authorization())
.headerFilter(eachHeader("X-Secret"::equalsIgnoreCase, "<secret>"))
.build();
You can configure as many filters as you want - they will run consecutively.
Logbook uses a correlation id to correlate requests and responses. This allows match-related requests and responses that would usually be located in different places in the log file.
If the default implementation of the correlation id is insufficient for your use case, you may provide a custom implementation:
Logbook logbook = Logbook.builder()
.correlationId(new CustomCorrelationId())
.build();
Formatting defines how requests and responses will be transformed to strings basically. Formatters do not specify where requests and responses are logged to — writers do that work.
Logbook comes with two different default formatters: HTTP and JSON.
HTTP is the default formatting style, provided by the DefaultHttpLogFormatter
. It is primarily designed to be used for local development and debugging, not for production use. This is because it’s not as readily machine-readable as JSON.
Incoming Request: 2d66e4bc-9a0d-11e5-a84c-1f39510f0d6b
GET http://example.org/test HTTP/1.1
Accept: application/json
Host: localhost
Content-Type: text/plain
Hello world!
Outgoing Response: 2d66e4bc-9a0d-11e5-a84c-1f39510f0d6b
Duration: 25 ms
HTTP/1.1 200
Content-Type: application/json
{"value":"Hello world!"}
JSON is an alternative formatting style, provided by the JsonHttpLogFormatter
. Unlike HTTP, it is primarily designed for production use — parsers and log consumers can easily consume it.
Requires the following dependency:
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-json</artifactId>
</dependency>
{
"origin": "remote",
"type": "request",
"correlation": "2d66e4bc-9a0d-11e5-a84c-1f39510f0d6b",
"protocol": "HTTP/1.1",
"sender": "127.0.0.1",
"method": "GET",
"uri": "http://example.org/test",
"host": "example.org",
"path": "/test",
"scheme": "http",
"port": null,
"headers": {
"Accept": ["application/json"],
"Content-Type": ["text/plain"]
},
"body": "Hello world!"
}
{
"origin": "local",
"type": "response",
"correlation": "2d66e4bc-9a0d-11e5-a84c-1f39510f0d6b",
"duration": 25,
"protocol": "HTTP/1.1",
"status": 200,
"headers": {
"Content-Type": ["text/plain"]
},
"body": "Hello world!"
}
Note: Bodies of type application/json
(and application/*+json
) will be inlined into the resulting JSON tree. I.e.,
a JSON response body will not be escaped and represented as a string:
{
"origin": "local",
"type": "response",
"correlation": "2d66e4bc-9a0d-11e5-a84c-1f39510f0d6b",
"duration": 25,
"protocol": "HTTP/1.1",
"status": 200,
"headers": {
"Content-Type": ["application/json"]
},
"body": {
"greeting": "Hello, world!"
}
}
The Common Log Format (CLF) is a standardized text file format used by web servers when generating server log files. The format is supported via the CommonLogFormatSink
:
185.85.220.253 - - [02/Aug/2019:08:16:41 0000] "GET /search?q=zalando HTTP/1.1" 200 -
cURL is an alternative formatting style, provided by the CurlHttpLogFormatter
which will render requests as
executable cURL
commands. Unlike JSON, it is primarily designed for humans.
curl -v -X GET 'http://localhost/test' -H 'Accept: application/json'
See HTTP or provide own fallback for responses:
new CurlHttpLogFormatter(new JsonHttpLogFormatter());
Splunk is an alternative formatting style, provided by the SplunkHttpLogFormatter
which will render
requests and response as key-value pairs.
origin=remote type=request correlation=2d66e4bc-9a0d-11e5-a84c-1f39510f0d6b protocol=HTTP/1.1 sender=127.0.0.1 method=POST uri=http://example.org/test host=example.org scheme=http port=null path=/test headers={Accept=[application/json], Content-Type=[text/plain]} body=Hello world!
origin=local type=response correlation=2d66e4bc-9a0d-11e5-a84c-1f39510f0d6b duration=25 protocol=HTTP/1.1 status=200 headers={Content-Type=[text/plain]} body=Hello world!
Writing defines where formatted requests and responses are written to. Logbook comes with three implementations: Logger, Stream and Chunking.
By default, requests and responses are logged with an slf4j logger that uses the org.zalando.logbook.Logbook
category and the log level trace
. This can be customized:
Logbook logbook = Logbook.builder()
.sink(new DefaultSink(
new DefaultHttpFormatter(),
new DefaultHttpLogWriter())
.build();
An alternative implementation is to log requests and responses to a PrintStream
, e.g. System.out
or System.err
. This is usually a bad choice for running in production, but can sometimes be useful for short-term local development and/or investigation.
Logbook logbook = Logbook.builder()
.sink(new DefaultSink(
new DefaultHttpFormatter(),
new StreamHttpLogWriter(System.err)
))
.build();
The ChunkingSink
will split long messages into smaller chunks and will write them individually while delegating to another sink:
Logbook logbook = Logbook.builder()
.sink(new ChunkingSink(sink, 1000))
.build();
The combination of HttpLogFormatter
and HttpLogWriter
suits most use cases well, but it has limitations.
Implementing the Sink
interface directly allows for more sophisticated use cases, e.g. writing requests/responses
to a structured persistent storage like a database.
Multiple sinks can be combined into one using the CompositeSink
.
You’ll have to register the LogbookFilter
as a Filter
in your filter chain — either in your web.xml
file (please note that the xml approach will use all the defaults and is not configurable):
<filter>
<filter-name>LogbookFilter</filter-name>
<filter-class>org.zalando.logbook.servlet.LogbookFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>LogbookFilter</filter-name>
<url-pattern>/*</url-pattern>
<dispatcher>REQUEST</dispatcher>
<dispatcher>ASYNC</dispatcher>
</filter-mapping>
or programmatically, via the ServletContext
:
context.addFilter("LogbookFilter", new LogbookFilter(logbook))
.addMappingForUrlPatterns(EnumSet.of(REQUEST, ASYNC), true, "/*");
Beware: The ERROR
dispatch is not supported. You're strongly advised to produce error responses within the
REQUEST
or ASNYC
dispatch.
The LogbookFilter
will, by default, treat requests with a application/x-www-form-urlencoded
body not different from
any other request, i.e you will see the request body in the logs. The downside of this approach is that you won't be
able to use any of the HttpServletRequest.getParameter*(..)
methods. See issue #94 for some more
details.
As of Logbook 1.5.0, you can now specify one of three strategies that define how Logbook deals with this situation by
using the logbook.servlet.form-request
system property:
Value | Pros | Cons |
---|---|---|
body (default) |
Body is logged | Downstream code can not use getParameter*() |
parameter |
Body is logged (but it's reconstructed from parameters) | Downstream code can not use getInputStream() |
off |
Downstream code can decide whether to use getInputStream() or getParameter*() |
Body is not logged |
Secure applications usually need a slightly different setup. You should generally avoid logging unauthorized requests, especially the body, because it quickly allows attackers to flood your logfile — and, consequently, your precious disk space. Assuming that your application handles authorization inside another filter, you have two choices:
- Don't log unauthorized requests
- Log unauthorized requests without the request body
You can easily achieve the former setup by placing the LogbookFilter
after your security filter. The latter is a little bit more sophisticated. You’ll need two LogbookFilter
instances — one before your security filter, and one after it:
context.addFilter("SecureLogbookFilter", new SecureLogbookFilter(logbook))
.addMappingForUrlPatterns(EnumSet.of(REQUEST, ASYNC), true, "/*");
context.addFilter("securityFilter", new SecurityFilter())
.addMappingForUrlPatterns(EnumSet.of(REQUEST), true, "/*");
context.addFilter("LogbookFilter", new LogbookFilter(logbook))
.addMappingForUrlPatterns(EnumSet.of(REQUEST, ASYNC), true, "/*");
The first logbook filter will log unauthorized requests only. The second filter will log authorized requests, as always.
The logbook-httpclient
module contains both an HttpRequestInterceptor
and an HttpResponseInterceptor
to use with the HttpClient
:
CloseableHttpClient client = HttpClientBuilder.create()
.addInterceptorFirst(new LogbookHttpRequestInterceptor(logbook))
.addInterceptorFirst(new LogbookHttpResponseInterceptor())
.build();
Since the LogbookHttpResponseInterceptor
is incompatible with the HttpAsyncClient
there is another way to log responses:
CloseableHttpAsyncClient client = HttpAsyncClientBuilder.create()
.addInterceptorFirst(new LogbookHttpRequestInterceptor(logbook))
.build();
// and then wrap your response consumer
client.execute(producer, new LogbookHttpAsyncResponseConsumer<>(consumer), callback)
The logbook-jaxrs
module contains:
A LogbookClientFilter
to be used for applications making HTTP requests
client.register(new LogbookClientFilter(logbook));
A LogbookServerFilter
for be used with HTTP servers
resourceConfig.register(new LogbookServerFilter(logbook));
The logbook-netty
module contains:
A LogbookClientHandler
to be used with an HttpClient
:
HttpClient.create()
.tcpConfiguration(tcpClient ->
tcpClient.doOnConnected(connection ->
connection.addHandlerLast(new LogbookClientHandler(logbook))))
A LogbookServerHandler
for use used with an HttpServer
:
HttpServer.create()
.tcpConfiguration(tcpServer ->
tcpServer.doOnConnection(connection ->
connection.addHandlerLast(new LogbookServerHandler(logbook))))
Users of Spring WebFlux can pick any of the following options:
- Programmatically create a
NettyWebServer
(passing anHttpServer
) - Register a custom
NettyServerCustomizer
- Programmatically create a
ReactorClientHttpConnector
(passing anHttpClient
) - Register a custom
WebClientCustomizer
Users of Micronaut can follow the official docs on how to integrate Logbook with Micronaut.
The logbook-okhttp2
module contains an Interceptor
to use with version 2.x of the OkHttpClient
:
OkHttpClient client = new OkHttpClient();
client.networkInterceptors().add(new LogbookInterceptor(logbook);
If you're expecting gzip-compressed responses you need to register our GzipInterceptor
in addition.
The transparent gzip support built into OkHttp will run after any network interceptor which forces
logbook to log compressed binary responses.
OkHttpClient client = new OkHttpClient();
client.networkInterceptors().add(new LogbookInterceptor(logbook);
client.networkInterceptors().add(new GzipInterceptor());
The logbook-okhttp
module contains an Interceptor
to use with version 3.x of the OkHttpClient
:
OkHttpClient client = new OkHttpClient.Builder()
.addNetworkInterceptor(new LogbookInterceptor(logbook))
.build();
If you're expecting gzip-compressed responses you need to register our GzipInterceptor
in addition.
The transparent gzip support built into OkHttp will run after any network interceptor which forces
logbook to log compressed binary responses.
OkHttpClient client = new OkHttpClient.Builder()
.addNetworkInterceptor(new LogbookInterceptor(logbook))
.addNetworkInterceptor(new GzipInterceptor())
.build();
The logbook-spring
module contains a ClientHttpRequestInterceptor
to use with RestTemplate
:
LogbookClientHttpRequestInterceptor interceptor = new LogbookClientHttpRequestInterceptor(logbook);
RestTemplate restTemplate = new RestTemplate();
restTemplate.getInterceptors().add(interceptor);
Logbook comes with a convenient auto configuration for Spring Boot users. It sets up all of the following parts automatically with sensible defaults:
- Servlet filter
- Second Servlet filter for unauthorized requests (if Spring Security is detected)
- Header-/Parameter-/Body-Filters
- HTTP-/JSON-style formatter
- Logging writer
Instead of declaring a dependency to logbook-core
declare one to the Spring Boot Starter:
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-spring-boot-starter</artifactId>
<version>${logbook.version}</version>
</dependency>
Every bean can be overridden and customized if needed, e.g. like this:
@Bean
public BodyFilter bodyFilter() {
return merge(
defaultValue(),
replaceJsonStringProperty(singleton("secret"), "XXX"));
}
Please refer to LogbookAutoConfiguration
or the following table to see a list of possible integration points:
Type | Name | Default |
---|---|---|
FilterRegistrationBean |
secureLogbookFilter |
Based on LogbookFilter |
FilterRegistrationBean |
logbookFilter |
Based on LogbookFilter |
Logbook |
Based on condition, filters, formatter and writer | |
Predicate<HttpRequest> |
requestCondition |
No filter; is later combined with logbook.exclude and logbook.exclude |
HeaderFilter |
Based on logbook.obfuscate.headers |
|
PathFilter |
Based on logbook.obfuscate.parameters |
|
QueryFilter |
Based on logbook.obfuscate.parameters |
|
BodyFilter |
BodyFilters.defaultValue() , see filtering |
|
RequestFilter |
RequestFilters.defaultValue() , see filtering |
|
ResponseFilter |
ResponseFilters.defaultValue() , see filtering |
|
Strategy |
DefaultStrategy |
|
Sink |
DefaultSink |
|
HttpLogFormatter |
JsonHttpLogFormatter |
|
HttpLogWriter |
DefaultHttpLogWriter |
Multiple filters are merged into one.
Some classes from logbook-spring
are included in the auto configuration.
You can autowire LogbookClientHttpRequestInterceptor
with code like:
private final RestTemplate restTemplate;
MyClient(RestTemplateBuilder builder, LogbookClientHttpRequestInterceptor interceptor){
this.restTemplate = builder
.additionalInterceptors(interceptor)
.build();
}
The following tables show the available configuration:
Configuration | Description | Default |
---|---|---|
logbook.include |
Include only certain URLs (if defined) | [] |
logbook.exclude |
Exclude certain URLs (overrides logbook.include ) |
[] |
logbook.filter.enabled |
Enable the LogbookFilter |
true |
logbook.filter.form-request-mode |
Determines how form requests are handled | body |
logbook.secure-filter.enabled |
Enable the SecureLogbookFilter |
true |
logbook.format.style |
Formatting style (http , json , curl or splunk ) |
json |
logbook.strategy |
Strategy (default , status-at-least , body-only-if-status-at-least , without-body ) |
default |
logbook.minimum-status |
Minimum status to enable logging (status-at-least and body-only-if-status-at-least ) |
400 |
logbook.obfuscate.headers |
List of header names that need obfuscation | [Authorization] |
logbook.obfuscate.paths |
List of paths that need obfuscation. Check Filtering for syntax. | [] |
logbook.obfuscate.parameters |
List of parameter names that need obfuscation | [access_token] |
logbook.write.chunk-size |
Splits log lines into smaller chunks of size up-to chunk-size . |
0 (disabled) |
logbook.write.max-body-size |
Truncates the body up to max-body-size and appends ... . |
-1 (disabled) |
logbook:
include:
- /api/**
- /actuator/**
exclude:
- /actuator/health
- /api/admin/**
filter.enabled: true
secure-filter.enabled: true
format.style: http
strategy: body-only-if-status-at-least
minimum-status: 400
obfuscate:
headers:
- Authorization
- X-Secret
parameters:
- access_token
- password
write:
chunk-size: 1000
For basic Logback configuraton
appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
/appender>
configure Logbook with a LogstashLogbackSink
HttpLogFormatter formatter = new JsonHttpLogFormatter();
LogstashLogbackSink sink = new LogstashLogbackSink(formatter);
for outputs like
{
"@timestamp" : "2019-03-08T09:37:46.239+01:00",
"@version" : "1",
"message" : "GET http://localhost/test?limit=1",
"logger_name" : "org.zalando.logbook.Logbook",
"thread_name" : "main",
"level" : "TRACE",
"level_value" : 5000,
"http" : {
// logbook request/response contents
}
}
- The Logbook Servlet Filter interferes with downstream code using
getWriter
and/orgetParameter*()
. See Servlet for more details. - The Logbook Servlet Filter does NOT support
ERROR
dispatch. You're strongly encouraged to not use it to produce error responses. - The Logbook HTTP Client integration is handling gzip-compressed response entities incorrectly if the interceptor runs before a decompressing interceptor. Since logging compressed contents is not really helpful it's advised to register the logbook interceptor as the last interceptor in the chain.
If you have questions, concerns, bug reports, etc., please file an issue in this repository's Issue Tracker.
To contribute, simply make a pull request and add a brief description (1-2 sentences) of your addition or change. For more details, check the contribution guidelines.
- Apache HttpClient Wire Logging
- Client-side only
- Apache HttpClient exclusive
- Support for HTTP bodies
- Spring Boot Access Logging
- Spring application only
- Server-side only
- Tomcat/Undertow/Jetty exclusive
- No support for HTTP bodies
- Tomcat Request Dumper Filter
- Server-side only
- Tomcat exclusive
- No support for HTTP bodies
- logback-access
- Server-side only
- Any servlet container
- Support for HTTP bodies
Grand Turk, a replica of a three-masted 6th rate frigate from Nelson's days - logbook and charts
by JoJan is licensed under a
Creative Commons (Attribution-Share Alike 3.0 Unported).