fundingcircle / loga

Unified logging across environments

Loga

Description

Loga provides consistent logging across frameworks and environments.

Includes:

• One tagged logger for all environments
• Human readable logs for development
• Structured logs for production (GELF)
• One Rack logger for all Rack based applications

TOC

• Installation
  • Rails
    • Reduced logs
    • Request log tags
  • Sinatra
• Output example
  • GELF Format
  • Simple Format
• Road map
• Contributing
  • Running tests
• Credits
• License

Installation

Add this line to your application's Gemfile:

gem 'loga'

Rails

Let Loga know what your application name is and Loga will do the rest.

# config/application.rb
class MyApp::Application < Rails::Application
  config.loga = { service_name: 'my_app' }
end

Loga hooks into the Rails logger initialization process and defines its own logger for all environments.

The logger configuration is adjusted based on the environment:

	Production	Test	Development	Others
Output	STDOUT	log/test.log	STDOUT	STDOUT
Format	gelf	simple	simple	simple

You can customize the configuration to your liking:

# config/application.rb
class MyApp::Application < Rails::Application
  config.loga = {
    device:       File.open("log/application.log", 'a'),
    format:       :gelf,
    service_name: 'my_app',
  }
end

Loga leverages existing Rails configuration options:

• config.filter_parameters
• config.log_level
• config.log_tags

Use these options to customize Loga instead of the Loga options hash.

Inside your application use Rails.logger instead of Loga.logger, even though they are equivalent, to prevent lock-in.

Reduced logs

When the format set to gelf requests logs are reduced to a single log entry, which could include an exception.

This is made possible by silencing these loggers:

• Rack::Request::Logger
• ActionDispatch::DebugExceptions
• ActionController::LogSubscriber
• ActionView::LogSubscriber
• ActionMailer::LogSubscriber

Request log tags

To provide consistency between Rails and other Rack frameworks, tags (e.i config.log_tags) are computed with a Loga::Rack::Request as opposed to a ActionDispatch::Request.

Sinatra

With Sinatra Loga needs to be configured manually:

# config.ru
require 'sinatra/base'
require 'loga'

Loga.configure(
  filter_parameters: [:password],
  format: :gelf,
  service_name: 'my_app',
  tags: [:uuid],
)

class MyApp < Sinatra::Base
  set :logging, false # suppress Sinatra request logger

  get '/' do
    Loga.logger.info('Everything is Awesome!')
    'Hello World!'
  end
end

use Loga::Rack::RequestId
use Loga::Rack::Logger

run MyApp

You can now use Loga.logger or assign it to your existing logger. The above configuration also inserts two middleware:

• Loga::Rack::RequestId makes the request id available to the request logger
• Loga::Rack::Logger logs requests

You can easily switch between formats by using the LOGA_FORMAT environment variable. The format key in the options takes precedence over the environment variable therefore it must be removed.

LOGA_FORMAT=simple rackup

Sidekiq

Loga 2.7 provides an out-of-the-box support for Sidekiq ~> 7.0.

Output Example

GELF Format

Rails request logger: (includes controller/action name):

curl localhost:3000/ok -X GET -H "X-Request-Id: 12345"

{
   "_request.status":     200,
   "_request.method":     "GET",
   "_request.path":       "/ok",
   "_request.params":     {},
   "_request.request_id": "12345",
   "_request.request_ip": "127.0.0.1",
   "_request.user_agent": null,
   "_request.controller": "ApplicationController#ok",
   "_request.duration":   0,
   "_type":               "request",
   "_service.name":       "my_app",
   "_service.version":    "1.0",
   "_tags":               "12345",
   "short_message":       "GET /ok 200 in 0ms",
   "timestamp":           1450150205.123,
   "host":                "example.com",
   "level":               6,
   "version":             "1.1"
}

Sinatra request output is identical to Rails but without the _request.controller key.

Logger output:

# Rails.logger
# or
# Loga.logger

Loga.configure(service_name: 'my_app', format: :gelf)

Loga.logger.info('I love Loga')
Loga.logger.tagged(%w(USER_123 CRM)) do
  Loga.logger.info('I love Loga with tags')
end

{
  "_service.name":    "my_app",
  "_service.version": "v1.0.0",
  "_tags":            "",
  "host":              "example.com",
  "level":            6,
  "short_message":    "I love Loga",
  "timestamp":        1479402679.663,
  "version":          "1.1"
}

{
  "_service.name":    "my_app",
  "_service.version": "v1.0.0",
  "_tags":            "USER_123 CRM",
  "host":              "example.com",
  "level":            6,
  "short_message":    "I love Loga",
  "timestamp":        1479402706.102,
  "version":          "1.1"
}

Simple Format

Request logger:

curl localhost:3000/ok -X GET -H "X-Request-Id: 12345"

Rails

I, [2016-11-15T16:05:03.614081+00:00 #1][12345] Started GET "/ok" for ::1 at 2016-11-15 16:05:03 +0000
I, [2016-11-15T16:05:03.620176+00:00 #1][12345] Processing by ApplicationController#ok as HTML
I, [2016-11-15T16:05:03.624807+00:00 #1][12345]   Rendering text template
I, [2016-11-15T16:05:03.624952+00:00 #1][12345]   Rendered text template (0.0ms)
I, [2016-11-15T16:05:03.625137+00:00 #1][12345] Completed 200 OK in 5ms (Views: 4.7ms)

Sinatra

I, [2016-11-15T16:10:08.645521+00:00 #1][12345] GET /ok 200 in 0ms type=request data={:request=>{"status"=>200, "method"=>"GET", "path"=>"/ok", "params"=>{}, "request_id"=>"12345", "request_ip"=>"127.0.0.1", "user_agent"=>nil, "duration"=>0}}

Logger output:

Loga.configure(service_name: 'my_app', format: :simple)

Loga.logger.info('I love Loga')
Loga.logger.tagged(%w(USER_123 CRM)) do
  Loga.logger.info('I love Loga with tags')
end

I, [2016-11-17T17:07:46.714215+00:00 #595] I love Loga
I, [2016-11-17T17:07:46.725624+00:00 #595][USER_123 CRM] I love Loga with tags

Road map

Consult the milestones.

Contributing

Loga is in active development, feedback and contributions are welcomed.

Running tests

This project uses appraisal to run tests against different versions of dependencies (e.g. Rails, Sinatra).

Install Loga dependencies with bundle install and then appraisals with bundle exec appraisal install.

Run all tests with bundle exec appraisal rspec.

You can run tests for one appraisal with bundle exec appraisal appraisal-name rspec. Refer to the Appraisals file for a complete lists of appraisals.

Prefix test command with RACK_ENV to switch between environments for Rack based tests RACK_ENV=production bundle exec appraisal rspec.

Experiment Guard support introduced to ease running tests locally bundle exec guard.

CI results are the source of truth.

Credits

• Lograge
• LogStashLogger
• Rails
• RackLogstasher

License

Copyright (c) 2015 Funding Circle. All rights reserved.

Distributed under the BSD 3-Clause License.

loga's Issues

Use single Rails dummy applications

The specs run against two dummy Rails applications. There is a lot of duplication between two setups.
Ideally we would only have on dummy application.

Dummy applications source

Improve passing context to logger

Something like:

Loga.logger.info('This is my test message', uuid: 'asdfdsaf', random_key: 'asdf')

open source this repo

I think this is a good candidate for a open sourcing.

Needs updating of licence.

Sidekiq integration

Sidekiq comes with a middleware layer allowing us to insert/swap a custom logger.

• The Loga Sidekiq logger should align with the format currently defined.
• Both Sidekiq server and client events should be logged.
• Investigate tagged logging support.

Loga use to have Sidekiq support but it was removed as it was incomplete source.

Replace this with lograge?

As it is now, one simple request to /heartbeat is converted to three log entries in Graylog. To make logs useful, can we either replace this with (or incorporate) lograge to produce one line of things that makes sense?

Hutch integration

Explore options to substitute Hutch logging beyond Hutch::Logging.logger = Logging.logger

Frozen string literals cause the gem to raise error

Offending line

GNU which does not support -s flag

The GNU version of which does not support the -s flag for silent output.

$ which -s git
/usr/bin/which: invalid option -- 's'
/usr/bin/git

ActionMailer integration

Hook into ActionMailer instrumentation and log event in a formatted structure.

Example ActionMailer::LogSubscriber

Publish gem to gemfury server

On a successful build for a new gem version, publish the gem to gemfury server.

More info here

Add a way to log the current user in the apps

We could add some hook that is called with the current request and log message. In this way applications could configure additional things that will be logged on every time. The main case is logging the current user.

The hook is de facto a custom formatter. So changing the formatter is should also work but this should be documented and tested.

Release versioning

📝 These comments are based on an assumption that this project is using semantic versioning. If that's incorrect, please let me know!

This project still hasn't incremented the minor or major version number and is currently at 0.0.12. What's stopping it from hitting version 1.0.0?

Create Python package

We should create a Python package with a similar interface to the Ruby gem.

CI Setup

Setup continuous integration with multiple rubies:

• ruby-1.9.3
• ruby-2.0.0

Configuration to limit exception backtrace size

The setting will limit the size of the backtrace array.

# example
Log.configuration.backtrace_size = 5

Add testing instructions

The README for this application does not contain instructions on how to run the test suite, which makes it difficult for contributors to make changes.

Loga to rule them all

💭 If we'll use Loga for all of our stack is probably worth considering splitting Loga into different clients:

• loga-ruby
• loga-clj
• ...

I know ATM we only have Ruby but I think it's something worth to bare in mind.

Extract Sinatra application into file

Extract the dummy Sinatra application used in the specs into it's own file under spec/fixtures/.

This is to allow for cleaner specs.

Inline application source.

Loga "enabled" does not work for non Rails projects

This is handled only in the Railtie. In Sinatra projects it cannot be disabled.

Configuration to filter sensitive request parameters

The configuration setting will filter the values provided the sensitive keys to filter.

# example log output
GET /login?username=FILTERED&password=FILTERED