vectordotdev / timber-elixir

🌲 Great Elixir logging made easy

🌲 Timber - Great Elixir Logging Made Easy

Timber.io is a hosted service for aggregating logs across your entire stack - any language, any platform, any data source.

Unlike traditional logging tools, Timber integrates with language runtimes to automatically capture in-app context, turning your text-based logs into rich structured events. Timber integrates with Elixir through this library. And Timber's rich free-form query tools and real-time tailing, make drilling down into important stats easier than ever.

---

Features

• Simple Integration. Installs as an Elixir logger backend.
• Fast & light-weight. Designed with performance in mind.
• Support for local and global context.
• Automatic context capturing.
• Strong structured logging support.
• Integrates with popular 3rd party libraries such as Phoenix and Plug.

---

Get Started

• Installation
• Configuration
• Usage
• Guides
• Integrations
• Performance

---

Timber • Docs • Pricing • Security • Compliance

timber-elixir's Issues

Add information about setting levels

Implement optional clear sequence ANSI formatting

In the ruby gem we offer an option to completely hide the context using ANSI formatting. The idea is to pre-prend the context and apply a "clear sequence" every 20 characters. This is somewhat hacky, but offers the option to tail your logs in the console without the context appearing.

The clear sequence formatting codes are: \e8\e[K. This moves the cursor to the beginning of the line and hides any current contents. Again, this requires the context to be prepended with a @original callout that signals the start of the original line.

Configuration option for including log level

Similar to what we're doing for timestamps, we should do the same for the log level. This should be part of the context and the timber app will allow us to filter and see which the log level is.

Handle Error Logger Reports

Update install instructions to use details & include "send your logs" section

Add ability to sample timings

We added a new time_ms attribute. It'd be great if we could add a sample feature to this, where they could sample a timing every 1 second, for example.

Example Application

Add `config :timber, :capture_errors, true` to the example pull request

Cannot use library without Plug installed due to issues with struct

Same issue as #52 but for Plug.

Explicitly list out identified headers because the `-` cannot be converted to an atom

Update callout to use @metadata

Use @metadata instead of @timber.io

Cannot use library without Ecto installed due to issues with struct

Was trying this with something and got the following while trying to compile:

== Compilation error on file lib/timber/ecto.ex ==
** (CompileError) lib/timber/ecto.ex:70: Ecto.LogEntry.__struct__/0 is undefined, cannot expand struct Ecto.LogEntry
    (stdlib) lists.erl:1354: :lists.mapfoldl/3

Logging an event map should accept root key format

Currently this is supported:

Logger.info("message", event: %{type: :key, data: %{}})

I'd like to support this as well:

Logger.info("message", event: %{key: %{}})

I find this easier for quick one-off lines.

Output Timber Document Locally as Logfmt

This will

• deprecate hiding of the context since that will no longer be necessary
  • this will in turn fix issues with the ANSI sequence appearing on lines in the Timber console
• allow context to either be printed in debug mode or not
• print out the document in either logfmt or JSON based on a configuration setting

Fix heroku drain instructions

Point to new logs.timber.io host

Plug Test

Remove Hackney pool

This adds a layer of complication that isn't necessary. Clients can setup a pool if they need it externally to timber.

Support Advanced Handling of `:error_logger` messaging

In the latest version of the library we are dropping handling of :error_logger messaging to support issues where the :ranch library is not dispatch error messages in a standard manner; we instead are decoding messages as they are sent translated by Elixir.Logger. This has necessary tradeoffs. While it allows us to capture errors from :ranch, it limits our ability to rebuild contextual documents that were bound to processes since the data is stripped during processing.

Based on conversations in the Elixir slack #general channel with James Fish and others, utilizing an :error_logger GenEvent handler is still the best option for us, but we can improve the way we have handled messages. Work is currently being done on the rollbax package in ForzaElixir/rollbax#62 and is scheduled for the sentry package in getsentry/sentry-elixir#117 that improve the handling of :error_logger messages by parsing them at a more granular level. Having a similar parsing methodology for Timber is the end goal of this issue.

Should method be recorded as an atom?

Support Environment Variables at Runtime

Currently our system requires setting the API key in the application configuration as a binary. For customers sending logs from multiple environments using releases, it's not ideal to require the configuration value to be present at compile time. Therefore, we should support tuple based configuration of this value, such as {:system, "TIMBER_API_KEY"}

• Based on customer feedback

Allow request-id header name to be configurable

Log colorization support for IODevice

Handle SASL Reports

Timber.Plug - HTTP response calls :erlang.byte_size which crashes

The body of the response at this point is actually IO.chardata

Retry logic for IODevice transport

Allow for a configurable JSON encoder

Scrub blank context attributes

Minor, but this would save on bandwidth if we removed blank attributes. Notice query_params and headers in the below line.

2016-10-19T04:00:04.975196+00:00 app[web.1]: 2016-10-19T04:00:04.974Z [info] Sent 422 in 11ms @timberio {"context":[{"index":0,"http_request":{"scheme":"http","query_params":{},"port":80,"path":"/heroku/logplex_frames","method":"post","host":"timber-odin-staging-ingester.herokuapp.com","headers":{"user-agent":"Logplex/v100","request_id":"3f810bc3-c586-4907-aeb3-eece0e2f49d0","remote_addr":"54.160.59.216","content-type":"application/logplex-1"}},"dt":"2016-10-19T04:00:04.962813Z"},{"sql_query":{"time_ms":3,"sql":"SELECT a0.\"id\", a0.\"name\", a0.\"slug\", a0.\"app_type\", a0.\"api_key\", a0.\"organization_id\", a0.\"inserted_at\", a0.\"updated_at\" FROM \"applications\" AS a0 WHERE (a0.\"api_key\" = $1) AND (a0.\"id\" = $2)"},"index":0,"dt":"2016-10-19T04:00:04.967011Z"},{"index":0,"http_response":{"status":422,"headers":{},"bytes":154},"dt":"2016-10-19T04:00:04.974636Z"}]}

Coerce context / event attributes to their proper type

Specifically Events.User.id, this has a high potential to be passed an integer when we're expecting a string.

Test Specific Transport

Timber Context hiding support for IODevice

Escape new lines on heroku

Logplex automatically splits lines with \n characters. As a result, associated metadata is only included on the last line. We should escape \n so that we can see it as a single line (as it was originally logged).

Ecto Logger

Change `HTTPRequest` context to `HTTP`

Instruct users to remove `plug Plug.Logger` if present

Support tags

Escape New Lines in IODevice via Charlists

Currently we're handling this by transforming the message to a binary from its charlist representation and then running a substitution over the binary. Ideally, we'd run a substitution algorithm over the charlist itself, but the problem is that a charlist is of the type maybe_improper_list, so it can't be iterated over normally. (The tail is not guaranteed to be an empty list).

In order to do this properly, we'll need to iteratively build a new charlist using the transformed data from each iteration over the current charlist. The tranformation will have to handle three cases: character data (integer substitution), binary (binary substitution), list (recurse).

The character \n / [?\n] / [10] / << ?\n :: utf8>>, needs to be replaced with the sequence \\n or [92, 110] / << 92 :: utf8, 110 :: utf8 >>.

Handle Poison encoding errors better

If you try to log an event that contains data that cannot be encoded by json (ex: tuples), we need to be thoughtful around that. Users should not be doing this, but in the event it does happen an error better than the generic Poison error would be helpful.

Phoenix Template Rendering Instrumentation

I'm not sure if this is even possible currently. It will definitely need more investigation

Retry logic for HTTP device

We should add configuration to attempt retries on failures with a backoff rate.

Response line has sql query context

We should be popping off contexts when they are no longer relevant, correct? I'm seeing this for the HTTP response line:

2016-10-19T04:00:04.975196+00:00 app[web.1]: 2016-10-19T04:00:04.974Z [info] Sent 422 in 11ms @timberio {"context":[{"index":0,"http_request":{"scheme":"http","query_params":{},"port":80,"path":"/heroku/logplex_frames","method":"post","host":"timber-odin-staging-ingester.herokuapp.com","headers":{"user-agent":"Logplex/v100","request_id":"3f810bc3-c586-4907-aeb3-eece0e2f49d0","remote_addr":"54.160.59.216","content-type":"application/logplex-1"}},"dt":"2016-10-19T04:00:04.962813Z"},{"sql_query":{"time_ms":3,"sql":"SELECT a0.\"id\", a0.\"name\", a0.\"slug\", a0.\"app_type\", a0.\"api_key\", a0.\"organization_id\", a0.\"inserted_at\", a0.\"updated_at\" FROM \"applications\" AS a0 WHERE (a0.\"api_key\" = $1) AND (a0.\"id\" = $2)"},"index":0,"dt":"2016-10-19T04:00:04.967011Z"},{"index":0,"http_response":{"status":422,"headers":{},"bytes":154},"dt":"2016-10-19T04:00:04.974636Z"}]}

HTTP Transport

Send pre-flight request on HTTP initialization

This ensures that HTTP is working properly

Make timestamp configurable for IODevice transport

Allow custom header sanitization

Add a syslog transport

See https://github.com/Vagabond/erlang-syslog/blob/master/src/syslog.erl for example of how this has been done before

Don't accept timer as an argument

Timing should be exact, passing it into a function makes this value less exact. It also puts as at risk of tainting our client's timings.

Create a test matrix across supported libraries and elixir versions

Test all context and event hashes against the log event json schema

This ensures changes are valid with the specified schema version

Consider making setup instructions production specific

This starts to get hairy if you're using the HTTP client. Also the use Phoenix.Controller, log: false call is not environment specific.

Phoenix Controller Instrumentation