paulway / lnav-formats

Log format description files for lnav

Log format description files for lnav

http://lnav.org is a powerful log file reader, designed to take multiple log files and order them by date. It understands log file formats using regular expresions and can highlight sections of line, find errors and warnings, and more.

This repository is for the log file format descriptions I've written so far.

You can install these by simply checking this directory out in your '~/.lnav/formats' directory.

lnav version

Some of these formats use features that are only available in lnav version 0.8.0 or above. If you get warnings, try renaming the offending files to .json-unused to see if this fixes things.

Easy creation of formats

I now provide a new tool, make_format.pl, which takes a series of lines from stdin and turns them into sample lines in a new format. This can, for instance, be used by marking several significant lines in lnav and issuing the command:

:pipe-to perl path/to/make_format.pl -name example -output example_log.json

Using the -name or -n option sets the name in the format. This is lazy, so you'll need to edit it to suit yourself. If this option is not specified the format name is 'NEW' throughout.

Using the -output or -o option sets the destination for the new format. If this is not specified the result will be written to standard output.

lnav-formats's Issues

Some log levels for mythbackend_log are invalid

mythbackend.json has some invalid log levels, which causes some issues, such as a bunch of errors being dumped to stderr every time LNAV is run, regardless of whether the format is even being used:

$ lnav
warning:mythbackend.json:/mythbackend_log/level/config <message-regex> -- The regular expression to match a log message and capture fields.
warning:mythbackend.json:/mythbackend_log/level/config:line 0:unexpected data, expecting one of the following data types --
warning:mythbackend.json:/mythbackend_log/level/config:accepted paths --
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/regex/[^/]+/
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/(json|convert-to-local-time|hide-extra|multiline)
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/timestamp-divisor
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/(file-pattern|level-field|timestamp-field|body-field|url|url#|title|description|timestamp-format#|module-field|opid-field)$
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/level/(trace|debug\d*|info|stats|warning|error|critical|fatal)
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/value/.+/(kind|collate|unit/field)$
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/value/.+/(identifier|foreign-key|hidden)$
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/value/.+/unit/scaling-factor/.*$
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/value/.+/action-list#
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/action/[^/]+/label
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/action/[^/]+/capture-output
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/action/[^/]+/cmd#
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/sample#/line
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/line-format#/(field|default-value)
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/line-format#/min-width
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/line-format#
warning:mythbackend.json:/mythbackend_log/level/config:  /\w+/search-table/.+/pattern
warning:mythbackend.json:/mythbackend_log/level/information:line 0:unexpected data, expecting one of the following data types --
warning:mythbackend.json:/mythbackend_log/level/information:accepted paths --
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/regex/[^/]+/
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/(json|convert-to-local-time|hide-extra|multiline)
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/timestamp-divisor
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/(file-pattern|level-field|timestamp-field|body-field|url|url#|title|description|timestamp-format#|module-field|opid-field)$
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/level/(trace|debug\d*|info|stats|warning|error|critical|fatal)
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/value/.+/(kind|collate|unit/field)$
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/value/.+/(identifier|foreign-key|hidden)$
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/value/.+/unit/scaling-factor/.*$
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/value/.+/action-list#
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/action/[^/]+/label
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/action/[^/]+/capture-output
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/action/[^/]+/cmd#
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/sample#/line
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/line-format#/(field|default-value)
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/line-format#/min-width
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/line-format#
warning:mythbackend.json:/mythbackend_log/level/information:  /\w+/search-table/.+/pattern
warning:mythbackend.json:/mythbackend_log/level/notice:line 0:unexpected data, expecting one of the following data types --
warning:mythbackend.json:/mythbackend_log/level/notice:accepted paths --
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/regex/[^/]+/
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/(json|convert-to-local-time|hide-extra|multiline)
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/timestamp-divisor
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/(file-pattern|level-field|timestamp-field|body-field|url|url#|title|description|timestamp-format#|module-field|opid-field)$
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/level/(trace|debug\d*|info|stats|warning|error|critical|fatal)
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/value/.+/(kind|collate|unit/field)$
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/value/.+/(identifier|foreign-key|hidden)$
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/value/.+/unit/scaling-factor/.*$
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/value/.+/action-list#
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/action/[^/]+/label
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/action/[^/]+/capture-output
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/action/[^/]+/cmd#
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/sample#/line
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/line-format#/(field|default-value)
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/line-format#/min-width
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/line-format#
warning:mythbackend.json:/mythbackend_log/level/notice:  /\w+/search-table/.+/pattern

Specifically, the issues are with the the levels config, information, and notice, none of which are recognized by LNAV. In addition to spitting out a bunch of error messages to stderr on launch, it doesn't know how to categorize those lines and defaults them to info (which is a bug with LNAV, as it should be unknown; see tstack/lnav#329 for more on that).

It does recognize info, so changing information to info would deal with that part of the issue. As for the others, I recommend either consolidating them into info (e.g. with "info": "I|N|C") or using the various new debug2-5 levels. The downside to the latter option is that those levels require the as-yet-unreleased 0.8.1 version of LNAV in order for them to work.

Also, the regular expression for the level ((?<level>[CEINW])) doesn't match against D for debug. This could be remedied either by adding D to the character class or by changing the expression to match any single word character (i.e. (?<level>\\w)).

-------

---