microsoft / logmerge

Merge multiple log files by timestamp, accommodating multiple timestamp formats.

logmerge

logmerge merges multiple log files into a single stream, preserving the total ordering of events across the multiple log files.

Features

• Supports multiple timestamp formats
  • Date and time with either fractional seconds or comma-separated milliseconds
  • Integer or floating point time_t (base-1970)
• Can tag each merged line with an arbitrary text tag per source log
• Can color-code lines per log (using ANSI escape sequences)

Limitations

• Assumes all timestamps are UTC. The currently-supported timestamp formats don't support a timezone tag.
• Timestamps are not reformatted. If merged logs use different timestamp formats, the merged log will expose that.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

logmerge's Issues

Colorization should be suppressed if stdout isn't a tty/pty

Arguably, colorization escape sequences don't make sense when capturing the merged log to a file.

Should support ISO 8601 format

Including timezone specifiers.

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Allow colors to be assigned (by color name) on the command line

The built-in color assignment mechanism breaks down after the first 15 logs, and may make some choices that a human would consider to be poor. The tool should define at least a dozen colors and allow the user to specify them on the command line.

The tool should also allow the user to select a specific color (from the 255 standard ANSI colors) even if that color has no defined name.

Help text calls the program "logparse"

Should probably reference argv[0] or the equivalent, instead of something hardcoded.

Fix the typo present in the repo description

Merge multiple log files by timestamp, accomodating multiple timestamp formats.

accomodating -> accommodating

Trivial but still...

Accommodate syslog date formats

Linux still follows the "syslog" date format from RFC3164. RFC5424 intended to obsolete that and implement a new timestamp format, but that transition has not occurred simply due to industry momentum (we're too lazy to change ;) ).

Issue #1 requests support for ISO 8601 format. That's valid. But Again, ref section 6.2.3 of RFC5424 :

The TIMESTAMP field is a formalized timestamp derived from RFC3339. Whereas RFC3339 makes allowances for multiple syntaxes, this document imposes further restrictions.

And in the RFC3339 Abstract :

This document defines a date and time format for use in Internet protocols that is a profile of the ISO 8601 standard...

Rephrasing ... we use RFC3164, which is obsoleted by RFC5424, which uses a modified version of ISO 8601. So all three formats are valid. But since common systems don't use RFC5424 yet, we only really need RFC3164. ISO 8601 is valid for other applications. RFC5424 may be valid for rare environments, and should be implemented for forward-thinking applications.

Therefore, for this request, I believe logmerge should support RFC3164 directly in the parsing to find a valid date format.

I will post a PR on this.

Until then, the simplest command-line for Linux logfiles is:

./logmerge.py -r "^(.{14})" -f "%b %d %H:%M:%S" file1 file2 ...

or

./logmerge.py -r "^(...............)" -f "%b %d %H:%M:%S" file1 file2 ...

Those are "naive" regex patterns to just select the first 14 characters and match them to the formatting codes. That is the equivalent of saying "It doesn't matter what those first 14 characters are, I'm telling you what they mean" - there's no actual need to be more rigorous from the command-line. See below.

For the code, I'm supporting this pattern:

./logmerge.py -r "(.{3} \d\d \d\d:\d\d:\d\d)" -f "%b %d %H:%M:%S" file1 file2 ...

The more-specific regex can avoid errors, but I don't think any log files are going to deviate from that. The "naive" regex should not be used in the code because the regex will match any 14 characters, and when the format is applied in strptime() it will abort if the local system does not use that format. The initial three characters are the localized month. I'm using three "any characters" because I'm not positive about how different localizations will translate to three "alpha characters". The RFC specifies a three-character month identifier. The %b function will not work reliably in a fringe environment where the system parsing the log is in a different locale from the system saving the log.

Timestamp decoding should be extendible from the command line

Adding a new timestamp format should be as easy as providing a regex which matches the format and a strptime expression that decodes it to a datetime.

logmerge | more plus quitting can yield Broken Pipe exception

When the output of logmerge is being piped to more (or similar reader), and the pipe buffer fills before the merge is complete, and the user quits from more or otherwise terminates the pipeline, logmerge crashes with a Broken Pipe exception. logmerge should silently eat that exception and simply exit(0).