pamburus / hl Goto Github PK
View Code? Open in Web Editor NEWA fast and powerful log viewer and processor that translates JSON or logfmt logs into a pretty human-readable format.
License: MIT License
A fast and powerful log viewer and processor that translates JSON or logfmt logs into a pretty human-readable format.
License: MIT License
Is the following supported by having two filtered substrings?
./hl final.log -f message~=status -f container_name=datadog
I have it working if I only use one filter (each tested correctly by themselves) but when I combine both, the output is nothing. Trying to get logs for a container name with the word status
in the message field.
There is hl example.log -f component=tsdb
and its negation hl example.log -f component!=tsdb
.
Then there is hl example.log -f provider~=string
, but its negation doesn't work hl example.log -f provider!~=string
.
Thank you.
Thanks for hl, it's a useful tool!
I'm dealing with a rather unfortunate log format where the message field is not located at the top level of the JSON object, but in a nested field. Luckily, hl supports reading the message from a nested field, but I found that it duplicates the message in the output while doing so. This does not happen if the message is in a top level field.
Contents of good.json
:
{"ts": "1711402897", "level": "info", "message": "test message", "field1": "value1"}
Command: hl good.json
Output:
Mar 25 21:41:37.000 |INF| test message field1=value1
This is what I'd expect to happen. The message is not shown as a field but is displayed right after the log level.
Contents of bad.json
:
{"ts": "1711402897", "level": "info", "message": {"text": "test message", "nested1": "value2"}, "field1": "value1"}
Contents of config.yaml
:
fields:
predefined:
message:
names: [message.text]
Command: HL_CONFIG=config.yaml hl bad.json
Output:
Mar 25 21:41:37.000 |INF| test message message={ text=test message nested1=value2 } field1=value1
In this case, the message is correctly displayed after the log level, but I would not have expected it to also still be displayed in the fields.
I'm not sure if this is intended behavior, and if it isn't, whether it's even desirable to change this to avoid the duplication.
Predefined field names like message
, logger
, and caller
should be case-insensitive in filters.
I noticed if I put the timezone in EST and downloaded recent logs (within a couple of minutes). The hour is off by 1 because EST is on DST currently right now. Can this be corrected?
When configuring Kubernetes and components thereof to log in json format the standard is to put a timestamp in the field ts as float signifying milliseconds since epoch. For example 1710858913.7978325 for Tue Mar 19 14:35:13 2024 (UTC).
Can you make hl parse this and output a human readable time string? If yes, how?
Here is the responsible line of code: https://github.com/kubernetes/component-base/blob/9d90bf7eaa57df6d6e0dd316529274b3c5747f5f/logs/json/json.go#L71
Thank you for #154 , which improved parsing of kubernetes json logs: https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#json-log-format
The remaining thing that would be nice to have addressed is level. As you can see above it is handled a bit differently.
The presence of the field v means info. But a higher number is reasonable to interpret as debug.
So a variant for the level configuration could look like:
- names: [v]
values:
debug: [2, 3, 4]
info: [0,1]
Except that it doesn't work. Going back to the example I gave in #153:
{"ts":1710923450736.8257,"caller":"eligibility/eligibility.go:104","msg":"Scale-down calculation: ignoring 2 nodes unremovable in the last 5m0s\n","v":1}
For this with the configuration above hl still show (?)
as level. But if I put the value of v in quotes, making it a string, it does work. I.e. with this example hl shows the log level as info:
{"ts":1710923450736.8257,"caller":"eligibility/eligibility.go:104","msg":"Scale-down calculation: ignoring 2 nodes unremovable in the last 5m0s\n","v":"1"}
So I propose support for converting field value and values in the configuration to a common type before comparison.
An error in these logs is signified by the presence of the field err
, the content of which you would also like to see. As far as I can see this is not possible to configure at the moment. So I propose a new feature to make the presence of a field indicate a level. Maybe the configuration could look like this:
- names: [err]
values:
error: exists
I would love to be able to use hl
for logfmt logs.
Any chance this could be added as supported format?
For me, that's the only reason i keep using humanlog
Hi
It would be very useful if hl can merge logs based on time.
Those logs might comes from different component in a big system. In order to trace issues, having a time ordered a logs is very useful. Hopefully it can be supported in this tool
best regard
Some programs writes logs with custom field names. In my case, datetime
field instead of time
and level_name
instead of level
:
{"datetime":"2021-05-21T02:53:11+03:00","level_name":"info","message":"Rabbitmq connection created"}
And the hl
output:
--- |(?)| Rabbitmq connection created datetime='2021-05-21T02:53:11+03:00' level-name='info'
Hi, this little change (or similar) would make our life easier. Thank you for your work!
diff --git a/src/model.rs b/src/model.rs
index d209177..2a6fc94 100644
--- a/src/model.rs
+++ b/src/model.rs
@@ -159,7 +159,7 @@ impl<'de: 'a, 'a> Visitor<'de> for RecordVisitor<'a> {
"__REALTIME_TIMESTAMP" => {
rts = rts.or(Some(access.next_value()?));
}
- "msg" | "MESSAGE" | "Message" => {
+ "msg" | "MESSAGE" | "Message" | "message" => {
message = access.next_value()?;
}
"level" | "LEVEL" | "Level" => {
$ echo '{"filename": "whtvr", "lineno": 1}' | hl -P
--- |(?)| @ whtvr:
$ echo '{"filename": "whtvr", "lineno": "1"}' | hl -P
--- |(?)| @ whtvr:1
I tried making a PR but had to change the second type of FileLine
to String
, which doesn't look like it's the best way. First time diving into a rust codebase.
How does this compare to https://github.com/brocode/fblog ?
The performance section mentions a log file used for testing, can you provide a link to that or what you used to generate it?
I'd like to try it out with lnav to see how it performs and if there is anything I can improve.
Thanks
Hello,
First, thanks for the great work!
While trying to tweak the config file, I realized that the info
level does not parse INF
correctly. Lowercase inf
works fine. You can try with the attached files.
I'm on Windows with v0.29.5.
Thanks,
Laurent
example.log
example-hl-config.yaml.txt
It would be useful if hl had support for logs following the ECS standard:
https://www.elastic.co/guide/en/ecs/8.10/ecs-reference.html
Especially the general log related fields:
https://www.elastic.co/guide/en/ecs/8.10/ecs-log.html
As an example here is log line from a python application using the ecs_logging library (filtered through jq for readability):
{
"@timestamp": "2023-11-02T08:09:10.423Z",
"log.level": "info",
"message": "backup for database bar in foo is already done today",
"application": "backup-rds",
"ecs": {
"version": "1.6.0"
},
"language": "python",
"log": {
"logger": "__main__",
"origin": {
"file": {
"line": 107,
"name": "app.py"
},
"function": "is_backup_done"
},
"original": "backup for database bar in foo is already done today"
},
"process": {
"name": "MainProcess",
"pid": 1,
"thread": {
"id": 140610736142144,
"name": "MainThread"
}
}
}
Adding support for log.level is just a matter of adding the field name log.level to fileds.predefined.logger.names in config.yaml. But I have also tried to add caller information, but as far as I can see it is not possible to specify nested fields as predefined fields.
I'm using version 0.20.0-beta.14.9 of hl.
Below is an example format from our service, which logs to syslog and got some prefix:
Dec 12 02:20:57 abc.def myserver[12606]: {"level":"info","action":"LOGIN","clientId":"myserver:v1.2.3","deviceId":"abcd", "msg":"login succeeded"}
It would be nice if hl
can leave the prefix part as is and format the json part, like what jl does
With `jl':
echo 'Dec 12 02:20:57 abc.def myserver[12606]: {"level":"info","action":"LOGIN","clientId":"myserver:v1.2.3","deviceId":"abcd", "msg":"login succeeded"}' | jl
Dec 12 02:20:57 abc.def myserver[12606]: INFO: login succeeded [action=LOGIN clientId=myserver:v1.2.3 deviceId=abcd]
With hl
:
echo 'Dec 12 02:20:57 abc.def myserver[12606]: {"level":"info","action":"LOGIN","clientId":"myserver:v1.2.3","deviceId":"abcd", "msg":"login succeeded"}' | hl -P
Dec 12 02:20:57 abc.def myserver[12606]: {"level":"info","action":"LOGIN","clientId":"myserver:v1.2.3","deviceId":"abcd", "msg":"login succeeded"}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.