Hi there ๐, I'm Jorge - aka jorgelbg
- ๐ญ Iโm currently working on dashflare
- โ๐ป I ocassionally write about everything on my blog
- ๐ Passionate about observability/monitoring topics
- ๐ Odd mix between SRE/DevOps ๐คฃ
Simple API/UI for testing filebeat dissect patterns against a collection of sample log lines.
Home Page: https://dissect-tester.jorgelbg.me/
License: Apache License 2.0
We, currently, log directly in JSON (nice for further processing) but a bit verbose for normal usage. We could consider using a more user-friendly format by default.
Since this is basically just a stateless API it would be interesting to test if it could be refactored (the API/UI at least) into a lambda/cloud function.
I've noticed that in my (very small) VM instance the CPU usage sometimes stays at 100% for a long time. Using pprof added in 7acd50c the CPU profile looks like:
It would be nice to have a CLI version of the tester
Amazing tool @jorgelbg (Thanks) -
But how do we test the dissect patterns when we have values in quotes in the log event ?
I tried with the same Envoy example in the blog -
Dissect Pattern -
[%{timestamp}] "%{request}" %{status} - %{bytes_sent} "%{forwarded_ips}"
"%{user_agent}" "%{unknown_id}" "%{destination_host}" "%{destination_address}"
[2020-02-21T14:29:08.671Z] "GET /stats?format=prometheus&usedonly=1 HTTP/1.1"
200 - 0 105150 6 - "10.22.10.103" "Prometheus/2.7.0" "-" "10.22.10.121:12345" "-"
but get the following error -
sample: 0, error: could not find delimiter: ] "
in remaining: GET /stats?format=prometheus&usedonly=1 HTTP/1.1"
, (offset: 28)
I tried going to: https://dissect-tester.jorgelbg.me/
I found the website via this article: https://jorgelbg.me/2020/02/web-ui-for-testing-dissect-patterns/
I really need some way to debug my dissect stuff easily, and this was exactly what I was looking for. Quite sad it is down. Please get it online :)
It could be interesting to allow sharing of the current set of samples and pattern in the UI. I had this same issue today where I wanted to share a working configuration with a colleague and couldn't.
Ideally I would like to implement this in a way that doesn't mean running additional infrastructure to store the shareables. From the top of my head maybe we can encode it in the URL.
Currently, we only publish a docker hub.docker.com/jorgelb/dissect-tester. It would be nice to also publish releases supported by Github.
The log sample seems to treat less than ('<') as a wildcard, so pasting any log content with such character (e.g. a weblogic server log: https://tekslate.com/weblogic-logging-filtering) will fail.
dissect pattern '####<%{Timestamp}> <%{Severity}> <%{Subsystem}> <%{MachineName}> <%{ServerName}> <%{ThreadID}> <%{UserID}> <%{TransactionID}> <%{ContextID}> <%{RawTimeValue}> <%{MessageID}> <%{MessageText}>'
You can manually retype the log text - this will work once, but if you click elsewhere then back to log sample it will be reformatted to replace 'special' chars with e.g. > < etc. which will then not parse
probably the same is the case with other 'special' chars.
I note this is problem with "log sample" but not with the dissect pattern
This is, in general, a good practice and a nice to have feature since the API could be running as a service.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.