jorgelbg / dissect-tester

Simple API/UI for testing filebeat dissect patterns against a collection of sample log lines.

Hi there 👋, I'm Jorge - aka jorgelbg

• 🔭 I’m currently working on dashflare
• ✍🏻 I ocassionally write about everything on my blog
• 📊 Passionate about observability/monitoring topics
• 🚀 Odd mix between SRE/DevOps 🤣

Connect with me:

📕 Latest Blog Posts

• Introducing pinentry-touchid
• Displaying geohash tags from a Loki datasource in a Grafana Worldmap Panel
• Web UI for testing dissect patterns
• Better URL search with Elasticsearch
• How to rename index patterns in Kibana

dissect-tester's Issues

Change the format of the logger

We, currently, log directly in JSON (nice for further processing) but a bit verbose for normal usage. We could consider using a more user-friendly format by default.

Move into a lambda/cloud function (or similar)

Since this is basically just a stateless API it would be interesting to test if it could be refactored (the API/UI at least) into a lambda/cloud function.

Add README

High CPU usage

I've noticed that in my (very small) VM instance the CPU usage sometimes stays at 100% for a long time. Using pprof added in 7acd50c the CPU profile looks like:

Add support for a CLI version

It would be nice to have a CLI version of the tester

could not find delimiter

Amazing tool @jorgelbg (Thanks) -

But how do we test the dissect patterns when we have values in quotes in the log event ?

I tried with the same Envoy example in the blog -

Dissect Pattern -
[%{timestamp}] "%{request}" %{status} - %{bytes_sent} "%{forwarded_ips}"
"%{user_agent}" "%{unknown_id}" "%{destination_host}" "%{destination_address}"

[2020-02-21T14:29:08.671Z] "GET /stats?format=prometheus&usedonly=1 HTTP/1.1"
200 - 0 105150 6 - "10.22.10.103" "Prometheus/2.7.0" "-" "10.22.10.121:12345" "-"

but get the following error -

sample: 0, error: could not find delimiter: ] " in remaining: GET /stats?format=prometheus&usedonly=1 HTTP/1.1", (offset: 28)

Website 502

I tried going to: https://dissect-tester.jorgelbg.me/

I found the website via this article: https://jorgelbg.me/2020/02/web-ui-for-testing-dissect-patterns/

I really need some way to debug my dissect stuff easily, and this was exactly what I was looking for. Quite sad it is down. Please get it online :)

Share the pattern and samples

It could be interesting to allow sharing of the current set of samples and pattern in the UI. I had this same issue today where I wanted to share a working configuration with a colleague and couldn't.

Ideally I would like to implement this in a way that doesn't mean running additional infrastructure to store the shareables. From the top of my head maybe we can encode it in the URL.

Support Github releases

Currently, we only publish a docker hub.docker.com/jorgelb/dissect-tester. It would be nice to also publish releases supported by Github.

log sample won't accept input with '<'

The log sample seems to treat less than ('<') as a wildcard, so pasting any log content with such character (e.g. a weblogic server log: https://tekslate.com/weblogic-logging-filtering) will fail.
dissect pattern '####<%{Timestamp}> <%{Severity}> <%{Subsystem}> <%{MachineName}> <%{ServerName}> <%{ThreadID}> <%{UserID}> <%{TransactionID}> <%{ContextID}> <%{RawTimeValue}> <%{MessageID}> <%{MessageText}>'

You can manually retype the log text - this will work once, but if you click elsewhere then back to log sample it will be reformatted to replace 'special' chars with e.g. &gt; &lt; etc. which will then not parse
probably the same is the case with other 'special' chars.
I note this is problem with "log sample" but not with the dissect pattern

Add httprof/debug endpoints

This is, in general, a good practice and a nice to have feature since the API could be running as a service.