Comments (5)
@tomwilkie Yes, I've seen the grok_exporter as well.
My usual suggestion for where to produce metrics from logs is at the closest point to ingestion. To me it makes sense to do this in promtail, as it is reading, tagging, and compressing logs. I don't think it would be good to waste CPU time decompressing logs just to process them for metrics. Also doing things in promtail makes things scale better. Doing things in the Loki service means having to scale it up much more to handle the processing.
I personally like the mtail way of being able to define extractions via regexp. There are lots of cases where the log lines also contain numbers to extract.
from loki.
@ventris promtail already includes metrics about itself.
from loki.
Another thing would be to also expose metrics about the promtail exporter. Such as input records, output errors etc.
from loki.
@ventris we've exposed a bunch of metrics from promtail already, and have ideas for more (see #327). If you have any suggestions, let me know! We also have example dashboards from promtail here: weaveworks/common#146
@SuperQ we're starting to think about where to expose metrics from, and supporting mtail like usecase is top of our list. We're also looking at https://github.com/fstab/grok_exporter, have you seen that?
Open questions:
- where is the best place to do this?
promtail
seems obvious, but we also support fluentd-based ingestion, and would be nice to grab metrics from that too. This implied doing it in Loki might be preferred. Perhaps we can do it in both? - how to do this? were starting to think about adding a query language, and we could consider supporting prometheus-style recording rules in this language, and use them to record metrics from logs. For instance, recording
count(logs{job="foo"}[1m] | "^bar$")
as a recording rule of the number of records which match "^bar$" in the last 5mins, correctly propagating labels etc...
@slim-bean is thinking about this too.
from loki.
With the 0.1.0 release there is included a pipeline which allows for extracting data from log messages using regex/json and further creating metrics from this data
from loki.
Related Issues (20)
- Clean installation documentation. HOT 6
- `otlp_config` has no effect HOT 6
- OTLP endpoint: Convert OpenTelemetry resource attributes (service.namespace, service.name and service.instance.id) to job and instance labels
- Historical filter on loki HOT 1
- Permission denied loki components
- Loki v3.0.0 quietly gives incomplete results for some queries including a line_format HOT 1
- Loki doesn't create `_extracted` label suffix if there is more than one parser
- Adding custom resource in loki helm value for memcached HOT 2
- Loki v3.0.0 brokes storage after restart HOT 2
- label semantics with parsers changed in v3.0.0 HOT 3
- Query frontend is getting panic when queried for labels after upgrading to v3.0.0 HOT 5
- I saw "Loki no labels received" in Grafana using Fluent-Bit and Loki.
- [Helm] Loki 3.0 helm chart distributed mode doesn't work for ingress without gateway
- Helm chart is using wrong image tag and proxy config for GEL in SSD mode
- logcli warning - "CFPropertyListCreateFromXMLData(): Old-style plist parser: missing semicolon in dictionary on line 1. Parsing will be abandoned" HOT 2
- Method to pretty-print duration from seconds or milliseconds
- lambda-promtail dlq - which one to use Asynchronous invocation or Destinations ? HOT 3
- Storage resizing doesn't work with SingleBinary deployment. HOT 1
- Add time() function for querying timestamp HOT 1
- Switch enable_alertmanager_v2 default to true
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from loki.