kpetremann / salt-exporter

Salt Prometheus exporter working out of the box without any configuration on Salt side. Comes with an event watcher TUI.

Automation enthusiast

A little about me

Currently a proud SRE@Criteo in the Network Datacenter team.

• 🔭 I’m currently working on AFK, the new Criteo's network automation framework.
• 🔧 Actively maintaining Salt Exporter and MQTT Exporter
• 💬 Ask me about Python, Golang, Network, Salt, OpenConfig, SONiC

Articles

For Criteo:

• netprobify: do you probe your network?
• SONiC: let’s migrate from proprietary to Open Source!

Personal:

• Salt Exporter: the story behind the tool
• How I do technology watch

My personal projects

Project	Category	Opensource	Status	Tech
Salt Exporter	Infrastructure, observability	true	stable	Go
MQTT Exporter	IoT, observability	true	stable	Python
DHT2MQTT	IoT	true	alpha	Go
ClawNetwork	Infrastructure	true	alpha	Go
2D game	Game development	false	prototype	Go, Ebitengine, LDTK

My professional projects

Project	Category	Opensource	Status	Tech
netprobify	Probing	true	stable	Python
AFK	Infrastructure, network, automation	true	alpha	Go, Python, Salt, NetBox

Languages and Tools

Support

If you like my personal projects, don't hesitate to buy me a coffee :)

salt-exporter's Issues

Differentiate running a function with `test=True` or `mock=True`

When running a function in test mode (for example salt <target> state.highstate test=True), we get metrics reported as if the function was executed successfully/failing according to the result:

salt_expected_responses_total{function="state.highstate",state="highstate"} 1
# HELP salt_function_responses_total Total number of response per function processed
# TYPE salt_function_responses_total counter
salt_function_responses_total{function="state.highstate",state="highstate",success="false"} 1
# HELP salt_function_status Last function/state success, 0=Failed, 1=Success
# TYPE salt_function_status gauge
salt_function_status{function="state.highstate",minion="myminion.company.com",state="highstate"} 0
# HELP salt_new_job_total Total number of new job processed
# TYPE salt_new_job_total counter
salt_new_job_total{function="state.highstate",state="highstate",success="true"} 1
# HELP salt_responses_total Total number of response job processed
# TYPE salt_responses_total counter
salt_responses_total{minion="myminion.company.com",success="false"} 1

It would be benefitial to differentiate running states in test mode, mock mode or "production".

Salt live: improve responsiveness

Two kind of issues:

• sometimes, we cannot navigate in the list
• when there are a lot of new events, the list is refreshing a lot making the TUI blinking

Panic at some point

Hi @kpetremann,
Thank you for creating salt-exporter, really great work!

If you have time, could you please take a look at this panic message?

panic: interface conversion: interface {} is []interface {}, not string

goroutine 12 [running]:
github.com/kpetremann/salt-exporter/pkg/events.extractStateFromArgs(...)
	/home/runner/work/salt-exporter/salt-exporter/pkg/events/events.go:47
github.com/kpetremann/salt-exporter/pkg/events.(*SaltEvent).ExtractState(0xc0000a2b60?)
	/home/runner/work/salt-exporter/salt-exporter/pkg/events/events.go:61 +0x47d
github.com/kpetremann/salt-exporter/internal/metrics.ExposeMetrics({0x980978, 0xc000106dc0}, 0xc000072240)
	/home/runner/work/salt-exporter/salt-exporter/internal/metrics/metrics.go:66 +0x89d
created by main.main
	/home/runner/work/salt-exporter/salt-exporter/main.go:62 +0x53e

Probably, events with incorrect schema should be just ignored.

Document salt-live

The documentation should have:

• a demo
• usage guide
• flags explanation
• a visibility in the root README

Maybe a GitHub Page would be nice.

Feature: Grafana dashboard

Hello, can you provide us a Grafana Dashboard, please ?

[salt-live] Invalid memory address

When running salt-live with both soft-filter.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x546d68]

goroutine 3763 [running]:
github.com/charmbracelet/bubbles/list.filterItems.func1()
        /home/runner/go/pkg/mod/github.com/charmbracelet/[email protected]/list/list.go:1211 +0x1a8
github.com/charmbracelet/bubbletea.(*Program).handleCommands.func1.1()
        /home/runner/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:268 +0x38
created by github.com/charmbracelet/bubbletea.(*Program).handleCommands.func1
        /home/runner/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:267 +0x14d

RFE: Allow filtering out minion-specific metrics via a cmdline option

Can you please add a cmdline option that filters out/disables all minion-specific metrics? This is needed to avoid having high-cardinality labels in Prometheus with masters that have a large number (1k+) of minions connecting to them.
Setting the '-health-minions' option to false doesn't work as I still see the following metrics:

    salt_responses_total(minion="XXXX", success=true) = <count>

Salt live: weird display when the event return is big

When the event is big, if we scroll in the raw details panel, the left panel is acting weird.

[salt-exporter] Run the `salt-call state.apply` command from minion without metrics

Salt live: segmentation fault

Happening when a filter is applied.
Due to an oversight while migrating to the "frozen" buffer.

stacktrace:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5432d2]

goroutine 6720 [running]:
github.com/charmbracelet/bubbles/list.filterItems.func1()
	/home/runner/go/pkg/mod/github.com/charmbracelet/[email protected]/list/list.go:1197 +0x1b2
github.com/charmbracelet/bubbletea.(*Program).handleCommands.func1.1()
	/home/runner/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:248 +0x38
created by github.com/charmbracelet/bubbletea.(*Program).handleCommands.func1
	/home/runner/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:247 +0x14d

Add support of tag with 20 characters or less

According the Salt docstring here:

In the new style, when the tag is longer than 20 characters, an end of tag
string is appended to the tag given by the string constant TAGEND, that is, two
line feeds '\n\n'.  When the tag is less than 20 characters then the tag is
padded with pipes "|" out to 20 characters as before.  When the tag is exactly
20 characters no padded is done.

Currently the exporter only looks for the '\n\n' delimiter.

Tag with 20 characters or less support must be added.

Add total number of connected minions

On every scrape, the exporter should include a gauge metric with the number of connected minions to that master node.
This way we can keep track of new minions connected, but more importantly, when minions disappear.

cannot match salt job with salt response (wrong parsing of salt job?)

hi,

not sure if its error on my side or not but exporter shows "job" as salt01, and not as job_ID. which makes hard to match new job and return for failed jobs.

currently running on salt 3005.1
0.8.0 version of salt-exporter

thanks

please add -version switch

hi,

please add -version switch
for now only option to determinate version is to run exporter (then version is print to output) which could be hard to use in eg. shell scripts.

thanks,

Salt-live: frozen state should be more visible

It is easy to miss/forgot that the list is frozen as the user is scrolling in the list.

The frozen state should be way more visible, with a quick message to explain how to switch back to "follow mode".

Publish a Docker image

Publish a Docker image on both Github and Docker Hub.
Only for the exporter.

It would not make sense for salt-live as the TUI is not working properly via a docker shell.

salt_function_status() counter doesn't increment

The salt_function_status() counter doesn't increment but always stays 1 - here's an example when running total of 7 states on a single minion:

#  curl -s http://localhost:2112/metrics|grep ^salt_
salt_function_responses_total{function="state.apply",state="highstate",success="true"} 3
salt_function_responses_total{function="state.highstate",state="highstate",success="true"} 2
salt_function_responses_total{function="state.sls",state="sys.passwd",success="true"} 2
salt_function_status{function="state.apply",minion="gen12472",state="highstate"} 1
salt_function_status{function="state.highstate",minion="gen12472",state="highstate"} 1
salt_function_status{function="state.sls",minion="gen12472",state="sys.passwd"} 1
salt_responses_total{minion="gen12472",success="true"} 7
#

Salt live: display issues with filter

When adding a filter, the topbar is weird.

Configuration via environment variables

Be able to configure the exporter via environment variables.

Precedence order will be same as viper:

• flags
• environment variables
• configuration file

Salt live: implement input filter

Input filter would filter events during parsing. Filtered out events would be discarded before going to the buffer.

No salt_function_status() metrics except for highstate regardless of the -health-states-filter option passed

I've tried to use the example from README, but it doesn't seem to be working in version 0.4.0, i.e.:

./salt-exporter -health-states-filter=test.ping,state.apply -health-functions-filter=""

But running 'salt-call test.ping' against the master with the above cmdline option doesn't seem to produce the below metric:

   salt_function_status{function="test.ping",minion="<minion>",state=""} 1