containers / prometheus-podman-exporter

Prometheus exporter for podman environments exposing containers, pods, images, volumes and networks information.

prometheus-podman-exporter

Prometheus exporter for podman v4.x environment exposing containers, pods, images, volumes and networks information.

prometheus-podman-exporter uses the podman v4.x (libpod) library to fetch the statistics and therefore no need to enable podman.socket service unless using the container image.

• Installation
• Usage and Options
• Collectors
• License

Installation

Building from source, using container image or installing packaged versions are detailed in install guide.

Usage and Options

Usage:
  prometheus-podman-exporter [flags]

Flags:
  -a, --collector.enable-all                  Enable all collectors by default.
  -i, --collector.image                       Enable image collector.
  -n, --collector.network                     Enable network collector.
  -o, --collector.pod                         Enable pod collector.
  -b, --collector.store_labels                Convert pod/container/image labels on prometheus metrics for each pod/container/image.
  -s, --collector.system                      Enable system collector.
  -v, --collector.volume                      Enable volume collector.
  -w, --collector.whitelisted_labels string   Comma separated list of pod/container/image labels to be converted
                                              to labels on prometheus metrics for each pod/container/image.
                                              collector.store_labels must be set to false for this to take effect.
  -d, --debug                                 Set log level to debug.
  -h, --help                                  help for prometheus-podman-exporter
      --version                               Print version and exit.
      --web.config.file string                [EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.
  -e, --web.disable-exporter-metrics          Exclude metrics about the exporter itself (promhttp_*, process_*, go_*).
  -l, --web.listen-address string             Address on which to expose metrics and web interface. (default ":9882")
  -m, --web.max-requests int                  Maximum number of parallel scrape requests. Use 0 to disable (default 40)
  -p, --web.telemetry-path string             Path under which to expose metrics. (default "/metrics")

By default only container collector is enabled, in order to enable all collectors use --collector.enable-all or use --collector.enable-<name> flag to enable other collector.

Example: enable all available collectors:

$ ./bin/prometheus-podman-exporter --collector.enable-all

The exporter uses plain HTTP without any form of authentication to expose the metrics by default. Use --web.config.file with a configuration file to use TLS for confidentiality and/or to enable authentication. Visit this page for more information about the syntax of the configuration file.

Collectors

The table below list all existing collector and their description.

Name	Description
container	exposes containers information
image	exposes images information
network	exposes networks information
pod	exposes pod information
volume	exposes volume information
system	exposes system (host) information

Collectors examples output

container

# HELP podman_container_info Container information.
# TYPE podman_container_info gauge
podman_container_info{id="19286a13dc23",image="docker.io/library/sonarqube:latest",name="sonar01",pod_id="",pod_name="",ports="0.0.0.0:9000->9000/tcp"} 1
podman_container_info{id="22e3d69be889",image="localhost/podman-pause:4.1.0-1651853754",name="959a0a3530db-infra",pod_id="959a0a3530db",pod_name="pod02",ports=""} 1
podman_container_info{id="390ac740fa80",image="localhost/podman-pause:4.1.0-1651853754",name="d05cda23085a-infra",pod_id="d05cda23085a",pod_name="pod03",ports=""} 1
podman_container_info{id="482113b805f7",image="docker.io/library/httpd:latest",name="web_server",pod_id="",pod_name="",ports="0.0.0.0:8000->80/tcp"} 1
podman_container_info{id="642490688d9c",image="docker.io/grafana/grafana:latest",name="grafana",pod_id="",pod_name="",ports="0.0.0.0:3000->3000/tcp"} 1
podman_container_info{id="ad36e85960a1",image="docker.io/library/busybox:latest",name="busybox01",pod_id="3e8bae64e9af",pod_name="pod01",ports=""} 1
podman_container_info{id="dda983cc3ecf",image="localhost/podman-pause:4.1.0-1651853754",name="3e8bae64e9af-infra",pod_id="3e8bae64e9af",pod_name="pod01",ports=""} 1

# HELP podman_container_state Container current state (-1=unknown,0=created,1=initialized,2=running,3=stopped,4=paused,5=exited,6=removing,7=stopping).
# TYPE podman_container_state gauge
podman_container_state{id="19286a13dc23",pod_id="",pod_name=""} 2
podman_container_state{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_state{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 5
podman_container_state{id="482113b805f7",pod_id="",pod_name=""} 4
podman_container_state{id="642490688d9c",pod_id="",pod_name=""} 2
podman_container_state{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 5
podman_container_state{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 2

# HELP podman_container_block_input_total Container block input.
# TYPE podman_container_block_input_total counter
podman_container_block_input_total{id="19286a13dc23",pod_id="",pod_name=""} 49152
podman_container_block_input_total{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_block_input_total{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 0
podman_container_block_input_total{id="482113b805f7",pod_id="",pod_name=""} 0
podman_container_block_input_total{id="642490688d9c",pod_id="",pod_name=""} 1.41533184e+08
podman_container_block_input_total{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 0
podman_container_block_input_total{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 0

# HELP podman_container_block_output_total Container block output.
# TYPE podman_container_block_output_total counter
podman_container_block_output_total{id="19286a13dc23",pod_id="",pod_name=""} 1.790976e+06
podman_container_block_output_total{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_block_output_total{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 0
podman_container_block_output_total{id="482113b805f7",pod_id="",pod_name=""} 8192
podman_container_block_output_total{id="642490688d9c",pod_id="",pod_name=""} 4.69248e+07
podman_container_block_output_total{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 0
podman_container_block_output_total{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 0

# HELP podman_container_cpu_seconds_total total CPU time spent for container in seconds.
# TYPE podman_container_cpu_seconds_total counter
podman_container_cpu_seconds_total{id="19286a13dc23",pod_id="",pod_name=""} 83.231904
podman_container_cpu_seconds_total{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_cpu_seconds_total{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 0
podman_container_cpu_seconds_total{id="482113b805f7",pod_id="",pod_name=""} 0.069712
podman_container_cpu_seconds_total{id="642490688d9c",pod_id="",pod_name=""} 3.028685
podman_container_cpu_seconds_total{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 0
podman_container_cpu_seconds_total{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 0.011687

# HELP podman_container_cpu_system_seconds_total total system CPU time spent for container in seconds.
# TYPE podman_container_cpu_system_seconds_total counter
podman_container_cpu_system_seconds_total{id="19286a13dc23",pod_id="",pod_name=""} 0.007993418
podman_container_cpu_system_seconds_total{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_cpu_system_seconds_total{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 0
podman_container_cpu_system_seconds_total{id="482113b805f7",pod_id="",pod_name=""} 4.8591e-05
podman_container_cpu_system_seconds_total{id="642490688d9c",pod_id="",pod_name=""} 0.00118734
podman_container_cpu_system_seconds_total{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 0
podman_container_cpu_system_seconds_total{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 9.731e-06

# HELP podman_container_created_seconds Container creation time in unixtime.
# TYPE podman_container_created_seconds gauge
podman_container_created_seconds{id="19286a13dc23",pod_id="",pod_name=""} 1.655859887e+09
podman_container_created_seconds{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 1.655484892e+09
podman_container_created_seconds{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 1.655489348e+09
podman_container_created_seconds{id="482113b805f7",pod_id="",pod_name=""} 1.655859728e+09
podman_container_created_seconds{id="642490688d9c",pod_id="",pod_name=""} 1.655859511e+09
podman_container_created_seconds{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 1.655859858e+09
podman_container_created_seconds{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 1.655859839e+09

# HELP podman_container_started_seconds Container started time in unixtime.
# TYPE podman_container_started_seconds gauge
podman_container_started_seconds{id="19286a13dc23",pod_id="",pod_name=""} 1.659253804e+09
podman_container_started_seconds{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} -6.21355968e+10
podman_container_started_seconds{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 1.66064284e+09
podman_container_started_seconds{id="482113b805f7",pod_id="",pod_name=""} 1.659253804e+09
podman_container_started_seconds{id="642490688d9c",pod_id="",pod_name=""} 1.660642996e+09
podman_container_started_seconds{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 1.66064284e+09
podman_container_started_seconds{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 1.66064284e+09

# HELP podman_container_exit_code Container exit code, if the container has not exited or restarted then the exit code will be 0.
# TYPE podman_container_exit_code gauge
podman_container_exit_code{id="19286a13dc23",pod_id="",pod_name=""} 0
podman_container_exit_code{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_exit_code{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 1
podman_container_exit_code{id="482113b805f7",pod_id="",pod_name=""} 0
podman_container_exit_code{id="642490688d9c",pod_id="",pod_name=""} 0
podman_container_exit_code{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 130
podman_container_exit_code{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 0

# HELP podman_container_exited_seconds Container exited time in unixtime.
# TYPE podman_container_exited_seconds gauge
podman_container_exited_seconds{id="19286a13dc23",pod_id="",pod_name=""} 1.659253805e+09
podman_container_exited_seconds{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} -6.21355968e+10
podman_container_exited_seconds{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 1.660643511e+09
podman_container_exited_seconds{id="482113b805f7",pod_id="",pod_name=""} 1.659253805e+09
podman_container_exited_seconds{id="642490688d9c",pod_id="",pod_name=""} 1.659253804e+09
podman_container_exited_seconds{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 1.660643511e+09
podman_container_exited_seconds{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 1.660643511e+09

# HELP podman_container_mem_limit_bytes Container memory limit.
# TYPE podman_container_mem_limit_bytes gauge
podman_container_mem_limit_bytes{id="19286a13dc23",pod_id="",pod_name=""} 9.713655808e+09
podman_container_mem_limit_bytes{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_mem_limit_bytes{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 0
podman_container_mem_limit_bytes{id="482113b805f7",pod_id="",pod_name=""} 9.713655808e+09
podman_container_mem_limit_bytes{id="642490688d9c",pod_id="",pod_name=""} 9.713655808e+09
podman_container_mem_limit_bytes{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 0
podman_container_mem_limit_bytes{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 9.713655808e+09

# HELP podman_container_mem_usage_bytes Container memory usage.
# TYPE podman_container_mem_usage_bytes gauge
podman_container_mem_usage_bytes{id="19286a13dc23",pod_id="",pod_name=""} 1.029062656e+09
podman_container_mem_usage_bytes{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_mem_usage_bytes{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 0
podman_container_mem_usage_bytes{id="482113b805f7",pod_id="",pod_name=""} 2.748416e+06
podman_container_mem_usage_bytes{id="642490688d9c",pod_id="",pod_name=""} 3.67616e+07
podman_container_mem_usage_bytes{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 0
podman_container_mem_usage_bytes{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 49152

# HELP podman_container_net_input_total Container network input.
# TYPE podman_container_net_input_total counter
podman_container_net_input_total{id="19286a13dc23",pod_id="",pod_name=""} 430
podman_container_net_input_total{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_net_input_total{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 0
podman_container_net_input_total{id="482113b805f7",pod_id="",pod_name=""} 430
podman_container_net_input_total{id="642490688d9c",pod_id="",pod_name=""} 4323
podman_container_net_input_total{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 0
podman_container_net_input_total{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 430

# HELP podman_container_net_output_total Container network output.
# TYPE podman_container_net_output_total counter
podman_container_net_output_total{id="19286a13dc23",pod_id="",pod_name=""} 110
podman_container_net_output_total{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_net_output_total{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 0
podman_container_net_output_total{id="482113b805f7",pod_id="",pod_name=""} 110
podman_container_net_output_total{id="642490688d9c",pod_id="",pod_name=""} 12071
podman_container_net_output_total{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 0
podman_container_net_output_total{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 110

# HELP podman_container_pids Container pid number.
# TYPE podman_container_pids gauge
podman_container_pids{id="19286a13dc23",pod_id="",pod_name=""} 94
podman_container_pids{id="22e3d69be889",pod_id="959a0a3530db",pod_name="pod02"} 0
podman_container_pids{id="390ac740fa80",pod_id="d05cda23085a",pod_name="pod03"} 0
podman_container_pids{id="482113b805f7",pod_id="",pod_name=""} 82
podman_container_pids{id="642490688d9c",pod_id="",pod_name=""} 14
podman_container_pids{id="ad36e85960a1",pod_id="3e8bae64e9af",pod_name="pod01"} 0
podman_container_pids{id="dda983cc3ecf",pod_id="3e8bae64e9af",pod_name="pod01"} 1

pod

# HELP podman_pod_state Pods current state current state (-1=unknown,0=created,1=error,2=exited,3=paused,4=running,5=degraded,6=stopped).
# TYPE podman_pod_state gauge
podman_pod_state{id="3e8bae64e9af"} 5
podman_pod_state{id="959a0a3530db"} 0
podman_pod_state{id="d05cda23085a"} 2

# HELP podman_pod_info Pod information
# TYPE podman_pod_info gauge
podman_pod_info{id="3e8bae64e9af",infra_id="dda983cc3ecf",name="pod01"} 1
podman_pod_info{id="959a0a3530db",infra_id="22e3d69be889",name="pod02"} 1
podman_pod_info{id="d05cda23085a",infra_id="390ac740fa80",name="pod03"} 1

# HELP podman_pod_containers Number of containers in a pod.
# TYPE podman_pod_containers gauge
podman_pod_containers{id="3e8bae64e9af"} 2
podman_pod_containers{id="959a0a3530db"} 1
podman_pod_containers{id="d05cda23085a"} 1

# HELP podman_pod_created_seconds Pods creation time in unixtime.
# TYPE podman_pod_created_seconds gauge
podman_pod_created_seconds{id="3e8bae64e9af"} 1.655859839e+09
podman_pod_created_seconds{id="959a0a3530db"} 1.655484892e+09
podman_pod_created_seconds{id="d05cda23085a"} 1.655489348e+09

image

# HELP podman_image_info Image information.
# TYPE podman_image_info gauge
podman_image_info{id="48565a8e6250",parent_id="",repository="docker.io/bitnami/prometheus",tag="latest",digest="sha256:4d7fdebe2a853aceb15019554b56e58055f7a746c0b4095eec869d5b6c11987e"} 1
podman_image_info{id="62aedd01bd85",parent_id="",repository="docker.io/library/busybox",tag="latest",digest="sha256:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74"} 1
podman_image_info{id="75c013514322",parent_id="",repository="docker.io/library/sonarqube",tag="latest",digest="sha256:548f3d4246cda60c311a035620c26ea8fb21b3abc870c5806626a32ef936982b"} 1
podman_image_info{id="a45fa0117c2b",parent_id="",repository="localhost/podman-pause",tag="4.1.0-1651853754",digest="sha256:218169c5590870bb95c06e9f7e80ded58f6644c1974b0ca7f2c3405b74fc3b57"} 1
podman_image_info{id="b260a49eebf9",parent_id="",repository="docker.io/library/httpd",tag="latest",digest="sha256:ba846154ade27292d216cce2d21f1c7e589f3b66a4a643bff0cdd348efd17aa3"} 1
podman_image_info{id="c4b778290339",parent_id="b260a49eebf9",repository="docker.io/grafana/grafana",tag="latest",digest="sha256:7567a7c70a3c1d75aeeedc968d1304174a16651e55a60d1fb132a05e1e63a054"} 1

# HELP podman_image_created_seconds Image creation time in unixtime.
# TYPE podman_image_created_seconds gauge
podman_image_created_seconds{id="48565a8e6250",repository="docker.io/bitnami/prometheus",tag="latest"} 1.655436988e+09
podman_image_created_seconds{id="62aedd01bd85",repository="docker.io/library/busybox",tag="latest"} 1.654651161e+09
podman_image_created_seconds{id="75c013514322",repository="docker.io/library/sonarqube",tag="latest"} 1.654883091e+09
podman_image_created_seconds{id="a45fa0117c2b",repository="localhost/podman-pause",tag="4.1.0-1651853754"} 1.655484887e+09
podman_image_created_seconds{id="b260a49eebf9",repository="docker.io/library/httpd",tag="latest"} 1.655163309e+09
podman_image_created_seconds{id="c4b778290339",repository="docker.io/grafana/grafana",tag="latest"} 1.655132996e+09

# HELP podman_image_size Image size
# TYPE podman_image_size gauge
podman_image_size{id="48565a8e6250",repository="docker.io/bitnami/prometheus",tag="latest"} 5.11822059e+08
podman_image_size{id="62aedd01bd85",repository="docker.io/library/busybox",tag="latest"} 1.468102e+06
podman_image_size{id="75c013514322",repository="docker.io/library/sonarqube",tag="latest"} 5.35070053e+08
podman_image_size{id="a45fa0117c2b",repository="localhost/podman-pause",tag="4.1.0-1651853754"} 815742
podman_image_size{id="b260a49eebf9",repository="docker.io/library/httpd",tag="latest"} 1.49464899e+08
podman_image_size{id="c4b778290339",repository="docker.io/grafana/grafana",tag="latest"} 2.98969093e+08

network

# HELP podman_network_info Network information.
# TYPE podman_network_info gauge
podman_network_info{driver="bridge",id="2f259bab93aa",interface="podman0",labels="",name="podman"} 1
podman_network_info{driver="bridge",id="420272a98a4c",interface="podman3",labels="",name="network03"} 1
podman_network_info{driver="bridge",id="6eb310d4b0bb",interface="podman2",labels="",name="network02"} 1
podman_network_info{driver="bridge",id="a5a6391121a5",interface="podman1",labels="",name="network01"} 1

volume

# HELP podman_volume_info Volume information.
# TYPE podman_volume_info gauge
podman_volume_info{driver="local",mount_point="/home/navid/.local/share/containers/storage/volumes/vol01/_data",name="vol01"} 1
podman_volume_info{driver="local",mount_point="/home/navid/.local/share/containers/storage/volumes/vol02/_data",name="vol02"} 1
podman_volume_info{driver="local",mount_point="/home/navid/.local/share/containers/storage/volumes/vol03/_data",name="vol03"} 1

# HELP podman_volume_created_seconds Volume creation time in unixtime.
# TYPE podman_volume_created_seconds gauge
podman_volume_created_seconds{name="vol01"} 1.655484915e+09
podman_volume_created_seconds{name="vol02"} 1.655484926e+09
podman_volume_created_seconds{name="vol03"} 1.65548493e+09

system

# HELP podman_system_api_version Podman system api version.
# TYPE podman_system_api_version gauge
podman_system_api_version{version="4.1.1"} 1

# HELP podman_system_buildah_version Podman system buildahVer version.
# TYPE podman_system_buildah_version gauge
podman_system_buildah_version{version="1.26.1"} 1

# HELP podman_system_conmon_version Podman system conmon version.
# TYPE podman_system_conmon_version gauge
podman_system_conmon_version{version="2.1.0"} 1

# HELP podman_system_runtime_version Podman system runtime version.
# TYPE podman_system_runtime_version gauge
podman_system_runtime_version{version="crun version 1.4.5"} 1

License

Licensed under the Apache 2.0 license.

prometheus-podman-exporter's Issues

feat: allow cached scraping of container size

Is your feature request related to a problem? Please describe.
When creating containers one can miss mounting volumes to important locations.

• This way important data can be written to container layer instead of a volume or a mount. If the container is deleted (or stopped with the --rm flag) the data is gone with it.
• The container could also have logs or other data that grows without bound that is not part of volume and may be difficult to resize.

Ideally containers would be of size 0 with all data in volumes but this is rarely the case. This would be a good sanity check.

Describe the solution you'd like
I would like to see tunable cached container size metrics added. Specifically the size attributes from podman ps --size --format=json.

• This is an expensive metric to collect because containers are usually mutable and their size unpredictable so it would need to be cached.
  • Cache duration should be tunable as this will affect different systems differently.
  • The default cache duration should be once every 24 hours.
  • Once the cache expires serve the old data and start fetching new data?
  • Ideally size collection for each container would distributed over the course of the tunable period to avoid sending too much IO to the system at the same time.

Additional context

• It is likely not important for this metric to be collected frequently. I think it will be most helpful with long lived containers.
• Image and volume sizes do not cover this aspect.

Thank you for considering this.

the net_output and net_input what value is it?

I want to make my dashboard more readable, but i can't figure out what value the net_ouput and net_input is

is it in kb ? byte ?

never mind, it is in bytes/s

feat: Add image_id to podman_container_info

Is your feature request related to a problem? Please describe.
I'm trying to have the leanest dashboard possible
I gather infos about the running containers, but i'd love to be able to show container image age
So far, to achieve this, I have to create a separate panel to display container image information

Describe the solution you'd like
If podman_container_info could contain used image id, I would be able to create a query that would display container image info in the same panel

Additional context
Here are the two panels I'd love to combine :
container info :

image info :

Thank you very much for taking this request into consideration

noob questions (sorry): How do you filter podman_container_state on value ?

again sorry for the noob question here

so i just want to do this

podman_container_state=2

so it only shows me the running containers.

i've looked at examples of other querys, like cpu times and it just states the metrics name and then the value like

cpu_memory_time>300

but i can't get it wokring with the podman_container_state

Recurring error in logs "error gathering metrics....was collected before with the same name and label values"

Describe the bug
Every 30 seconds in the logs I get the following error:

Aug 21 08:21:50 lvps podman-exporter_run[51905]: ts=2022-08-21T15:21:50.073Z caller=stdlib.go:105 level=error msg="error gathering metrics: 2 error(s) occurred:\n* [from Gatherer #2] collected metric \"podman_image_size\" { label:<name:\"id\" value:\"9c6f07244728\" > gauge:<value:5.830724e+06 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"podman_image_created_seconds\" { label:<name:\"id\" value:\"9c6f07244728\" > gauge:<value:1.660065593e+09 > } was collected before with the same name and label values"
Aug 21 08:21:50 lvps podman[51871]: ts=2022-08-21T15:21:50.073Z caller=stdlib.go:105 level=error msg="error gathering metrics: 2 error(s) occurred:\n* [from Gatherer #2] collected metric \"podman_image_size\" { label:<name:\"id\" value:\"9c6f07244728\" > gauge:<value:5.830724e+06 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"podman_image_created_seconds\" { label:<name:\"id\" value:\"9c6f07244728\" > gauge:<value:1.660065593e+09 > } was collected before with the same name and label values"

To Reproduce

Run prometheus-podman-exporter with --collector.enable-all. Have two identical images with different tags:

[ansible@lvps ~]$ podman images | grep 9c6f072
docker.io/library/alpine     3.16          9c6f07244728  11 days ago        5.83 MB
docker.io/library/alpine     latest        9c6f07244728  11 days ago        5.83 MB

Removing one of the images makes the error go away.

Expected behavior

I wouldn't expect error messages every 30 seconds in the logs for the above duplicate images. I believe it's not abnormal to have a latest tag that matches a numbered tag for a given image.

Desktop (please complete the following information):

• OS CentOS Stream 9
• Podman Version 4.1.1

Thanks!

image names are parsed incorrectly for repositories that have port numbers.

Describe the bug
Incorrect parsing of repository name for images

To Reproduce
Include a port number in repository name

Expected behavior
Port number should be part of repo name not part of tag

Screenshots
Example metric from this exporter:
podman_image_info{id="20091fb4b6c2",repository="registry.test.example.org",tag="5000/example/ipxe"} 1
The registry name is actually registry.test.example.org:5000 the image is example/ipxe and the tag is not in the parsed value.

Desktop (please complete the following information):

• Fedora 36
• Podman 4.1.1

Additional context
Podman output:

# podman images |grep ipxe
registry.test.example.org:5000/example/ipxe                        20220428            20091fb4b6c2  5 days ago     507 MB

Grafana Dasboard

I created this Grafana Dashboard for the exporer. Haven't found any on the internet.
https://grafana.com/grafana/dashboards/17138-podman/

feat: enhance metrics ( include more labels )

Is your feature request related to a problem? Please describe.
For example, New Relic does not support joining two metrics with the same label ATOW. This makes it very hard to understand graphs only with containerid legend.

Describe the solution you'd like
Would be superb to enhance all metrics with the same fields as for podman_container_info metric This should also include a feature request: #34.

--enhance-metrics=true

Additional context
Add any other context or screenshots about the feature request here.

make `--collector.image` option use less IO and time by not calculating image size on every scrape

Is your feature request related to a problem? Please describe.
When the --collector.image option is used the scrape time increases significantly along with I/O. For me hundreds of MBs of reads happen on every scrape.

• I am not certain if containers/storage#1060 is required for this to be done.
  • here uidmap is listed as part of the problem but we have proper uidmap support now and it is still a problem.
• containers/podman#13810 - tried to make size calculation optional in the podman repo for podman image ls but the change was reverted before it shipped.

Describe the solution you'd like
Do size calculation only when new images are added or detected. (I presume they should not change size without the ID changing).

Additional context
Using podman version 4.4.1

feature: add status change timestamp

A feature of e.g. podman ps -a is to show the relative time of the last state change (Up 3 minutes ago).

Would an additional gauge with this timestamp be possible?

Right now I can't detect restart loops because the state is not exited long enough for prometheus to notice.

Thanks for this project! :)

Generate multi-arch Container images

Is your feature request related to a problem? Please describe.
The published container image is only build for amd64 arch. This can't run on other arch like raspberry pi 4 with arm64 arch

Describe the solution you'd like
Build and publish the container image for multiple arch

Additional context
I might work on this but I see any existing pipeline to build and publish the container image

Exporter crashes when attempting to export metrics for a pod with no namespace

Describe the bug
The exporter will crash as soon as the metrics endpoint is accessed if the --collector.pod tag is enabled, if a pod is created without an infrastructure container and namespace.

These are the default configurations used in the podman-compose up and podman-compose systemd commands in the podman compose package.

To Reproduce

Create at least one pod with no infrastructure container.

podman pod create --infra=false --share="" <pod_name>

podman pod ls -ns
POD ID        NAME           STATUS      CREATED       INFRA ID               CGROUP      NAMESPACES  # OF CONTAINERS
5e76f99ca  <pod_with_infra>     Running     44 hours ago  997264653234c603  user.slice          2
1f3fb57f3  <pod_with_no_infra>  Running     5 days ago                          user.slice              1

prometheus-podman-exporter -o -d

Expected behavior
The metrics endpoints are displayed.

Error Logs

prometheus-podman-exporter -o -d
ts=2022-10-20T16:31:34.408Z caller=exporter.go:57 level=info msg="Starting podman-prometheus-exporter" version="(version=1.0.0, branch=, revision=1)"
ts=2022-10-20T16:31:34.408Z caller=handler.go:93 level=info msg="enabled collectors"
ts=2022-10-20T16:31:34.408Z caller=handler.go:104 level=info collector=container
ts=2022-10-20T16:31:34.408Z caller=handler.go:104 level=info collector=pod
ts=2022-10-20T16:31:34.408Z caller=exporter.go:68 level=info msg="Listening on" address=:9882
ts=2022-10-20T16:31:34.409Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
ts=2022-10-20T16:31:51.484Z caller=handler.go:34 level=debug msg="collect query:" filters="unsupported value type"
panic: runtime error: slice bounds out of range [:12] with length 0

goroutine 48 [running]:
github.com/containers/prometheus-podman-exporter/pdcs.Pods()
        /builddir/build/BUILD/prometheus-podman-exporter/pdcs/pod.go:30 +0x396
github.com/containers/prometheus-podman-exporter/collector.(*podCollector).Update(0xc00026b0e0, 0x0)
        /builddir/build/BUILD/prometheus-podman-exporter/collector/pod.go:58 +0x38
github.com/containers/prometheus-podman-exporter/collector.execute({0x14bfd51, 0x0}, {0x16ddc60, 0xc00026b0e0}, 0x0, {0x16ddee0, 0xc000269780})
        /builddir/build/BUILD/prometheus-podman-exporter/collector/collector.go:115 +0x9c
github.com/containers/prometheus-podman-exporter/collector.PodmanCollector.Collect.func1({0x14bfd51, 0x0}, {0x16ddc60, 0xc00026b0e0})
        /builddir/build/BUILD/prometheus-podman-exporter/collector/collector.go:103 +0x3d
created by github.com/containers/prometheus-podman-exporter/collector.PodmanCollector.Collect
        /builddir/build/BUILD/prometheus-podman-exporter/collector/collector.go:102 +0xd5

Desktop (please complete the following information):

• OS RHEL 9.0
• Podman Version 4.1.1

Exporter does not list all rootless containers running under different users

I have a RHEL8 servers with podman and I have several rootless containers running under different users (this is a security requirement in my env).

The podman-exporter is currently running in a container under root account and it can only list metrics related to the cotnainers running under root.

Is there a way to scape the metrics from all rootless containers running under different users?

Flag `--version` require conmon to be installed

Describe the bug
The flag --version can't be used in a CI as it needs the conmon utility to be available, and it seems that we are not exposing any conmon information at this stage.

To Reproduce
Steps to reproduce the behavior:

Compile the program and try to use the flag --version

$ ./bin/prometheus-podman-exporter --version
2022/10/11 12:11:59 could not find a working conmon binary (configured options: [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon]): invalid argument

Expected behavior

$ bin/prometheus-podman-exporter --version
prometheus-podman-exporter (version=1.2.0, branch=main, revision=dev.1)

Additional context
Building inside of a rockylinux:8 container image, without conmon installed.

Pod names are not displayed in all metrics

Hello,

We need to have the pod name appear in all metrics. Currently only one metric contain the name of the pod and this is the 'podman_container_info'.

It is vital to have the podname in all other metrics so as to display the podname in the chart ledgent in grafana!

That will avoid unnecessary joins of type:

join(podman_container_mem_limit_bytes{id=~"$id"}, 1s, podman_container_info{id=~"$id"})

Thank you!
Georgios

Exclude infra containers?

Is your feature request related to a problem? Please describe.
More of a question, but is it ever useful to monitor infra containers? It's easy enough to filter those but if they never do anything useful then it could be nice to prune them here.

Unable to get this working with podman.sock

Struggling to get this running just on localhost to test it with user podman.sock. I'm not sure what to use for CONTAINER_HOST`` so I assumed the unix://` prefix and bind mounted the podman socket. Am I missing something obvious here?

$ podman run -it --rm -e CONTAINER_HOST=unix:///run/podman/podman.sock -v /run/user/1000/podman/podman.sock:/run/podman/podman.sock:Z quay.io/navidys/prometheus-podman-exporter:latest
2022/07/21 21:14:10 unable to connect to Podman socket: Get "http://d/v4.1.1/libpod/_ping": dial unix /run/podman/podman.sock: connect: permission denied

Thanks!
(Fedora 36, podman 4.1.1)

feat: Add support for HTTPS & authentication

Is your feature request related to a problem? Please describe.
Currently, the only way to add encryption or authentication to the exporter is to introduce a reverse proxy.
I would like to do this without relying on yet an additional tool.

Describe the solution you'd like
Add native support for HTTPS & authentication, using available features from the exporter_toolkit, which is already a dependency for prometheus-podman-exporter.

Additional context
n/a.

Exporter only returns stats about pods from current running user?

Describe the bug
(I'm sorry to call this a "bug"; it's likely just a lack of my understanding over how to use this exporter successfully)

When I run prometheus-podman-exporter, it only exports container-level stats about the containers that are being run as the same user that I'm running the exporter as. Therefore, I don't know how to deploy prometheus-podman-exporter as a server-level monitoring tool -- it seems that I would need to run an instance of it for every user that could run pods?

To Reproduce
Steps to reproduce the behavior:

• Run prometheus-podman-exporter as a user account (ie. non-root user)
• Observe metrics like podman_container_cpu_seconds_total only reports id=* for containers running as the same user
• Repeat, running as root -- even as a superuser, only root's containers are displayed

Expected behavior

I expected that running as root would return metrics from all pods. Alternatively, I expected the documentation in the repo to indicate the right way, or the expected runtime model, for prometheus-podman-exporter to be used effectively for monitoring at a server-level.

Screenshots
N/A; can provide if needed.

Desktop (please complete the following information):
NixOS 22.11

Additional context
I'm currently working on packaging prometheus-podman-exporter for NixOS. Building the software was straight-forward, but determining how to set up a systemd unit to run it has left me confused because of this. While it could run as a root user, which isn't typically how I'd want it to run anyway, it also wouldn't collect info on all the containers... leaving me a little confused on the recommended packaging approach.

Support Ubuntu Jammy 22.04 / 22.10

Is your feature request related to a problem? Please describe.
Currently, Ubuntu is supporting 3.4.x podman. And podman exporter errors out with

❯ podman run -it -e CONTAINER_HOST=unix:///run/podman/podman.sock -v $XDG_RUNTIME_DIR/podman/podman.sock:/run/podman/podman.sock --userns=keep-id --security-opt label=disable -u 1000 quay.io/navidys/prometheus-podman-exporter
2022/11/24 10:18:27 unable to connect to Podman socket: server API version is too old. Client "4.0.0" server "3.4.4"

But the api is accessible and works well.. may be some parts might not work in podman v3.4
e.g.
curl --unix-socket /run/user/1000/podman/podman.sock http://d/v4.3.1/libpod/info

Describe the solution you'd like
Allow to use podman-exporter even with ubuntu 22.04 / 22.10

Additional context
Add any other context or screenshots about the feature request here.

feat: add a metric representing a container's health

Is your feature request related to a problem? Please describe.
I would like to be able to query a container's health in addition to its state.

Describe the solution you'd like
I propose to add a new gauge metric named podman_container_health. The metric would map the values defined in podman/libpod/define/healthchecks.go (healthy, unhealthy & starting) to integers and the help message would help map those values back to their intended meaning (similar to the way podman_container_state is defined).

Additional context
n/a

Add metric for volume storage size

I'd like to be able to monitor the amount of storage currently being utilized by my volumes. Looks like this data should be available:
Command line: podman system df -v
API: https://docs.podman.io/en/latest/_static/api.html#tag/system/operation/SystemDataUsageLibpod

prometheus-podman-exporter/vendor/github.com/containers/podman/v4/pkg/domain/entities/engine_container.go

Line 95 in 38ae028

SystemDf(ctx context.Context, options SystemDfOptions) (*SystemDfReport, error)

[Question] Is there anyway to make this work with podman v3.0.1?

Hello, I'm using Podman on Debian 11 and Podman v4x hasn't been added to the Debian repository yet, however, this project satisfies all of my container monitoring needs, I know you currently only support podman v4x but I was wondering if I use the image version since I have to enable and mount the podman.sock, would this exporter work with Podman v3.0.1 in container mode with podman.sock enabled and mounted?

podman_container_mem_usage_bytes metrics disappeared

Describe the bug
I used to query podman_container_mem_usage_bytes in prometheus, but now noticed it's not exposed anymore. a manual crawl of the /metrics file confirms this.

To Reproduce
unfortunately I can't really say what has changed. First I assumed a permission problem on the podman socket and created a separate one only for the prometheus-podman-exporter container:

/ $ id
uid=65534(nobody) gid=65534(nobody)
/ $ ls -la /run/podman/podman.sock 
srw-------    1 nobody   nobody           0 Dec  5 17:35 /run/podman/podman.sock
/ $ 

logs (last 7 lines repeat):

ts=2022-12-05T17:35:47.710Z caller=exporter.go:63 level=info msg="Starting podman-prometheus-exporter" version="(version=1.3.0, branch=main, revision=dev.1)"
ts=2022-12-05T17:35:47.711Z caller=handler.go:93 level=info msg="enabled collectors"
ts=2022-12-05T17:35:47.711Z caller=handler.go:104 level=info collector=container
ts=2022-12-05T17:35:47.711Z caller=handler.go:104 level=info collector=image
ts=2022-12-05T17:35:47.711Z caller=handler.go:104 level=info collector=network
ts=2022-12-05T17:35:47.711Z caller=handler.go:104 level=info collector=pod
ts=2022-12-05T17:35:47.711Z caller=handler.go:104 level=info collector=system
ts=2022-12-05T17:35:47.711Z caller=handler.go:104 level=info collector=volume
ts=2022-12-05T17:35:47.711Z caller=exporter.go:74 level=info msg="Listening on" address=127.0.0.1:9882
ts=2022-12-05T17:35:47.712Z caller=tls_config.go:232 level=info msg="Listening on" address=127.0.0.1:9882
ts=2022-12-05T17:35:47.712Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=127.0.0.1:9882
ts=2022-12-05T17:35:58.307Z caller=handler.go:34 level=debug msg="collect query:" filters="unsupported value type"
ts=2022-12-05T17:35:58.312Z caller=collector.go:135 level=debug msg="collector succeeded" name=network duration_seconds=0.002719641
ts=2022-12-05T17:35:58.323Z caller=collector.go:135 level=debug msg="collector succeeded" name=pod duration_seconds=0.013724029
ts=2022-12-05T17:35:58.349Z caller=collector.go:135 level=debug msg="collector succeeded" name=volume duration_seconds=0.040115157
ts=2022-12-05T17:35:58.363Z caller=collector.go:135 level=debug msg="collector succeeded" name=container duration_seconds=0.053379088
ts=2022-12-05T17:35:58.511Z caller=collector.go:135 level=debug msg="collector succeeded" name=system duration_seconds=0.202251355
ts=2022-12-05T17:35:58.558Z caller=collector.go:135 level=debug msg="collector succeeded" name=image duration_seconds=0.24884938

testing the socket itself seems OK

$ echo -e "GET /containers/json HTTP/1.0\r\n" | podman unshare nc -U ${SOCKET} 
HTTP/1.0 200 OK
Api-Version: 1.41
Content-Type: application/json
Libpod-Api-Version: 4.3.1
Server: Libpod/4.3.1 (linux)
X-Reference-Id: 0xc0003c8000
Date: Mon, 05 Dec 2022 18:13:37 GMT

[{"Id":"1ef8fdf2102ba787e29125f3def643e6a4c5da4b22266daccffd2b24423b2549","Names" [...]

Expected behavior
have podman_container_mem_usage_bytes available in metrics exposed by prometheus-podman-exporter

environment

• CentOS 9 Stream
• Podman Version 4.3.1
• exporter run in a podman container: /bin/podman_exporter --collector.enable-all --collector.store_labels --debug --web.listen-address 127.0.0.1:9882

Additional context
Any help in debugging this is welcome

feat: export also container/image labels

Is your feature request related to a problem? Please describe.
In an environment where the only determining field is a label it hard to visualize metrics
from prometheus-podman-exporter, because it does not export labels.

Describe the solution you'd like
Would be superb to have an option to export also container/image labels, like cAdvisor does:

-store_container_labels=false -whitelisted_container_label="com.docker.compose.config-hash,com.docker.compose.container-number,com.docker.compose.oneoff,com.docker.compose.project,com.docker.compose.service"

Additional context
No

arm64 images

Is your feature request related to a problem? Please describe.
It would be really appreciated if there were builds for arm64 architecture.

Describe the solution you'd like
Official arm64 builds (I was not able to get them running by building images myself, using same Containerfile but "arm64" as ARCH variable)

Thanks and best regards

feat: implement federation endpoint to get a scrape from another prometheus

Is your feature request related to a problem? Please describe.
Prometheus podman exporter does not have a federation endpoint to be scraped from other Prometheus servers

Describe the solution you'd like
Add an /federation Endpoint