jdmaloney / slurm_tig Goto Github PK
View Code? Open in Web Editor NEWTelegraf checks and Grafana dashboards for monitoring Slurm HPC clusters
Telegraf checks and Grafana dashboards for monitoring Slurm HPC clusters
I am encountering a metric parse error in the Telegraf inputs.exec plugin, preventing it from collecting Slurm data correctly. The error message indicates an issue with the format of the data being collected by the slurm_detail_stats.sh script.
Details:
The error message "expected field at 1:189" suggests that Telegraf is encountering an unexpected character or element in the data stream at position 189. This could be due to extra characters, malformed data, or an incompatibility between the script's output format and Telegraf's expectations:
sysadmin@mdrvpremst01:/etc/telegraf/slurm$ sudo systemctl status telegraf
● telegraf.service - Telegraf
Loaded: loaded (/lib/systemd/system/telegraf.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-07-16 16:31:15 CEST; 9min ago
Docs: https://github.com/influxdata/telegraf
Main PID: 1714435 (telegraf)
Tasks: 10 (limit: 4561)
Memory: 48.0M
CPU: 5.621s
CGroup: /system.slice/telegraf.service
└─1714435 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded inputs: cpu disk diskio exec (4x) kernel mem processes swap system
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded aggregators:
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded processors:
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded secretstores:
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded outputs: influxdb_v2
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Tags enabled: host=mdrvpremst01
Jul 16 16:31:15 mdrvpremst01 systemd[1]: Started telegraf.service - Telegraf.
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"mdrvpremst01", Flush Interval:10s
Jul 16 16:35:00 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:35:00Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:189: "slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=l>
Jul 16 16:40:00 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:40:00Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:189: "slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=l>
Here is the output from the script that Telegraf is attempting to parse:
sysadmin@mdrvpremst01:/etc/telegraf/slurm$ ./slurm_detail_stats.sh
slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=long job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=highmem job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=short job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=all job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0
slurm_detail_node_data,node=bioinfpre2.vhio.org,partition=long job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre2.vhio.org,partition=highmem job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre2.vhio.org,partition=short job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre2.vhio.org,partition=all job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0
slurm_detail_node_data,node=bioinfpre3.vhio.org,partition=long job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre3.vhio.org,partition=highmem job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre3.vhio.org,partition=short job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre3.vhio.org,partition=all job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0
My InfluxDB shows that tables have been created and data has been successfully added from other scripts:
SHOW MEASUREMENTS
name: measurements
name
cpu
disk
diskio
kernel
mem
processes
slurm_job_accounting_data
slurm_jobsumdata
slurm_nodedata
slurm_nodesumdata
slurm_partition_node_state
slurm_responsiveness
slurm_scheduler_data
slurm_userjobdata
swap
system
I need assistance in identifying and resolving the cause of the metric parse error so that Telegraf can properly handle the slurm_detail_node_data metrics.
Thanks,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.