jdmaloney / slurm_tig Goto Github PK

I am encountering a metric parse error in the Telegraf inputs.exec plugin, preventing it from collecting Slurm data correctly. The error message indicates an issue with the format of the data being collected by the slurm_detail_stats.sh script.

Details:

The error message "expected field at 1:189" suggests that Telegraf is encountering an unexpected character or element in the data stream at position 189. This could be due to extra characters, malformed data, or an incompatibility between the script's output format and Telegraf's expectations:

sysadmin@mdrvpremst01:/etc/telegraf/slurm$ sudo systemctl status telegraf
● telegraf.service - Telegraf
Loaded: loaded (/lib/systemd/system/telegraf.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-07-16 16:31:15 CEST; 9min ago
Docs: https://github.com/influxdata/telegraf
Main PID: 1714435 (telegraf)
Tasks: 10 (limit: 4561)
Memory: 48.0M
CPU: 5.621s
CGroup: /system.slice/telegraf.service
└─1714435 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d

Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded inputs: cpu disk diskio exec (4x) kernel mem processes swap system
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded aggregators:
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded processors:
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded secretstores:
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Loaded outputs: influxdb_v2
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! Tags enabled: host=mdrvpremst01
Jul 16 16:31:15 mdrvpremst01 systemd[1]: Started telegraf.service - Telegraf.
Jul 16 16:31:15 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:31:15Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"mdrvpremst01", Flush Interval:10s
Jul 16 16:35:00 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:35:00Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:189: "slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=l>
Jul 16 16:40:00 mdrvpremst01 telegraf[1714435]: 2024-07-16T14:40:00Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:189: "slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=l>

Here is the output from the script that Telegraf is attempting to parse:

sysadmin@mdrvpremst01:/etc/telegraf/slurm$ ./slurm_detail_stats.sh
slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=long job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=highmem job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=short job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre1.vhio.org,partition=all job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0
slurm_detail_node_data,node=bioinfpre2.vhio.org,partition=long job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre2.vhio.org,partition=highmem job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre2.vhio.org,partition=short job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre2.vhio.org,partition=all job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0
slurm_detail_node_data,node=bioinfpre3.vhio.org,partition=long job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre3.vhio.org,partition=highmem job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre3.vhio.org,partition=short job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0,master_state=IDLE,primary_state=IDLE,other_states=none
slurm_detail_node_data,node=bioinfpre3.vhio.org,partition=all job_count=0,cores_used=0,mem_allocated=0,cores_avail_total=6,mem_avail_total=10240,gpu_used=0,gpu_avail_total=0

My InfluxDB shows that tables have been created and data has been successfully added from other scripts:

SHOW MEASUREMENTS
name: measurements
name

cpu
disk
diskio
kernel
mem
processes
slurm_job_accounting_data
slurm_jobsumdata
slurm_nodedata
slurm_nodesumdata
slurm_partition_node_state
slurm_responsiveness
slurm_scheduler_data
slurm_userjobdata
swap
system

I need assistance in identifying and resolving the cause of the metric parse error so that Telegraf can properly handle the slurm_detail_node_data metrics.

Thanks,