Incorrect telegraf service termination during operating system reboot about telegraf

[powersj]

Hi,

Can you reproduce this case and give me answer, why telegraf service does-not stop their service correctly?

I do not have access to Windows Server or Exchange, so I will not be able to reproduce on a similar system. I do have access to Windows 11 and I do not see any difference in reboot time between before the service was installed, while the service was installed, and again after uninstalling the service.

My suggestion to you is to do the following:

1. Does this reproduce on the CLI? Namely, run telegraf for few collection cycles and then hit Ctrl-C. Does it close relatively quickly?
2. Start removing plugins from your config and see if you can narrow it down if one particular one is causing the issue. Start with your single output, switch to say [[outputs.file]] and see if that helps. Then start removing the perf counters one by one.

[Erikov-K]

Hi!

If I starting telegraf using CLI or PowerShell, it work as expected.
I can terminate telegraf process using Ctrl + C and it terminating successfully.

Following your recommendations, I replaced output plugin to [[outputs.file]] (instead of [[outputs.prometheus_client]]) and then stay only one processor-object from [[inputs.win_perf_counters.object]].

Telegraf config file now looks like this:

[global_tags]

[agent]
interval = "15s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "15s"
flush_jitter = "0s"
precision = ""
debug = true
quiet = false
logfile = "/Program Files/Telegraf/telegraf-metrics.log"
logfile_rotation_max_size = "1MB"
logfile_rotation_max_archives = 5
hostname = ""
omit_hostname = false

[[outputs.file]]
files = ["C:/Program Files/Telegraf/telegraf-metrics_outputs.file.txt"]
data_format = "influx"

[[inputs.win_perf_counters]]

[[inputs.win_perf_counters.object]]
ObjectName = "Processor"
Instances = ["*"]
Counters = [
"% Idle Time"
]
Measurement = "win_cpu"

Then I started telegraf service and send to server reboot command (delayed for 30 seconds).

• As you can see at screenshot, command to start telegraf service was invoked at 05:28:19.
  Then telegraf service writed at it own log-file, that it started at 05:28:21.
• At 05:28:44 command to restart operating system (with 30 seconds delay) was invoked.
• At 05:29:16 telegraf service writed at it own log-file, that it stopped successfully.
• At 05:29:19 and at 05:30:03 I asked operating system about telegraf service status, using command get-service, and get information, that telegraf service still continue running.
  Also you can see on server console window, that it still waiting stopping services during reboot process.

Below I write steps, which you can use to reproduce my case.

Step 1

[DateTime]::UtcNow.ToString('u')
Get-Service -name telegraf-metrics | Stop-Service -PassThru -Verbose

Step 2

Change telegraf-config

Step 3

[DateTime]::UtcNow.ToString('u')
Get-Service -name telegraf-metrics | Start-Service -PassThru -Verbose

Step 4

[DateTime]::UtcNow.ToString('u')
Invoke-Expression -Command "shutdown /r /t 30"

Step 5

[DateTime]::UtcNow.ToString('u')
Get-Service -name telegraf-metrics

Attachments:

• telegraf-metrics.log
• telegraf-metrics_outputs.file.txt
• Screenshot

[Erikov-K]

Hi!

I did some research.

I consistently rolled back the telegraf version to an earlier one and It was experimentally determined that starting from version v1.18.1 the problem is reproducible stably.

On version 1.18.0, the telegraf service is terminated correctly by the operating system and the operating system reboot process occurs without freezing.
When the server is rebooted with telegraf service on version 1.18.0, Event 7036 appears in the operating system event log with information about the successful stop of the telegraf service “The Telegraf for Metrics service entered the stopped state.”