splunk / collectd-plugins

SAI x Collectd: https://docs.splunk.com/Documentation/InfraApp/latest/Admin/ManageAgents

Collectd-plugins

C plugins for collectd:

• write_splunk: write plugin: Write data to Splunk over HEC or UDP
• processmon: read plugin: monitor processes running on your system. Linux only.
• docker: read plugin: monitor docker containers running in your system.

How to build the plugins

1. Clone the collectd repo from github: git clone https://github.com/collectd/collectd.git

2. Switch to collectd-5.9 branch: cd collectd && git checkout collectd-5.9 For collectd-5.8, use collectd-5.8 branch here.

3. Copy the plugin source files to the collectd repo: cp ${collectd_plugins_dir}/src/* src/

4. Apply the patch file to the collectd repo: git apply ${collectd_plugins_dir}/add-splunk-plugins.patch

5. Compile collectd: ./build.sh && ./configure && make Use ./configure --help to see the configuration options. Read the collectd README file to see the requirements to compile it. For docker plugin: libyajl2 and libcurl dev packages are required. For write_splunk plugin: libcurl dev paackage is required.

6. Run Tests: make check-TESTS

7. Install collectd: make install

8. Update settings for the plugins in collectd.conf file: vi /opt/collectd/etc/collectd.conf Make sure you enable write_splunk and other plugins you want and also update the token, server etc. for write_splunk:

<LoadPlugin "write_splunk">
        FlushInterval 30
</LoadPlugin>

9. Start collectd: /opt/collectd/sbin/collectd

Example: Getting started on new Ubuntu 16.04 installation

apt-get update
git clone https://github.com/collectd/collectd.git
cd collectd
apt-get install -y autoconf libtool pkg-config bison byacc flex
apt-get install -y libyajl-dev libcurl4-openssl-dev
git checkout collectd-5.9
cp /collectd-plugins/src/* src/
git apply /collectd-plugins/add-splunk-plugins.patch
./build.sh
./configure CFLAGS="-g -O2 -fstack-protector -D_FORTIFY_SOURCE=2 -fpie" LDFLAGS="-Wl,-z,relro"
make
make check-TESTS
make install
vi /opt/collectd/etc/collectd.conf
/opt/collectd/sbin/collectd

collectd-plugins's Issues

processmon fails in case of short-living process disappears and is suspended for 2x interval then

Looks like shortliving processes are not proper catched by processmon

See log:

Dec 07 12:40:06 *** collectd[1844373]: Initialization complete, entering read-loop.
Dec 07 12:40:06 *** collectd[1844373]: processmon plugin: Error reading /proc/1844371/stat
Dec 07 12:40:06 *** collectd[1844373]: processmon plugin: Error reading /proc/1844372/stat
Dec 07 12:40:06 *** collectd[1844373]: read-function of plugin `processmon' failed. Will suspend it for 120.000 seconds.

A patch which catch this case is like following:

diff --git a/processmon.c.orig b/processmon.c
index 754d732..2092b88 100644
--- a/processmon.c.orig
+++ b/processmon.c
@@ -233,6 +233,12 @@ static int pm_read_stat(process_data_t *pd, char *process_state) {
   // read the proc stat file for pid.
   snprintf(fname, sizeof(fname), "/proc/%s/stat", pd->pid);

+  // check in advance whether it is still existing
+  if (access(fname, F_OK) != 0) {
+    WARNING("processmon plugin: Disappeared before reading %s", fname);
+    return 0;
+  };
+
   ssize_t status = read_file_contents(fname, buf, sizeof(buf) - 1);
   if (status <= 0) {
     ERROR("processmon plugin: Error reading %s", fname);

afterwards following appears in log, processmon is no longer suspended:

Dec 07 13:57:36 *** collectd[1847744]: processmon plugin: File disappeared before reading reading /proc/1847741/stat

write_splunk not sending metrics for docker containers on aarch64 compute

Attempting to configure collectd daemon on docker aarch64 containers with dockerplugin and write_splunk plugin to capture metrics. Currently, seeing inconsistent/broken metrics after collectd has been provisioned to send metrics to HEC with plugin types.

The write_splunk.so file was compiled using the instructions provided by the README.md section of this repository.

As it stands, the only behavior that might provide insight into consistent/broken metrics is the collectd's TCP connection is stuck in a CLOSE_WAIT state to the HEC>

What troubleshooting steps or configuration changes can be made to address the docker metrics collection on aarch64?

NOTE: This is a repost from user "ghbfnnp"

processmon.count do not contains process_name dimension

Hi,
First of all, big thanks for the product.
I have the enhancement request from real-word ITSI/SAI project.

Use Case:
A company running several app instances with a different configuration via CMD line
ps output looks like this:
/opt/sdp/sdpnode /opt/sdp/conf/package_name1.pkg
/opt/sdp/sdpnode /opt/sdp/conf/package_name2.pkg

And they want to track in SAI and ITSI if they have a particular count of package_name1 or package_2 are running on the host.

A configuration sample for OOTB processes plugin for collectd that solve the issue looks like this:
ProcessMatch "package_1" ".*/sdpnode.*package_name1.pkg"

Then they have a set of metrics like this: processes.package_1.ps_count that isn't good because Please add process_name as a dimension to processmon.count metrics.

Test issue

Testing, please disregard

write_splunk.solaris.so plugin provided for Solaris is not of the correct architecture.

write_splunk plugin from splunk app for infrastructure provided for solaris is not of the correct architecture.

collectd: /opt/csw/lib/collectd/write_splunk.so: wrong ELF data format: ELFDATA2LSB. The most common cause for this problem is missing dependencies. Use ldd(1) to check the dependencies of the plugin / shared object.

/export/pkgs/splunk/install/ufwinstall# ldd /opt/csw/lib/collectd/write_splunk.so
ldd: /opt/csw/lib/collectd/write_splunk.so: ELF machine type: EM_386: is incompatible with system
This write_splunk.so is copied over by Splunk's install_agent.sh:
SunOS)
$_sudo cp collectd.conf /etc/opt/csw/
$_sudo cp "/export/pkgs/splunk/install/ufwinstall/write_splunk-solaris.so" "/opt/csw/lib/collectd/write_splunk.so"

Using SunOS knpdb11 5.11 11.4.5.3.0 sun4v sparc sun4v

We believe that plugin is supported by Solaris under an x86_64 architecture, but not SPARC.

thinking of creating a build for SPARC. Below is the link
https://github.com/splunk/collectd-plugins

Note: Universal Forwarders: 8.03
Heavy Forwarders: 8.04
Indexer: 8.03

No upgrade have been performed. Heavy forwarders were implemented after the uf's and indexer, hence the newer version.

Would like to know if there is a sparc version of this plugin available.

‘strncpy’ specified bound 128 equals destination size

#OS: Fedora 31
make check-TESTS command ends with this error.

src/write_splunk_test.c: In function ‘test_splunk_transforms’:
src/write_splunk_test.c:201:7: error: ‘strncpy’ specified bound 128 equals destination size [-Werror=stringop-truncation]
201 | strncpy(vl.plugin_instance, cases[i].plugin_instance,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
202 | sizeof(vl.plugin_instance));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/write_splunk_test.c:204:7: error: ‘strncpy’ specified bound 128 equals destination size [-Werror=stringop-truncation]
204 | strncpy(vl.type, cases[i].type, sizeof(vl.type));
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/write_splunk_test.c:206:7: error: ‘strncpy’ specified bound 128 equals destination size [-Werror=stringop-truncation]
206 | strncpy(vl.type_instance, cases[i].type_instance,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
207 | sizeof(vl.type_instance));
| ~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors