griesbacher / histou

Adds templates to Grafana in combination with nagflux

Histou

Adds templates to Grafana in combination with Nagflux.

Histou is designed to add templates to Grafana from Nagios Data. Therefor Nagflux sends the informations from Nagios/Icinga(2) to an InfluxDB. On the otherhand, Grafana is used to display this performancedata. Histou adds a custom dashboard to Grafana which will display dynamic dashboards, depending on the given host and service name in the URL. Histou fetches first additional informations from the InfluxDB and select afterwards a template. Templates can be JSON or PHP Files which contains a Grafana dashboard, the have also a rule, which defines when this template is used.

Installation

Dependencies

• Webserver with PHP 5.3.3+
• PHP commandline tool / phpcgi

Webserver

• The whole Histoufolder accessible by copying to your webserver. On Debian with apache this would mean, to copy the whole Histoufolder to /var/www/

Grafana

• Move the file histou.js in the public/dashboards/ folder within Grafana.
• Depending on the URL the index.php is published on the webserver, change the Variable var url = 'http://localhost/histou/'; in in the histou.js file. If you copied the Histoufolder to the root of your webserver, the default URL is perfect.

OMD - the easy way

Nagflux is fully integrated in OMD-Labs, as well as Histou is. Therefor if you wanna try it out, it's maybe easier to install OMD-Labs.

Configuration

Configfile

Here are some of the important config-options:

Section	Config-Key	Meaning
general	socketTimeout	This timeout defines the duration in seconds, which Histou will wait for a response from the InfluxDB
general	phpCommand	This is the command which is used to call PHP on the commandline. If it's in the PATH php is mostly enough, if not write the full path to the file.
general	tmpFolder	Set a folder path, if the default PHP Tmp folder does not suite you.
general	specialChar	Can be used to create more specific regex within the rules. E.g. $host = '&host&' will be replaced with 'linux-server1' if the select hostname is linux-server1. This works likewise with host, service, command.
general	databaseType	Choose between influxDB and victoriametrics.
general	disablePanelTitle	If this is set to true the PanelTitels are hidden globaly, there is an URL Flag which does it just with the current page. It is usefull to get a bigger Graphpicture.
folder	defaultTemplateFolder	This is the path to the folder containing the default templates
folder	customTemplateFolder	This is the path to the folder containing the custom templates. The templates in this folder will override files in the default folder, if they have the same filename
influxdb	url	You can guess it...
influxdb	hostcheckAlias	This string will be used on hostchecks, because normaly there is no service name for it, but Histou needs one.

URL-Parameters

They are just valid for the current call, you can't change anything permanently.

Name	Type	Meaning
host	String	The hostname to generate the graph for
service	String	The servicename to generate the graph for
height	String	The height of an panel, e.g. default is: "400px"
legend	Bool	If false the legend will be hidden, default: true
annotations	Bool	If false the annotations will be unchecked
debug	Flag	If set an additional Panel will be added to the bottom which contains some debug infos
disablePanelTitle	Flag	If set the Paneltitel will be hidden
specificTemplate	String	If a filename is passed this away, only this template will be used, regardless the rules. E.g. ping.simple. Makes more or less just sens with simple templates
disablePerfdataLookup	Flag	If set no data will be fetched from the InfluxDB, so the only data available in the template is the host and servicename. Works only if specificTemplate is set!
customCSSFile	String	You could give the address of an CSS file which will be loaded into the Grafana context. Here an example:
reduce	Flag	If set distracting elements of a graph will be hidden, used for Thruk.

Templates

There are two types of templates, the simple and the PHP. Simple templates are static, PHP are dynamic. Both cointains a rule, which describes on which host/service combination the template is uses. The type of the template is defined by its file-ending, .simple or .php

Rule

A rule contains four keys: host, service, command and perfLabel. The values describes when the template is used. You can write for example an fixed hostname in the host field if the template should be uses just for one Host. It is also possible to use Regular Expressions in every field. Rules have a hierarchy on the to there is the host on the bottom the perfLabel. If a rule does not match in the hostname the perfLabel beneath will not be checked.

Typical Rules

• One rule for all ping checks, due to the fact that every check returns an pl and rta as perfLabel you can match on them
  • host: .*
  • service: .*
  • command: .*
  • perfLabel: rta, pl
• ping template just for the test systems
  • host: test-.*
  • service: .*
  • command: .*
  • perfLabel: rta, pl

Simple

Simple templates contain a rule followed by an JSON object. The reason for this kind of template is, to over the user an easy way to create dashboards. The tradeoff is, that in the simple template it is not possible to use a template created for service1 to use on service2 if the perfLabels differ between those services.

PHP

In the PHP template you can write PHP code which will be executed when the template gets chosen. You just have to return a JSON string or an object of the build-in PHP dashboard.

Presentation

Here is a presentation I held about Nagflux and Histou (only in German, sorry): http://www.slideshare.net/PhilipGriesbacher/monitoring-workshop-kiel-2016-performancedaten-visualisierung-mit-grafana-influxdb

histou's Issues

Manage protocol and servername dynamically

Hi,

when using with http and https, the histou.js dos not automatically get the correct protocol, leading to some errors when using with thruk.

I suggest the following patch:

--- histou.js   2018-08-23 22:51:48.465967766 +0200
+++ /usr/share/grafana/public/dashboards/histou.js      2018-08-23 22:52:42.155844954 +0200
@@ -12,7 +12,7 @@
         document.documentElement.style.background = '#FFF';
     }

-    var url = 'http://localhost/histou/';
+    var url = location.protocol+'//'+window.location.hostname+'/histou/';
     var configUrl = url+'index.php?host='+host+'&service='+service+'&height='+height+'&legend='+legend+debug+disablePanelTitle+disablePerfdataLookup+specificTemplate+'&annotations='+annotations;

     var flotAddons = url + 'flotAddons.js';

Thanks for your great work!

Show all data for the same service with many hosts - one grafana dashboard

First of all, thanks for the great effort doing this work!

Not an issue with Histou but a question:

I don't get how I can graph e.g. 100 CPU load graphs. Meaning the same service in Naemon in one Grafana dashboard. What would be the histou URL for that? Do I need to create a specific histou template for it? How is that addressed exactly?

Regards, Bjorn

Request: Support added for PNP

It would be great if PNP could be added as a supported Datasource.

https://github.com/sni/grafana-pnp-datasource
https://github.com/lingej/pnp-metrics-api

Math Expression in the Grafana query

Hello,

Is there an easy way to add a math expression to the select query in the template? I cannot find such a function in the code.

For example here is one of my InfluxDB queries from Grafana:

SELECT mean("value") *8 AS "Download Rate" FROM "metrics" WHERE ("host" = 'default-host' AND "service" = 'bandwidth AND "command" = 'check_bandwidth AND "performanceLabel" = 'iso.3.6.1.2.1.31.1.1.1.6.4') AND $timeFilter GROUP BY time($__interval) fill(null)

How to express that value multiplication "*8" in the above query in the template?

Thank you!

Regards,
cyb0x

Remote InfluxDB

Hi,

I managed to install a Thruk/Naemon monitoring station with influxdb/grafana on a remote host because of IO considerations.

Thanks to nagflux, performance data are stored into remote influxdb but I didn't find the right way to configure histou for Thruk with such an architecture.

First, I tried to install histou onto the monitoring station. histou.js has been published to grafana dashboard directory but Thruk doesn't display graphs.

Second, I tried to install histou on influxdb/grafana station, but without success too.

Finally, I wonder if remoting influxdb/grafana is likely to work today ?

jfr

Support for Last, Max and Average

is there a way to display the Last, Max and Average values like pnp does?

How to view histou graphs in grafana

Hello,

I just installed the OMD 2.20 labs edition today. I was trying to get data graphed in grafana and found that histou has templates to make it a bit easier. For some reason, I can not see these templates in grafana. I have read https://labs.consol.de/de/nagios/omd/2015/10/21/activate-nagflux.html and still I am unable to view them in grafana. I've also searched google for hours an I can't figure it out. I'm running this on SLES 11 SP3.

Simple template from Grafana 6 not working (no "rows" array in this schema)

Current Grafana schema didn't contain a "rows" array, instead it have only a high level "panels" array, and panel objects have a "row" attribute. So if I design a dashboard in Grafana and copy/paste the JSON, then histou didn't replace host and service names, because no "rows" array in simple template.

If I manually put my previously exported panel inside a "rows": [ {"panel": {...} } ] structure, then it works.

Maybe histou can be altered to work without a rows array, and check panels/tags in panels array (instead).

histou support for graphite

Hi,

I found nagflux and Histou really great and helpful, amazing job.
I wonder if you plan or if there is a way using Hisou (pop-ups) in Nagios when using Graphite/Graphios backend.

Many thanks,
David

Support for a host wide view

is the a way display all services per host, like the pnp url .../pnp4nagios/index.php/graph?host=

Renaming check_command leads to incomplete graphs

Every series in InfluxDB contains the check_command which was executed by the monitoring system.
Beneath hostname/servicedescription histou also uses check_command to fetch the data from the database, which is determined by the following query:

SELECT * FROM metrics WHERE host = '__hostname__' AND service = '__servicedesc__' 
GROUP BY performanceLabel LIMIT 1

Only one check_command is used to query the database. In case of a renamed check_command you won't get the series containing the other check_command anymore....

Proposal: if there are more than one check_comamnd, OR-combine them in the series query, e.g.

SELECT mean("value") AS "FastEthernet0/35_usage_in-value", 
...
...
AS "FastEthernet0/35_usage_in-crit-max" 
FROM "metrics" 
WHERE "host" = '__hostname__' 
AND "service" = 'snmp_interface_health'  
AND ("command" = 'check_nwc_health_local' OR "command" = 'check_nwc_health')  <<<<
AND "performanceLabel" = 'FastEthernet0/35_usage_in' 
AND time > 1497058120s and time < 1505828288s 
GROUP BY time(3h) fill(null);

generate bitmap images

I happily started using Histou (replacing pnp4nagios due to incompatibility to current PHP versions).
What I did before, was to have a simple "dashboard" (i.e. a HTML page) displayed by some raspi-driven monitor that showed the top 5 issues together with a performance graph of the last few days. This was easy with pnp4nagios, as pngs were generated by default, which the raspi could show without much need for computation.

Now, after having switched to Histou/Grafana for my naemon performance graphing needs (great stuff btw), I cannot do this anymore, as I need to include a large JS driven website as iframe for each of my 5 top alerts. The Raspi simply cannot render this in time for the next refresh (30s).

So, I am looking for a way to have some component generate svg/png/… instead.

Is there such functionality that I missed in my searches? Or are there plans in histou for this?

Thanks for any thoughts on this.

P.S.: thanks @Griesbacher for histou

[Question] Does this project work with Graphite & Grafana?

__autoload replaced with spl_autoload_register for PHP 8

I recently upgraded my server that Histou and Nagios is running on and it went from PHP 7.1 to PHP 8.1
After this I couldn't get histou to work properly so I found this in the logs:

PHP Fatal error: __autoload() is no longer supported, use spl_autoload_register() instead in /usr/local/nagios/share/histou/histou/bootstrap.php on line 17

The __autoload() function in PHP has been removed in PHP 8 and according to the log you should replace it with spl_autoload_register() instead.

After some reading in the PHP documentation I found this to fix it:

Remove this from /usr/local/nagios/share/histou/histou/bootstrap.php:

function __autoload($className) {
    $file = strtolower(str_replace('\\', DIRECTORY_SEPARATOR, $className)).'.php';
    if (file_exists($file)) {
        require_once $file;
    }
}

And replace it with this:

function my_autoloader($className) {
    $file = strtolower(str_replace('\\', DIRECTORY_SEPARATOR, $className)).'.php';
    if (file_exists($file)) {
        require_once $file;
    }
}
spl_autoload_register('my_autoloader');

I'm not really a developer otherwise I've would have felt comfortable making a PR for this.
If anyone else wants to do it, go ahead.

This solved my issues and I'm hoping it will save someone else some time troubleshooting.

Problem integration grafana error 200 doesn´t show graphical data

Hi, thanks in advance!

I have Nagios (core 4.4.6), Grafana(v8.2.5) Influxdb and Histou (0.4.3).
Histou show all data in json correctly, but whith the grafana integration (histou.js) shows an error in grafana dashboard:

{
  "gridPos": {
    "x": 0,
    "y": 0,
    "w": 24,
    "h": 8
  },
  "type": "text",
  "title": "Error Message below",
  "fill": 1,
  "content": "# HTTP code: 200\n# Message: load\n# Url: http://nagiosserver/histou/index.php?host=server1.es&service=&height=&legend=true&annotations=false\n# Probably the output is not valid json, because the returncode is 200!",
  "options": {
    "mode": "markdown",
    "content": "# Title\n\n  For markdown syntax help: [commonmark.org/help](https://commonmark.org/help/)\n           "
  },
  "id": 1,
  "pluginVersion": "8.2.5",
  "datasource": null
}

I can´t resolve this! is there any change using php7 in json data procces?

Thanks

Typo in histou.js

Hello,

please apply commit from ConSol-Monitoring/omd@a7f05fd.

With regards

Jonathan

Histou

HI,
I want to confirm where the histou package needs to be installed . Does It needs to be installed on grafana or nagios server ? I am running nagios and grafana on different servers.

Also var_url in /usr/share/grafana/public/dashboards/histou.js needs to be pointed at nagios or grafana ip??

use “fill(previous)” instead of “fill(null)” in histou application while querying influxdb timeseries database

Hi Philip Griesbacher,

I am referring to https://support.nagios.com/kb/article/nagios-core-performance-graphs-using-influxdb-nagflux-grafana-histou-802.html I am currently facing issue with the setup. For example, is there a way to run SQL query in InfluxDB for the time period between 17:20 till 18:00 where there is no data? For example http://externalIP:3000/dashboard/script/histou.js?orgId=1&host=dublin-dmz-non-dmz-cszk1.testmoconnect.io&service=Check%20Disk&refresh=30s&fullscreen&edit&panelId=1 I see the below issue.

Is there a way to use “fill(previous)” instead of “fill(null)”, which is the default one in histou application https://github.com/Griesbacher/histou as per the below screenshot?

Please let me know if you need any additional information and I look forward to hearing from you. Thanks in advance.

Best Regards,

Kaushal

Grafana + Histou dashboard stopped working

Information on the issue:

What Grafana version and what operating system are you using?
Grafana version 9.3.2
InfluxDB version 1.8.10
Nagflux version 1.0
Histou version 0.4.3

What happened?
I’ve been using a combination of InfluxDB + Nagflux + Grafana + Histou to display performance graphs in Nagios, but up until the end of Feb 2023 it stopped working.

I now get the error as seen as above when viewing the host using the url pattern: http://nagios_server:3000/dashboard/script/histou.js?host=$HOSTNAME$

I have confirmed that the hosts are not down, and that InfluxDB + Nagflux + Grafana + Histou are also running without issues aside from this one.

Did you follow any online instructions? If so, what is the URL?
https://nagiosenterprises.my.site.com/support/s/article/Performance-Graphs-Using-InfluxDB-Nagflux-Grafana-Histou-55707644#Nagflux_Config

I don't get the idea of templates

I have successfully installed naglux and histou on my machine. I have pretty graphs in my nagios/thruk that can be seen via grafana. But I have a feeling that I miss something. What is the idea of writing own templates, it's unclear to me how to write them and what for.

Is histou "only" a way to generate graphs on demand in grafana? Or maybe there is a way to browse thru all the graphs in some other way? Sorry for stupid question I have already spend a while to configure it and than trying to get the idea using the docker image that you provide. I have also read the prosentation that you make but looks like I'm missing something.

I'd be grateful for enlightenment.

Support for UOM of nagios plugins / Counter support

Hi,

I like to know if you plan adding UOM (Nagios plugins unit of measurement).
I use Histou to gather network I/O of docker containers and those values are continous counter of bytes. (like the auto generated template to detect that and show the right data)
if not, can you give me a hint where to start and I will try to add it and share( if I make it of course)

Thanks,
David

Drill down to Grafana as admin user

Hello, we setup Naemon+Nagflux+InfluxDB+Grafana using OMD. It's smooth and works great.
However, when we drill down to the performance graph, a new window going to Grafana UI with the graph is displayed. We note that the user logged in is omdadmin. This is security concern. Possible to disable drill down of the action url?

Besides, still figuring out where to specify the user (omdadmin) as no need to login the action_url.

Please help.
Thanks a lot.

are wildcards supported in rule:perfLabel array?

Hey,
I'm in the middle of testing the Histou implementation and started to play with templates.
I'm wondering if wildcards and other regex'es are supported in perdLabel when creating a rule?

I have tons of perfdata Labels like *_cpu and *_memory which I'd like to rule for. As soon as I set

$rule = new \histou\template\Rule(
    $host = '*',
    $service = '.*myservice$',
    $command = '*',
    $perfLabel = array('.*_cpu', '.*_memory')
);

I receive an error 500 in Grafana webfrontend. If I name the Labels explicitly, it works, but the list is quite long and changes frequently due to the nature of the processes monitored where CPU & memory is graphed.
Thanks for your help and keep up the work with this amazing implementation.

Edit:
It certainly does since check_nwc_health-interface.php contains regexp. I still cant get is running even though I'm using the syntax of the nwc_health example like this:

$perfLabel = array('^.*?_(cpu|memory)$')
Btw, my perfdata in Nagios for one service check looks like this:

env1_process1_cpu=0%;80;90;; 
env1_process1_memory=157MB;4000;5000;0;128963 
env1_process2_cpu=0%;80;90;; 
env1_process2_memory=19MB;4000;5000;0;128963 
env1_process3_cpu=0%;80;90;;
env1_process3_memory=9MB;4000;5000;0;128963 
....
env2_process1_cpu=0%;80;90;; 
env2_process1_memory=157MB;4000;5000;0;128963
env2_process2_cpu=0%;80;90;; 
env2_process2_memory=19MB;4000;5000;0;128963 
env2_process3_cpu=0%;80;90;; 
env2_process3_memory=9MB;4000;5000;0;128963 
....

Histou installation

Hello,

I install histou along with nagflux but I have troubles to get it working and I might miss something.
I have nagios complied and installed under /usr/local/nagios. histou installed under
/usr/local/nagios/share/histou and I copied histou.js under grafana /usr/share/grafana/public/dashboards/histou.js. How can I get Histou working meaning showing me perf data like PNP4nagios?
grafana URL for my installation is: http://{server-ip}:3000 and nagios is under http://{server-ip}
I'm new to grafana/influxdb.

Doess someone knows how to configure that?

Thanks,
David

PopUp graphs in Nagios with Histou

We are looking into too replace PNP4Nagios with Nagflux+InfluxDB+Grafana+Histou.
Everything is looking good so far, the DB is getting updated with performancedata and we can see the data when we click on the Graph symbol in Nagios/Thruk.
What we are missing and can not get to work is the PopUp's when we hover with the mousepointer over the Graph symbol like with PNP4Nagios.
Do you have a suggestion how to get this functionality to work?

--
Dan

Empty graphs or no data points

Hi,

I have installed the Histou latest along with Grafana 4.1.1-1484211277 and Influxdb 1.1.1-1, on Ubuntu 14.04.
In the generated graphs/templates I see no data, but when I edit the graph and go to "Display"->"Stacking & Null value", and change the "Null value" from "null" to "connected", I do see the data.
What can I do/change in order to set that as a default behavior of Histou or Grafana? I tried to look for it a little on the web and in the code, but could not find it.

Thanks,
David

Grafana image rendering function not working

Hello,

we are using the OMD version and noticed, that the "Direct link rendered image" function in Grafana is not working for the histou template. This function can be accessed from the "Share Panel" for a specific graph. Grafana generates a link like this:

https://omd-servername/omdsite/grafana/render/dashboard-solo/script/histou.js?orgId=1&host=hostname&service=servicename&annotations=true&refresh=30s&from=1607750379265&to=1607779179265&panelId=1&width=1000&height=500&tz=Europe%2FBerlin

After a few seconds, the webserver reports this error:

---

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /omdsite/grafana/render/dashboard-solo/script/histou.js.

Reason: Error reading from remote server

For manually created graphs in Grafana it is working.

Thank you in advance and best regards