Coder Social home page Coder Social logo

icinga / icingaweb2-module-fileshipper Goto Github PK

View Code? Open in Web Editor NEW
28.0 13.0 13.0 277 KB

Provide CSV, JSON, XML and YAML files as an Import Source for the Icinga Director and optionally ship hand-crafted additional Icinga2 config files

Home Page: https://icinga.com/docs/director/latest/fileshipper/doc/02-Installation/

License: GNU General Public License v2.0

PHP 100.00%
icinga-director module csv json xml yaml configuration icinga2

icingaweb2-module-fileshipper's Introduction

Icinga Web 2 Fileshipper module

The main purpose of this module is to extend Icinga Director using some of it's exported hooks. Based on them it offers an Import Source able to deal with CSV, JSON, YAML and XML files. It also offers the possibility to deploy hand-crafted Icinga 2 config files through the Icinga Director.

Icinga Web 2 Fileshipper

For getting started please read our Installation instructions, and then you should be ready to dive into Import Source definitions, supported file formats or and hand-crafted Config File Shipping.

Changes

v1.2.0

  • FEATURE: PHP 8 support
  • FEATURE: Give guidance on potential misconfiguration (#34)
  • FEATURE: do not fail on malformed config file (#35)

v1.1.0

  • FEATURE: Added XLSX file support

v1.0.1

  • FEATURE: CSV files should give NULL for columns with empty strings (#6)
  • FIX: Small documentation fix

icingaweb2-module-fileshipper's People

Contributors

hboetes avatar hrak avatar martialblog avatar rgevaert avatar thomas-gelf avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

icingaweb2-module-fileshipper's Issues

CSV Import use Contains in Assign_filter

Hi,

is it possible to use the contains string while importing Services from csv via the assign_filter attribute?
My CSV:
'apply_name','service_name','assign_where'
'port-channel1','Service-NWC-Interface-Health-extra-Port-Channel','host.vars.monitoring_interfaces contains port-channel1'
I tried "==" as well as "in", but it throws errors.

The normal assignment is like:
assign where "port-channel1" in host.vars.monitoring_interfaces

Best regards

When imports.ini is misformed the director form fails

Expected Behavior

It should gracefully display an error inside the form, alarming the user the config file should be corrected.

Current Behavior

Error in UI:

array_map(): Expected parameter 2 to be an array, string given

Steps to Reproduce (for bugs)

Broken /etc/icingaweb2/modules/fileshipper/imports.ini:

xamples]
basedir = "/opt/import"

Go to Director any create or modify any fileshipper import.

Your Environment

  • Module version (System - About): 1.7.2
  • Icinga Web 2 version and modules (System - About): 2.8.2
  • Icinga 2 version (icinga2 --version): 2.12.1
  • Operating System and version: CentOS 7
  • Webserver, PHP versions: PHP 7.3

Huge database tables

Hi,
we have fileshipper running for an automated Netbox -> Icinga2 export for a multitude of objects like switches, switch interfaces, PDUs, Firewalls and stuff.

Importing the files took longer and longer which is astonishing as the amount of objects didnt change. Today i had a look at the database tables and i was baffled:

i2director=# select count(*) from imported_row;
  count  
---------
 1179641
(1 row)

i2director=# select count(*) from imported_row_property;
  count  
---------
 6823277
(1 row)

I import around 150-180 devices with a total of 1000 objects, so my impression is that there is no cleanup/expire of old objects happening.

Provide info if each imported file contains one or multiple objects

One can e.g. configure the following import source within Director (Icingaweb2):


Import source name: Hosts
Source Type: Import from files (fileshipper)
Key column name: Hostname
File format: CSV
Field delimiter: ;
Value enclosure: "
Base directory: /srv/some_dir
File name: * (all files) 

In this case, Fileshipper is used to import all files within the provided directory. Thomas writes on icinga-users:

"All files" has been implemented for scenarios where you
have one file per object, like one YAML or JSON file per single host.
This is for systems using a directory as their "inventory".

But what you can't provide in the settings is if a file contains one or multiple objects. It would be great to have the functionality that Fileshipper is able to import multiple objects per file, not only one.

Step-wise Import Source preview

It would be nice to be able to have more customization options in the Import Source preview. For example selectively enabling only parts of the configured Import Source modifiers would sometimes help beginners to better understand what's going on.

Unable to add a new host group - Error

Hi,

when I try to add a new host group with icinga2-director, following error occurs:
(I have installed the latest version from github (master).)

#0 /usr/share/icingaweb2/modules/director/library/Director/Objects/IcingaObject.php(1133): Icinga\Module\Director\Objects\IcingaObject->assertCustomVarsSupport()
#1 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/IcingaObjectFieldLoader.php(51): Icinga\Module\Director\Objects\IcingaObject->vars()
#2 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/DirectorObjectForm.php(263): Icinga\Module\Director\Web\Form\IcingaObjectFieldLoader->setValues(Array, 'var_')
#3 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/DirectorObjectForm.php(566): Icinga\Module\Director\Web\Form\DirectorObjectForm->handleCustomVars(Object(Icinga\Module\Director\Objects\IcingaHostGroup), Array)
#4 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/QuickForm.php(430): Icinga\Module\Director\Web\Form\DirectorObjectForm->onRequest()
#5 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/QuickForm.php(437): Icinga\Module\Director\Web\Form\QuickForm->setRequest(Object(Icinga\Web\Request))
#6 /usr/share/icingaweb2/modules/director/library/Director/Web/Form/QuickForm.php(304): Icinga\Module\Director\Web\Form\QuickForm->getRequest()
#7 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(205): Icinga\Module\Director\Web\Form\QuickForm->handleRequest()
#8 /usr/share/php/Zend/Controller/Action.php(516): Icinga\Module\Director\Web\Controller\ObjectController->addAction()
#9 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch('addAction')
#10 /usr/share/php/Zend/Controller/Front.php(954): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#11 /usr/share/php/Icinga/Application/Web.php(384): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#12 /usr/share/php/Icinga/Application/webrouter.php(109): Icinga\Application\Web->dispatch()
#13 /usr/share/icingaweb2/public/index.php(4): require_once('/usr/share/php/...')
#14 {main}

For any other information about my setup, please tell me what information I have to give to you!

Thank you!
Klaus.

Autosubmission doesn't always take place as expected

Expected Behavior

Related fields should immediately be shown when changing related drop-down elements.

Current Behavior

When not filling values in top-down order it might happen, that some fields only show up on submission. This is not critical, as required fields will always block submissions - but it is irritating.

BaseDir is empty

Ive added the git to my icinga2 but when i try to create a sync. the base dir is empty.

the Folder is created and the Import.ini is also filled with the correct path and inside the path there is an *.csv file.

but im unable to choose the basedir.

MODULE VERSION STATE DESCRIPTION
audit 1.0.0 enabled Audit Module
businessprocess 2.1.0 enabled A Business Process viewer and modeler
director master enabled Director - Config tool for Icinga 2
fileshipper 1.0.1 enabled Fileshipper for Icinga Director
graphite 0.0.0.5 enabled Graphite graphs for perfdata
monitoring 2.5.3 enabled Icinga monitoring module
theme-dark 0.0.1 enabled A dark theme for Icingaweb2
vsphere 1.1.0 enabled VMware vSphere Import Source

Feature request: upload files in web UI

It would, I think, create an (even) better user experience if files could be uploaded directly in the web interface. I.e. provide an upload button when fileshipper is chosen as import source.

cannot import/ sync host templates

I don't know if its by design, but cannot seem to be able to sync object_type template. I'm getting errors

This Sync Rule failed when last checked at 2016-11-30 14:44:52: Exception while syncing Icinga\Module\Director\Objects\IcingaHost zone01_template: Sync is not allowed to modify template "zone01_template"

If i remove the sync properties for "object_type", I can create from the file.

Merged XSLX cells should return the same value for every cell

Expected Behavior

When merging cells over multiple rows and/or columns, I expect to see the same value in every column

Current Behavior

The value is returned where stored, this is the top/leftmost cell. All other merged cells return a NULL value.

Filename as key column name

Is it somehow possible to set the filename as key column name when creating a new import source?
screenshot from 2019-01-21 16-38-42

Thanks for any ideas.

Comments starting with '#' are deprecated in ...

I receive the following messages when I open in Director the Import Source / Modify Menu which uses the Fileshipper module (fd2c797) - though it also happens with older versions:

Deprecated: Comments starting with '#' are deprecated in Unknown on line 1 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265

Deprecated: Comments starting with '#' are deprecated in Unknown on line 2 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265

Deprecated: Comments starting with '#' are deprecated in Unknown on line 3 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265

Deprecated: Comments starting with '#' are deprecated in Unknown on line 4 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265

Deprecated: Comments starting with '#' are deprecated in Unknown on line 5 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265

Deprecated: Comments starting with '#' are deprecated in Unknown on line 6 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265

Deprecated: Comments starting with '#' are deprecated in Unknown on line 7 in /usr/share/php/Icinga/File/Ini/IniParser.php on line 265

Context

The problem is only annoying and not serious, or at least for me the module seems to work fine, it just presents these messages, it would be fine to get rid of them.
I am also not 100% percent sure, that this is related to the fileshipper module, but since it just pops up when I use it, I report it here.

Your Environment

  • icingaweb2-2.5.1-1.el7.icinga.noarch
  • icinga2-2.8.4-1.el7.icinga.x86_64
  • Director Version (fc58e50)
  • RHEL 7.5
  • PHP 5.4.16 (/bin/php) and PHP 7.1.8 (/opt/rh/rh-php71/root/usr/bin/php)

JSON Import: Command: Unable to set "plugin-check-command"

I am trying to import a command via JSON. However, there is no possibility to define the command type (plugin-check-comand) in the properties. There is only the possibility to define an import, but you can't set "plugin-check-command" there either, because otherwise errors occur.
If you look at the imported command, the value for Command Type is not defined, but it can still be rendered. Saving the imported command in the director is not possible, because this is a forced field.

It should be noted that the entire documentation for Fileshipper is more than poor. It could be documented, for example, what format the individual properties must have. So it's just guessing and dull trying until it might work sometime.

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

[
    {
        "arguments": {
            "-H": {
                "description": "fs_http_host_argument",
                "value": "$fs_http_host$"
            }
        },
        "command": "fs_http",
        "object_name": "fs_http"
    }
]

source
sync_modify
sync_properties
command

Context

Your Environment

  • Module version (System - About): 1.2.0
  • Icinga Web 2 version and modules (System - About): 2.8.2
  • Icinga 2 version (icinga2 --version): r2.12.3
  • Operating System and version: Debian 10
  • Webserver, PHP versions: Apache 2, PHP 7.3.27

Import unique id != hostname

Hi,
i am trying to import fileshipper jsons containing host information and i am trying to use something different as a unique id than the hostname.

The reason is that i'd like to be able to track host renames from the external source by following the unique id. If we cant track renames the host would be deleted and recreated with the new name.

I fail to understand the possibilities of the unique id in the import rule. My impression was that the unique id in the import rule is "only" used for lifetime tracking e.g. if the id disappears the object is gone and should be deleted.

In the sync source i set the object_name from the real name from the import and store the "id" in a custom field in icinga director.

if i now click trigger sync twice i get a nice backtrace because the object already exists.

What i am missing or misunderstand?

Flo

Feature request: use filename pattern

Hi

In some cases it could be helpful, if we could define a file name pattern, rather to choose a specific file, and either process all file where the pattern matches or take a single file according some predefined attribute: e.g. the newest/oldest/bigest/smallest/etc. out of the resulting list automatically.

In case we need to process a file from a directory, where there are multiple files with the same filename + some pattern e.g. myfile_{date}.csv, and we want to have the most current file.

Question: creating time-objects with fileshipper

In our environment we have DB's with different Backup-times. I wanted to create time objects for them via fileshipper, to apply them later with a sync rule. In the sync rules I’m able to connect the imported data with the field time-ranges. So I tried different notations in my csv-sheet because I always thought that the timeranges in icinga2 are some kind of array.

Problem

So instead of getting a timerange that looked like this:

object TimePeriod "TP_5x9" {
    import "legacy-timeperiod"
    display_name = "TP_5x9"
    ranges = {
        "friday"	= "08:00-17:00"
        "monday"	= "08:00-17:00"
        "thursday"	= "08:00-17:00"
        "tuesday"	= "08:00-17:00"
        "wednesday"	= "08:00-17:00"
    }
}

I got something like this:

object TimePeriod "tosnowaok" { 
   import "legacy-timeperiod"
   display_name = "testdb1" 

    ranges = {
               "0"   = "\"00:00-06:10;08:20-24: 00\""
     }
}

Here is an example what my test-csv looked like:

db-instanz;timeperiod
!testdb1!;!"00:00-06:10;08:20-24:00"!
!testdb2!;!"friday" = "00:00-06:10,08:20-24:00";"monday" = "00:00-06:10,08:20-24:00"!
!testdb3!;!"[ "friday" = "00:00-06:10,08:20-24:00", "monday" = "00:00-06:10,08:20-24:00" ]"!

I tried a few things to cover different cases how the fileshipper sets the time ranges and if there is any way to link the days to the times.

Can someone tell me how to do this?

I didn’t find any documentation about this special topic…

Your Environment

  • Module version (fileshipper ): 1.1.0
  • Icinga Web 2 version and modules (System - About): 2.6.1 (director 1.5.2)
  • Icinga 2 version : 2.10.1-1

Provide an Import Source for XSLX files

Expected Behavior

When I drop an MS Excel file in a specific folder, it's content should be mapped and deployed through the Icinga Director.

Current Behavior

Fileshipper doesn't know about XSLX files at all.

Possible Solution

Sit down and write some code.

File name * (all files) results in No key column "Key_column_name" in row 1

Choosing File name: * (all files) always results in a No key column "Key_column_name" in row 1 error.

Expected Behavior

All files should correctly be processed

Current Behavior

Error occurs even when there is a 1 valid file in the directory. I've duplicated with CSV and YAML.

image

Possible Solution

Steps to Reproduce (for bugs)

$ cat /etc/icingaweb2/modules/fileshipper/imports.ini
[csv]
basedir = /var/cache/icinga2/import/csv
$ ls -la /var/cache/icinga2/import/csv
total 4
drwxr-sr-x 2 root   icingaweb2 21 Oct  5 15:07 .
drwxr-sr-x 7 icinga icingaweb2 63 Oct  5 14:58 ..
-rw-r--r-- 1 root   icingaweb2 13 Oct  5 15:07 list.csv
$ cat -A /var/cache/icinga2/import/csv/list.csv
host$
my_host$

image

Using the same settings but selecting the single file instead of * (all files) works as expected.

Context

Trying to do an import of multiple files.

Your Environment

  • Module version (System - About): master (f5fff02)
  • Icinga Web 2 version and modules (System - About): 2.4.2 (7cec28a31fdff0feb470ec001838bc1dec9c4b69)
  • Icinga 2 version (icinga2 --version):
icinga2 - The Icinga 2 network monitoring daemon (version: r2.7.1-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-327.10.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: unknown
  • Webserver, PHP versions:
$ php --version
PHP 5.4.16 (cli) (built: Nov  6 2016 00:29:02)
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies
$ nginx -V
nginx version: nginx/1.12.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'
$ php-fpm --version
PHP 5.4.16 (fpm-fcgi) (built: Nov  6 2016 00:30:57)
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies

Multiple values in CSV import

How to achieve importing host to multiple groups (Group membership)?

Example1:
id, Host, Group1, Group2, Group3
1,"Alpha","g1","g2","g3"
+assignments:
${Group1} -> groups
${Group2} -> groups
${Group3} -> groups
This works for import (g1, g2, g3 detected as different groups and fields, but Sync job performs only host assignment to latest Group3 [g3]) .

Example2:
id, Host, Group,
1,"Alpha","g1 g2 g3"
+assignment: ${Group} -> groups
This works for import (g1, g2, g3 detected as different groups, but Sync job performs only host assignment to g3).

Space is valid csv multi-value field separator, but isn't valid for Sync job.

Allow Import from external URL

Expected Behavior

Import from an external url like http://example.com/hosts.json

Current Behavior

You can only import from the local filesystem

Context

We have a System where all Devices are already configured and that way we could directly import them from there

Memory size is exhausted while changes are deployed

When trying to import and sync a simple host file (just containing the host block and ip-address) the following error is shown, when deploying the changes:

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /usr/share/icingaweb2/modules/director/library/Director/Objects/IcingaObjectImports.php on line 221

It is not possible to deploy the changes from activity log nor revoke them. They result in the need of restoring the VM, since re-deploying an older configuration does not reset the changes waiting for deployment.

If logs or anything else is needed, just tell me what you need :)

PS: Not sure if this is fileshipper or director related, feel free to send me over to the other repo ^^

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.