uberspace / icingaweb2-module-netboximport

Import devices and virtual machines from netbox into icinga2 to monitor them.

Icinga Web 2 Netbox Import

Import devices and virtual machines from netbox into icinga2 to monitor them.

Installation

$ cd /usr/share/icingaweb2/modules
$ git clone https://github.com/Uberspace/icingaweb2-module-netboximport.git netboximport
$ icingacli module enable netboximport

Configuration

All configuration is done in the web interface under the "Automation" tab of icinga2 director. Please read to the official documentation before configuring a netbox import.

1. add an "Import Source"

• Key column name: name (the hostname)
• fill out all other required files according to the tooltips shown

2. test the Import source via the "Check for changes" button, "Preview" tab and finally "Trigger Import Run"
3. add a "Sync Rule"

• Object Type: "Host"
• by default will import all objects present in netbox. You can tailor this by setting "Filter". For example, only import objects, which have a certain field set: custom_fields__icinga2_host_template__label>0.

4. add the desired Properties to the rule

• setting object_name, address and address6 to name is generally desireable

5. test the Sync Rule via the "Check for changes" and finally "Trigger this Sync" buttons.
6. add an import job to run the import regularly
7. add an sync job to run the sync regularly

Data Format

This plugin pulls all available objects with all their fields into icinga. Since the data in netbox mostly consists of nested objects, you will either have to autoflatten (default for custom_fields and interfaces) the elements or you have to apply the "Get specific Array Element" modifier to deconstruct the objects. and access the object elements:

primary_ip4:

{
  address: "192.168.0.1/24",
  family: 4,
  id: 1,
  url: "<NETBOX URL>"
}

Apply modifier to export address field:

Because Icinga2 does not like the format Netbox uses (/subnet appended to address), we need to strip this away with another modifier:

Now you can access the primary_ipv4 directly as a property :arrow_right:

ipv4_address: "192.168.0.1"

A list of all possible fields can be seen in the "Preview" of your Import Source, in your Sync Rule while adding a new property or in your API itself: https://netbox.example.com/api/dcim/devices/, https://netbox.example.com/api/virtualization/virtual-machines/.

In some cases additional fields are provided:

• cluster is replaced by the actual cluster object as returned by the API, instead of just the id/name.
• interfaces is added, so configured IP addresses can be reused in icinga
• services is added, so configured services can be reused in icinga

Acknowledgements

The general structure and a few tips were lifted from icingaweb2-module-fileshipper. Thanks!

icingaweb2-module-netboximport's Issues

FR: Add option to add filters to NetBox requests

Collection all the information in our current NetBox environment takes too long and runs into a timeout.

As Icinga might need different attributes from different devices I would propose to add a filter option to this module.

Appending a ?role=Server (as an example) to the request URL would be great. This limits the amount of returned data and we can change the imported data based on the device type.

A text field should be sufficient. The content should be added to the URL upon request.

Just putting this here as a reminder. Currently on Holidays but will try to give it a shot myself once I'm back. From what I've seen it should not be too difficult to implement.

Requesting unknown URL

For GET /api/virtualization/clusters/ HTTP/1.1 the result is

{ "count": 1, "next": null, "previous": null, "results": [ { "id": 4, "name": "Proxmox", "type": { "id": 7, "url": "http://x-aachen.de/api/virtualization/cluster-types/7/", "name": "Proxmox", "slug": "proxmox" }, "group": null, "site": { "id": 1, "url": "http://x-aachen.de/api/dcim/sites/1/", "name": "x-aachen.de", "slug": "kullen" }, "comments": "", "tags": [], "custom_fields": {}, "created": "2017-12-21", "last_updated": "2017-12-21T15:12:49.124749+01:00" } ]

See https://pastebin.com/nTdUsQeJ for correct layout.

and the Script requests

GET /api/virtualization/clusters/4?limit=1000000 HTTP/1.1

gets redirected (301) to (see #3)

GET /api/virtualization/clusters/4/?limit=1000000 HTTP/1.1

This request results in a 404 because the page doesn't exist

Missing License?

Hey guys,
we love this tool, and its super handy. We have cloned the repo privately and added some features and fixes we needed. Happy to contribute or fork or whatever, can you please let us know what license you want to use? Thanks for your work in creating it, it has really helped with a few projects.

Allow non unique object names

NetBox allows to assign the same name for different objects as long as there is a secondary unique key.

Descussion on this issue: netbox-community/netbox#2669

Flattening causes Import failure because of Context Data

Expected Behavior

Import should work.

Current Behavior

If you try to import data from Netbox 2.4.7 it doesn't allow you to import because of the long names produced by the context_data in netbox like local_context_data__last_inform__port_table__3__mac_table__0__uptime.
The Director team doesn't want to increase the limit of the names for performance reasons and also because they think this module shouldn't flatten the data at all because the director supports structured data.

Possible Solution

A. Don't flatten the data
B. Dont expose the context_data to Icinga

I did try to disable the flattening in the module but this resulted in probably invalid JSON (see Icinga/icingaweb2-module-director#1711 (comment))

Steps to Reproduce (for bugs)

See Icinga/icingaweb2-module-director#1711

Your Environment

• Icinga Web 2 Netboximport version (System - About): 1.0.0
• Director version (System - About): master (e895978ff243c007a2a50a1c34ea54bd5ef59411)
• Icinga Web 2 version and modules (System - About): 2.6.1, icingaweb2-module-netboximport
• Netbox version (System - About): 2.4.7
• Icinga 2 version (icinga2 --version): r2.9.2-1
• Operating System and version: Ubuntu 16.04
• Webserver, PHP versions: Apache (behind nginx reverse), PHP7.2

Nautobot also supported?

Is this icingaweb2 module also compatible to the Nautobot API?
https://github.com/nautobot/nautobot

current version (dfd907928c309441b0307bf83bbc211b843d2b20) ends in a stacktrace

Trying to use this plugin with different versions of NetBox (2.8.3 and 2.9.3). Unfortunately it always ends in the same stack trace.

#0 /usr/share/icingaweb2/modules/netboximport/library/Netboximport/ProvidedHook/Director/ImportSource.php(138): Icinga\Application\ApplicationBootstrap->Icinga\Application\{closure}(Integer, String, String, Integer, Array)
#1 /usr/share/icingaweb2/modules/netboximport/library/Netboximport/ProvidedHook/Director/ImportSource.php(236): Icinga\Module\Netboximport\ProvidedHook\Director\ImportSource->fetchInterfaces()
#2 /usr/share/icingaweb2/modules/director/library/Director/Web/Table/ImportsourceHookTable.php(100): Icinga\Module\Netboximport\ProvidedHook\Director\ImportSource->fetchData()
#3 /usr/share/icingaweb2/modules/incubator/vendor/gipfl/icingaweb2/src/Table/SimpleQueryBasedTable.php(29): Icinga\Module\Director\Web\Table\ImportsourceHookTable->prepareQuery()
#4 /usr/share/icingaweb2/modules/incubator/vendor/gipfl/icingaweb2/src/Table/SimpleQueryBasedTable.php(15): gipfl\IcingaWeb2\Table\SimpleQueryBasedTable->getQuery()
#5 /usr/share/icingaweb2/modules/incubator/vendor/gipfl/icingaweb2/src/Table/QueryBasedTable.php(56): gipfl\IcingaWeb2\Table\SimpleQueryBasedTable->getPaginationAdapter()
#6 /usr/share/icingaweb2/modules/incubator/vendor/gipfl/icingaweb2/src/Table/QueryBasedTable.php(255): gipfl\IcingaWeb2\Table\QueryBasedTable->getPaginator(Object(gipfl\IcingaWeb2\Url))
#7 /usr/share/icingaweb2/modules/director/application/controllers/ImportsourceController.php(159): gipfl\IcingaWeb2\Table\QueryBasedTable->renderTo(Object(Icinga\Module\Director\Controllers\ImportsourceController))
#8 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\Module\Director\Controllers\ImportsourceController->previewAction()
#9 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch(String)
#10 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#11 /usr/share/php/Icinga/Application/Web.php(300): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#12 /usr/share/php/Icinga/Application/webrouter.php(99): Icinga\Application\Web->dispatch()
#13 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#14 {main}

Current settings:

Expected Behavior

Fetch the NetBox data successfully

Current Behavior

Fetching Data fails

Possible Solution

Steps to Reproduce (for bugs)

Can't really tell. Tried with different commits in this repo and the issue does not disappear.

Context

Your Environment

• Icinga Web 2 Netboximport version (System - About): dfd9079 (current master)
• Icinga Web 2 Director version (System - About): dfd9079 (current master)
• Icinga Web 2 version and modules (System - About): 2.8.2
• Netbox version (System - About): 2.8.3 and 2.9.3
• Version used (icinga2 --version): 2.12.0-1
• Operating System and version: CentOS 7.8.2003
• Enabled features (icinga2 feature list):
• Config validation (icinga2 daemon -C):

Thank You

add github issue template

Add an issue template to this repo asking for

• commit/version of importer
• icinga2 version
• netbox version

Invalid argument supplied for foreach()

Cannot run an import from NetBox to Icingaweb2:

#0 /usr/share/icingaweb2/modules/netboximport/library/Netboximport/ProvidedHook/Director/ImportSource.php(93): Icinga\Application\ApplicationBootstrap->Icinga\Application\{closure}(Integer, String, String, Integer, Array)
#1 /usr/share/icingaweb2/modules/netboximport/library/Netboximport/ProvidedHook/Director/ImportSource.php(167): Icinga\Module\Netboximport\ProvidedHook\Director\ImportSource->fetchInterfaces()
#2 /usr/share/icingaweb2/modules/director/library/Director/Web/Table/ImportsourceHookTable.php(109): Icinga\Module\Netboximport\ProvidedHook\Director\ImportSource->fetchData()
#3 /usr/share/icingaweb2/modules/director/library/vendor/ipl/Web/Table/SimpleQueryBasedTable.php(29): Icinga\Module\Director\Web\Table\ImportsourceHookTable->prepareQuery()
#4 /usr/share/icingaweb2/modules/director/library/vendor/ipl/Web/Table/SimpleQueryBasedTable.php(15): dipl\Web\Table\SimpleQueryBasedTable->getQuery()
#5 /usr/share/icingaweb2/modules/director/library/vendor/ipl/Web/Table/QueryBasedTable.php(56): dipl\Web\Table\SimpleQueryBasedTable->getPaginationAdapter()
#6 /usr/share/icingaweb2/modules/director/library/vendor/ipl/Web/Table/QueryBasedTable.php(237): dipl\Web\Table\QueryBasedTable->getPaginator(Object(dipl\Web\Url))
#7 /usr/share/icingaweb2/modules/director/application/controllers/ImportsourceController.php(74): dipl\Web\Table\QueryBasedTable->renderTo(Object(Icinga\Module\Director\Controllers\ImportsourceController))
#8 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\Module\Director\Controllers\ImportsourceController->previewAction()
#9 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch(String)
#10 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#11 /usr/share/php/Icinga/Application/Web.php(409): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#12 /usr/share/php/Icinga/Application/webrouter.php(104): Icinga\Application\Web->dispatch()
#13 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#14 {main}

Settings: