genebean / genebean-nxlog

A Puppet module to manage NXLog

nxlog

Table of Contents

1. Overview
2. Module Description - What the module does and why it is useful
3. Setup - The basics of getting started with nxlog
   • What nxlog affects
   • Setup requirements
   • Beginning with nxlog
4. Usage - Configuration options and additional functionality
5. Limitations - OS compatibility, etc.
6. Development - Guide for contributing to the module

Overview

This module manages NXLog. It is primarily targeted at Windows but should work on Linux too so long as the package is available via some repository. Automated testing is done via Travis CI and tests against both Puppet 3.x and 4.x.

Module Description

This module installs the nxlog package, builds a config file using puppetlabs-concat, and manages the nxlog service. Since the config file for NXLog is broken up into sections, and most of these can be repeated, the choice was made to utilize defined types similar to the way the Apache module does with virtual hosts.

Setup

What nxlog affects

• nxlog package
• nxlog config file
• nxlog service

Beginning with nxlog

This is an example of how to build a full config file:

class {'nxlog':
  conf_dir       => 'C:/Program Files (x86)/nxlog/conf', # note the /'s here
  conf_file      => 'nxlog.conf',
  ensure_setting => latest,
  nxlog_root     => 'C:\Program Files (x86)\nxlog',
}

nxlog::config::extension { 'json':
  ext_module => 'xm_json',
}

nxlog::config::input { 'eventlog_json':
  input_execs  => [
    'delete($Keywords)',
    '$raw_event = to_json()',
  ],
  input_module => 'im_msvistalog',
}

nxlog::config::output { 'local_json':
  output_file_path => 'C:\eventlog-json.txt',
  output_module    => 'om_file',
}

nxlog::config::output { 'logserver':
  output_address   => 'logserver.example.com',
  output_module    => 'om_udp',
  output_port      => '6371',
}

nxlog::config::route { 'local':
  route_destination => [ 'local_json', ],
  route_source      => [ 'eventlog_json', ],
}

nxlog::config::route { 'logserver':
  route_destination => [ 'logserver', ],
  route_source      => [ 'eventlog_json', ],
}

Usage

Class nxlog

These settings are used both in the config file and by the other parts of the module.

conf_dir - the directory where nxlog.conf resides

conf_file - the name of the config file

ensure_setting - this is passed to the package resource

nxlog_root - the installation directory for the nxlog program. On Windows this is a required setting as NXLog will not start otherwise.

Defined Types

Each of these builds a section of the config file.

nxlog::config::extension - builds an Extension section using the specified name.

• ext_module - the name of the extension module to use
• ext_options - an array of options for the added ext_module. Each item in the array will be an line in this section of the config file.

nxlog::config::input - builds an Input section using the specified name.

• input_execs - an array of Exec statements to include (Optional)
• input_file_path - defines the path to use if reading from a local file
• input_module - the name of the input module to use
• input_type - the name of the registered input reader function to use

nxlog::config::output - builds an Output section using the specified name.

• output_address - the address of the remote host to send data to
• output_execs - an array of Exec statements to include (Optional)
• output_file_path - defines the path to use if writing to a local file
• output_module - the name of the output module to use
• output_port - the port on the remote host to send data to

nxlog::config::processor - builds a Processor section using the specified name.

• processor_module - the name of the processor module to use
• porcessor_input_format - the format of the data being converted or processed
• processor_output_format - the format to convert the data to
• processor_csv_output_fields - fields which are placed in the CSV lines. The field names must have the dollar sign "$" prepended.

nxlog::config::route - builds a Route section using the specified name.

• route_destination - an array of outputs to send data to
• route_source - an array of inputs to send to the named destination

Limitations

On Windows it is assumed that you are using Chocolatey to install packages.

On Linux it is assumed that you have a custom repo which contains nxlog-ce. You can work around this by installing the program separately and setting ensure_setting => present.

nxlog::config::processor chooses a template based on the value of processor_module. Not all of the possible modules have had templates generated for them yet... if you want one that does not yet feel free to file a bug report or, better yet, send a pull request.

Development

Pull requests are welcome! A Vagrantfile is included in this module to aide in testing and development. All code must have tests before it will be merged but I am happy to help with that part.

Contributors

• Jaime Viloria (@cerealcake) - Added support for specifying output options.
• Jaime Viloria (@cerealcake) - Added support for specifying options for extension modules.
• @egouraud - Added support for specifying the input file used by some input modules.
• Camilo Cota (@camilocot) - Added the input_type setting

genebean-nxlog's Issues

Use github_changelog_generator

https://github.com/github-changelog-generator/github-changelog-generator#migrating-from-a-manual-changelog

Document with puppet-strings

_{Sent with GitHawk}

new params inside extension module

hi, thanks for this wonderful nxlog repo. It's really helpful. I haven't found the suggest features link here so I put inside the Issues tab.

Is it possible to add exec and schedule params in extension module? I'm trying to accomplish a log rotation like the below link. Thanks!

https://nxlog.co/documentation/nxlog-user-guide/log-rotation.html

Dependency issues

Hi there,

this module requires chocolatey/chocolatey which in turn requires the now deprecated badgerious/windows_env is it possible to update the dependencies to use puppetlabs/chocolatey instead? It should be an in place upgrade.

'No such file or directory' error

Hi,

I am getting this error with the default suggested config (please see full error below).
puppet server version: 6.15
Puppet agent: 7 (I also tried to run with lower versions with the same result).

Any ideas?

---

puppet : �[1;31mError: Could not set 'file' on ensure: No such file or directory - A directory component in
C:/Program Files (x86)/nxlog/conf/nxlog.conf20210508-4684-bdgpzl.lock does not exist or is a dangling symbolic
link�[0m
At line:1 char:7

• puppet agent -t

•   ~~~~~~~~~~~~~~~
  

  • CategoryInfo : NotSpecified: (�[1;31mError: C...mbolic link�[0m:String) [], RemoteException
  • FullyQualifiedErrorId : NativeCommandError

�[1;31mError: Could not set 'file' on ensure: No such file or directory - A directory component in C:/Program
Files (x86)/nxlog/conf/nxlog.conf20210508-4684-bdgpzl.lock does not exist or is a dangling symbolic link
Wrapped exception:
No such file or directory - A directory component in C:/Program Files
(x86)/nxlog/conf/nxlog.conf20210508-4684-bdgpzl.lock does not exist or is a dangling symbolic link�[0m
�[1;31mError: /Stage[main]/Nxlog::Config/Concat[C:/Program Files (x86)/nxlog/conf/nxlog.conf]/File[C:/Program
Files (x86)/nxlog/conf/nxlog.conf]/ensure: change from 'absent' to 'file' failed: Could not set 'file' on ensure:
No such file or directory - A directory component in C:/Program Files
(x86)/nxlog/conf/nxlog.conf20210508-4684-bdgpzl.lock does not exist or is a dangling symbolic link
(corrective)�[0m
�[mNotice: /Stage[main]/Nxlog::Service/Service[nxlog]: Dependency File[C:/Program Files (x86)/nxlog/conf/nxlog.conf
] has failures: true�[0m
�[1;33mWarning: /Stage[main]/Nxlog::Service/Service[nxlog]: Skipping because of failed dependencies�[0m
�[1;33mWarning: /Stage[main]/Nxlog/Anchor[nxlog::end]: Skipping because of failed dependencies�[0m

Configuration problem

I am trying to install nxlog using this puppet module on Windows Server 2012 R2 on an AWS EC2 micro instance. I am using the default config file which I copied from the documentation here: https://github.com/genebean/genebean-nxlog#overview.

I am getting the error below. Why is it looking in ProgramData when the config says nxlog root dir is here: nxlog_root => 'C:\Program Files (x86)\nxlog'. The directory in the error message indeed does not exist; should this be created by the puppet module at some point? Is there something additional I should configure besdies what is listed in the README.md?

Error Message:

(at C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/type/file/source.rb:133:in `block in copy_source_values')
Error: Cannot create C:/ProgramData/PuppetLabs/puppet/var/concat/C_/Program Files (x86)/nxlog/conf_nxlog.conf;
parent directory C:/ProgramData/PuppetLabs/puppet/var/concat/C_/Program Files (x86)/nxlog does not exist

WHy so difficult ?

Why don't you use the init and define that, create resources out of it and loop thought the in/outputs ?

Arbitrary key/value pairs

Hi!

It might be helpful to offer the option to define arbitrary key/value pairs for each block.

For example, you might have this definition:

nxlog::config::input { 'eventlog_json':
  input_execs  => [
    'delete($Keywords)',
    '$raw_event = to_json()',
  ],
  input_module => 'im_msvistalog',
  arbitrary    => {
    'Query' => "<QueryList> \
        <Query Id='0'> \
          <Select Path='Security'>*</Select> \
        </Query> \
      </QueryList>",
  },
}

Which would translate to something along the lines of:

<Input eventlog_json>
    Module  im_msvistalog
    Query   <QueryList> \
              <Query Id="0"> \
                <Select Path="Security">*</Select> \
              </Query> \
            </QueryList>
    Exec    'delete($Keywords)'; '$raw_event = to_json()';
</Input>

With the idea that you wouldn't need to map out every possible key that folks might want to specify in the nxlog conf., you could leave corner cases to some arbitrary hash table that you parse.

Cheers!