Coder Social home page Coder Social logo

ansible-rsyslog's Introduction

DebOps logo DebOps

Your Debian-based data center in a box

GitHub CI GitLab CI CII Best Practices REUSE status RSS commits

The DebOps project provides a set of general-purpose Ansible roles that can be used to manage Debian or Ubuntu hosts. In addition, a default set of Ansible playbooks can be used to apply the provided roles in a controlled way, using Ansible inventory groups.

The roles are written with a high customization in mind, which can be done using Ansible inventory. This way the role and playbook code can be shared between multiple environments, with different configuration in to each one.

Services can be managed on a single host, or spread between multiple hosts. DebOps provides support for different SQL and NoSQL databases, web servers, programming languages and specialized applications useful in a data center environment or in a cluster. The project can also be used to deploy virtualization environments using KVM/libvirt, Docker or LXC technologies to manage virtual machines and/or containers.

You can find out more about DebOps features on the project's documentation page.

Quick start

Start a Docker container which acts as an Ansible Controller host with DebOps support, based on Debian Buster:

docker run -it --rm debops/debops
cd src/controller ; debops run common --diff

Or, create a Vagrant VM which acts as an Ansible Controller host:

git clone https://github.com/debops/debops
cd debops && vagrant up && vagrant ssh
cd src/controller ; debops run common --diff

You can use configuration in the src/controller subdirectory to try out DebOps against the container/VM, or create your own DebOps project directory using debops project init command.

More quick start tips can be found in the DebOps quick start guide.

Installation

You can install the DebOps Python package, which includes the DebOps roles and playbooks, as well as additional scripts which can be used to setup separate project directories and run Ansible in a convenient way. To install the Python package with Ansible and other required dependencies, run the command:

pip install --user debops[ansible]

Alternatively, DebOps roles are available on Ansible Galaxy as an Ansible Collection which can be installed using the ansible-galaxy command:

ansible-galaxy collection install debops.debops

Read the installation instructions in the DebOps documentation for more details about required software and dependencies.

Getting started

Ansible uses SSH to connect to and manage the hosts. DebOps enforces the SSH security by disabling password authentication, therefore using SSH keys to connect to the hosts is strongly recommended. This can be changed using the inventory variables.

During initial deployments you might find that the firewall created by DebOps blocked you from accessing the hosts. Because of that it's advisable to have an out-of-band console access to the host which can be used to login and troubleshoot the connection.

Create a new environment within a DebOps "project directory", add some hosts in the Ansible inventory and run the default DebOps playbook against them to configure them:

# Create a new environment
debops project init ~/src/projects/my-environment
cd ~/src/projects/my-environment

# Modify the 'ansible/inventory/hosts' file to suit your needs, for example
# uncomment the local host to configure it with DebOps

# Run the full playbook against all hosts in the inventory
debops run site

# Run the common playbook against specific host in the inventory
debops run common -l <hostname>

You should read the Getting Started with DebOps guide for a more in-depth explanation of how the project can be used to manage multiple hosts via Ansible.

Development

Create a fork of this repository and clone it to your workstation. Create a development DebOps environment and symlink the forked repository in it. Now you can create new playbooks/roles in the forked repository and see their results in the development environment.

git clone [email protected]:<username>/debops ~/src/github.com/<username>/debops
cd ~/src/github.com/<username>/debops
git remote add upstream https://github.com/debops/debops.git

debops project init ~/src/projects/debops-devel
cd ~/src/projects/debops-devel
ln -s ~/src/github.com/<username>/debops debops

You can pull latest changes to the project from the upstream repository:

cd ~/src/github.com/<username>/debops
git checkout master
git fetch upstream
git rebase upstream/master

Read the development guide file for more details about the DebOps development process.

Contributing

DebOps development is done via a distributed development model. New features and changes are prepared in a fork of the official repository and are published to the original repository via GitHub pull requests. PRs are reviewed by the DebOps developer team and if accepted, are merged in the main repository.

GPG-signed git commits are preferred to ensure authenticity.

Read the contributing guide file for more details about how to contribute to DebOps.

Licensing

The DebOps project is licensed under the GNU General Public License 3.0 or later. You can find full text of the license in the LICENSES/GPL-3.0-or-later.txt file.

Some files included with the DebOps project use a different license. The licenses are marked in these files using the SPDX license identifiers and can be found in the LICENSES/ subdirectory. They are also included in the project tarballs, Ansible Collections and Python packages. The project uses the REUSE Specification and its associated tool to check and verify copyright and license information in all files.

ansible-rsyslog's People

Contributors

bleuchtang avatar drybjed avatar ganto avatar jbicha avatar le9i0nx avatar ypid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

ansible-rsyslog's Issues

Problem using network capability

Hi !

Im trying to configure a centralized rsyslog server.
After i enable the 'network' capability, the rsyslog throws an error:
rsyslogd-2209: input module name 'imudp' is unknown [v8.16.0 try http://www.rsyslog.com/e/2209 ]

It seems the role is generating the input file, but the network module file is empty.
After checking the code, some conditions does not match in defaults/main.yml file:
- comment: 'Enable UDP support' options: |- module(load="imudp") state: '{{ "present" if (rsyslog__send_over_tls_only) else "absent" }}'
...
- comment: 'Log messages from remote hosts over UDP' options: |- input( type="imudp" port="{{ rsyslog__udp_port }}" ruleset="remote" ) state: '{{ "present" if (not rsyslog__send_over_tls_only) else "absent" }}'

If rsyslog__send_over_tls_only is not enabled, the input is defined and the module is not loaded. If you use tls_only, the module is loaded but the input is not defined.

¿ is this some kind of restriction to force use of tls ? ¿ is there some configuration option to aviod it ?

Network capability does not work

Hi, possibly related to #16. Setting rsyslog role with network capability doesn't bind to a port:

Steps to reproduce, using examples/vagrant-multi-machine:

  • Add the following to ansible/inventory/groups
[debops_all_hosts]
web
db

[debops_service_rsyslog]
web
  • Add the following to ansible/inventory/host_vars/web.yml
# Enable network input channels and storage of remote logs in filesystem
rsyslog__capabilities: [ 'network', 'remote-files' ]

# Specify which subnets can send remote logs through the firewall
# allowing all for this test
rsyslog__host_allow: [ '0.0.0.0/0' ]

# Mask log forwarding configuration defined elsewhere
rsyslog__forward: []
rsyslog__group_forward: []
rsyslog__host_forward: []
  • Add the following to ansible/inventory/group_vars/all.yml
rsyslog__forward: [ '*.* @web.{{ ansible_domain }}' ]
  • vagrant up
  • debops
  • vagrant ssh web
  • netstat -ltupn # note that rsyslog is not listening
  • ls /var/log/remote/hosts # note there is only a 'web', but no 'db'

rsyslog__enabled: False turns off logging

Setting rsyslog__enabled: False has a surprising result: it turns off logging.

What I expect to happen

debops does not touch anything in /etc/rsyslog.d/

Log

Here's a log from a Ubuntu server. It's pretty verbose, but basically what it's doing is deleting /etc/rsyslog.d/20-ufw.conf and /etc/rsyslog.d/50-defaults.conf and replacing them with nothing.

TASK [debops.rsyslog : Remove custom config files when requested] **************
ok: [server1.example.com] => (item={u'comment': u'Global options', u'options':
u'global(\n defaultNetstreamDriver="ptcp"\n)', u'filename': u'00-global.conf'})
ok: [server1.example.com] => (item={u'type': u'modules', u'sections': [{u'comment': u'Log
messages sent to local UNIX socket', u'options': u'$ModLoad imuxsock'}, {u'comment': u'Log
kernel messages', u'state': u'present', u'options': u'$ModLoad imklog\n$KLogPermitNonKernelFacility
on'}, {u'comment': u'Log periodic -- MARK -- messages', u'state': u'present', u'options': u'$ModLoad immark\n$MarkMessagePeriod 3600'}], u'name': u'local-modules'})
ok: [server1.example.com] => (item={u'type': u'modules', u'state': u'absent', u'sections': [{u'comment': u'Enable UDP support', u'state': u'absent', u'options': u'module(load="imudp")'}, {u'comment': u'Enable TCP support', u'state': u'absent', u'options': u'module(load="imptcp")'}, {u'comment': u'Enable GnuTLS TCP support', u'state': u'absent', u'options': u'module(\n  load="imtcp"\n  streamDriver.name="gtls"\n  streamDriver.mode="1"\n  streamDriver.authMode="x509/name"\n      permittedPeer=["*.bicha.net"]\n  )'}], u'name': u'network-modules'})
ok: [server1.example.com] => (item={u'type': u'global', u'sections': [{u'comment': u'Set default permissions for all log files', u'options': u'$FileOwner syslog\n$FileGroup adm\n$FileCreateMode 0640\n$DirCreateMode 0755\n$Umask 0022\n$PrivDropToUser syslog\n$PrivDropToGroup syslog\n'}, {u'comment': u'Where to place spool and state files', u'options': u'$WorkDirectory /var/spool/rsyslog'}, {u'comment': u'Log every message', u'options': u'$RepeatedMsgReduction off'}], u'name': u'common-defaults'})
changed: [server1.example.com] => (item={u'divert_to': u'65-ufw.system', u'divert': True, u'state': u'present', u'filename': u'20-ufw.conf'})
--- before
+++ after
@@ -1,4 +1,4 @@
 {
     "path": "/etc/rsyslog.d/20-ufw.conf", 
-    "state": "file"
+    "state": "absent"
 }
changed: [server1.example.com] => (item={u'divert': True, u'state': u'present', u'filename': u'50-default.conf'})
--- before
+++ after
@@ -1,4 +1,4 @@
 {
     "path": "/etc/rsyslog.d/50-default.conf", 
-    "state": "file"
+    "state": "absent"
 }

Manage /etc/rsyncd.conf

Ran into this working with debops.fail2ban. By default, rsyslogd compresses messages which messes with fail2ban. The solution is to change the RepeatedMsgReduction setting in /etc/rsyncd.conf.

debops.rsyslog doesn't manage this file at this time. Does it make sense to have it do it?

Proposed /etc/rsyslog.d/ layout

At the moment I'm working on new debops.rsyslog configuration. The idea of the default configuration is to provide a set of capabilities, like Postfix has, and enable different parts of the default configuration depending on what capabilities (and some other variables) are set. For example, enable remote logging if 'network' capability is enabled, enable TLS logs when debops.pki is configured and so on.

Due to how rsyslog configuration works, I want to implement different rulesets like this:

# In default /etc/rsyslog.conf, include all default configuration files
$IncludeConfig /etc/rsyslog.d/*.conf

# System, or local logs in /var/log/
ruleset(name="system") {
  $IncludeConfig /etc/rsyslog.d/*.system
}

# Logs from remote host filtered separately in /var/log/remote/
ruleset(name="remote") {
  $IncludeConfig /etc/rsyslog.d/*.remote
  action(type="omfile" DynaFile="RemoteHost")
}

The issue I have is how to split different configuration files. Currently they are split using the extension:

00-global.conf
10-some-file.conf
20-local-logs.system
30-other-file.conf
30-default.template
50-remote-logs.remote

This makes things easier to manage from Ansible perspective (just store your config with a specific extension in /etc/rsyslog.d/, however might not be easy to process by a human (different file types are processed in different order than the apparent alphabetical one).

The other idea I've looked into is to create separate subdirectories for different files, something like:

/etc/rsyslog.d/template/
    30-default.conf

/etc/rsyslog.d/system/
    20-local-logs.conf

/etc/rsyslog.d/remote/
    50-remote-logs.conf

Different configuration types would be put in different directories, which makes parsing the configuration easier by a human, however Ansible needs to create separate directories based on a list, and adding more customization requires modification of that list, etc.

Any thoughts about preferred approach, or proposals for an alternative one? The default rsyslog configuration including /etc/rsyslog.d/*.conf makes it easier to include configuration from different Debian packages, however when rulesets are enabled, these files might need to be diverted to be read by a "system", ie. local ruleset correctly. So this might need to be done in either case.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.