cesnet / nemea-framework

Nemea framework is the heart of the Nemea system. It contains implementation of common communication interfaces, UniRec data format and useful datastructures and algorithms.

Nemea Framework

This repository contains base libraries for a Nemea system. The Nemea system is a modular system that consists of independent modules for network traffic analysis and anomaly detection.

The framework consists of:

• libtrap -- communication interface for messages transfer between Nemea modules
• UniRec -- flexible and efficient data format of flow-records
• common -- usefull common functions and data structures
• pytrap -- python wrapper for libtrap and UniRec that allows development of nemea modules in python
• pycommon -- python common modules and methods, there is currently a support of alerts creation in the IDEA format that can be stored into MongoDB or sent to the Warden incident sharing system

Dependencies

To be able to compile Nemea Framework on GNU/Linux distributions, you will need to have installed on your system the following depencies which were needed on Ubuntu 20.04 LTS:

make autoreconf libtool libxml2-dev

Installation

This repository is usually used as a git submodule of https://github.com/CESNET/Nemea However, it can be installed independently using:

./bootstrap.sh
./configure
make
sudo make install

For information about configuration options see:

./configure --help

Python parts must be installed separately when needed. It can be done using:

cd pytrap; sudo python3 setup.py install

and

cd pycommon; sudo python3 setup.py install

Project status:

License:

Travis CI build:

Coverity Scan:

CodeCov:

Ohloh:

nemea-framework's Issues

Memory leak in libtrap

Valgrind detect memory leak in trap_init.
It was probably caused by commit:
c98829f

You can check it even in example module:
https://github.com/CESNET/Nemea-Framework/tree/master/examples/c/module
valgrind --leak-check=full ./example_module -f -i u:soc,b
or check it with valid parameters and press Ctrl+c
valgrind --leak-check=full ./example_module -i u:soc,b

Valgrind output:
==12650== at 0x4C306B5: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==12650== by 0x4E45307: trap_ctx_init2 (trap.c:1848)
==12650== by 0x4E46683: trap_ctx_init (trap.c:1794)
==12650== by 0x4E4707C: trap_init (trap.c:530)
==12650== by 0x401852: main (example_module.c:120)
==12650==
==12650== LEAK SUMMARY:
==12650== definitely lost: 8 bytes in 1 blocks
==12650== indirectly lost: 0 bytes in 0 blocks
==12650== possibly lost: 0 bytes in 0 blocks
==12650== still reachable: 0 bytes in 0 blocks
==12650== suppressed: 0 bytes in 0 blocks

OS - openSUSE Leap 15.2
branch: master

Higher port numbers allowed in libtrap tcp interface

Try running for example flow_meter -i t:65570 or flow_meter -i t:655701. It allows you to run it and probably overflows, for port 65570 it was running on port 34 when listing netstat -l (it is not surprisingly 65570-65536).

Fix API of ip_prefix_search

Revisit https://github.com/CESNET/Nemea-Framework/blob/ip_prefix_search/common/include/ip_prefix_search.h

Public API - that means content of this header file MUST contain only a minimal set of functions that is necessary for usage. Remove all functions that are used just internally.

Remove included header files that are not necessary for users. E.g. you have included signal.h, stdio.h and others - these header files are probably not necessary to have in the public API.

Public API MUST have all public functions named with some common prefix (it should be logical) in order to recognize that they belong together. You can use ipps_ if you want.

Revisit https://github.com/CESNET/Nemea-Framework/blob/ip_prefix_search/common/ip_prefix_search/ip_prefix_search.c and mark all "private" (internally used) functions as static.

Improve error message when wrong number of IFC is given in -i

Scenario:
A module, which has 1 input and 1 output IFC, is started with just 1 IFC_SPEC in -i.
There is an error message: "Error in TRAP initialization: No allocated global context"
This is not enough for users to solve the problem.
(Giving 2 IFC solves the problem.)

pytrap: failing tests on i386 arch

Building 32b package in Copr failes in all distro versions.

Sample output from a build:

copying build/lib.linux-i686-2.6/pytrap.so -> 
runTest (test.pytrapmodule_unittest.DeviceTest) ... ok
runTest (test.pytrapmodule_unittest.TrapCtxInitTest) ... ok
runTest (test.unirectemplate_unittest.DataAccessGetTest) ... ok
runTest (test.unirectemplate_unittest.DataAccessSetTest) ... FAIL
runTest (test.unirectemplate_unittest.DataTypesIPAddr) ... ok
runTest (test.unirectemplate_unittest.DataTypesTime) ... ok
runTest (test.unirectemplate_unittest.DeviceTest) ... ok
runTest (test.unirectemplate_unittest.Template2Test) ... ok
runTest (test.unirectemplate_unittest.Template3Test) ... ERROR
runTest (test.unirectemplate_unittest.TemplateSizeTest) ... ok
runTest (test.unirectemplate_unittest.TemplateTest) ... ok
======================================================================
ERROR: runTest (test.unirectemplate_unittest.Template3Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/nemea-pytrap-0.9.8/test/unirectemplate_unittest.py", line 291, in runTest
    self.assertTrue(a.U64 == 0x100000000)
OverflowError: long int too large to convert to int
======================================================================
FAIL: runTest (test.unirectemplate_unittest.DataAccessSetTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/nemea-pytrap-0.9.8/test/unirectemplate_unittest.py", line 216, in runTest
    self.assertEqual(a.ABC, int(0))
AssertionError: 222 != 0
----------------------------------------------------------------------
Ran 11 tests in 0.013s
FAILED (failures=1, errors=1)
error: Bad exit status from /var/tmp/rpm-tmp.gByka6 (%check)
RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.gByka6 (%check)
Child return code was: 1
EXCEPTION: [Error()]
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 88, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.5/site-packages/mockbuild/util.py", line 562, in do
    raise exception.Error("Command failed. See logs for output.\n # %s" % (command,), child.returncode)
mockbuild.exception.Error: Command failed. See logs for output.
 # bash --login -c /usr/bin/rpmbuild -bb --target i686 --nodeps /builddir/build/SPECS/nemea-pytrap.spec

Affected distro versions: epel-6-i386, fedora-24-i386, fedora-25-i386, fedora-rawhide-i386

Feature request: Print a message if module can't connect to input interface

If I run a module (e.g. logger) manually from a terminal and don't see any data coming, it's not obvious whether there are actually no data on the input interface, or the module can't connect to it, e.g. because of a wrong interface name used.

I know there are ways to determine the status of each interface, e.g. using -vvv or supcli -x, but none of it is practical for the common case of testing something by running modules from a terminal. An explicit message from the module would be much more user-friendly.

I propose the following:
If the input interface (any of them) fails to establish a connection AND stderr is a tty, print a message to stderr (e.g. "Can't connect to {ifc}, retrying ...").

Ideally also with a reason why it can't connect (e.g. unix-socket with such name doesn't exist, tcp connection refused, incompatible data format) and with a "connected" message when it connects succesfully afterwards.

I think it souldn't break anything, since it will print messaes only to terminal, and it would make testing/debugging much easier.

Revision of ip_prefix_search README

Revisit README and try to improve language.

It would be much better to convert the file format into markdown (optional)...

libtrap: Wrong return value handling in tcpip and tls interface

During connection to TCP socket in both tcpip and tls interface if there an error occures (i.e. TRAP_E_TIMEOUT)

https://github.com/CESNET/Nemea-Framework/blob/master/libtrap/src/ifc_tcpip.c#L861
https://github.com/CESNET/Nemea-Framework/blob/master/libtrap/src/ifc_tls.c#L1011

After setting return value rv to some error the function can still go to end and return TRAP_E_OK which is clearly not intended.

Expected behavior:
Set rv respectively in function and in the end return rv instead of implicit TRAP_E_OK.

Libtrap: Change default location of UNIX sockets

Currently, all UNIX sockets (u type of TRAP IFC) are being created in /tmp, which is not the best idea since various services use PrivateTmp= (in systemd etc.)

If we move the default location to somewhere else (like into /var/run/nemea-supervisor), we can loose a possibility to start a module manually as a normal user.

The proposed solution is to create a separate directory e.g. /var/run/libtrap and set it writable to everyone and set sticky bit. (this is the way /tmp works)
However, this must be solved and tested both using just make(1) and using RPM installation.

Make the UniRec thread safe ...and therefore great!

It would be lovely if somebody took the challenge and made UniRec functions and macros thread safe.
Since currently they are not and it could be really confusing for newcomers.
Locking sections with initialization and deletion of UniRec templates and records is not the biggest issue, but the worst part comes when using TRAP_RECEIVE macro.

Eternal gratitude to whomever who solves this problem correctly and elegantly.

Add RPM build into Travis CI builds

Install rpmbuild in the .travis.yml and add a code that will test the creation of RPM packages in the Travis infrastracture (generally, we do this by make rpm).

The tests are necessary for the following packages:

• libtrap
• unirec
• nemea-common

Invalid help string for createMessage function in pytrap.

The help string for the createMessage function in pytrapmodule.c seems to incorrectly state that the maximum size for a message containing variable length fields is 100.

Nemea-Framework/pytrap/src/pytrapmodule.c

Line 617 in c372105

" # 100 is the maximal total size of fields with variable length\n" \

However, when I was using this module I had no issue making messages of size 500, 700, 1000, and 5000. This leads to me to believe the docstring is outdated. I would create a pull request but I don't know what the correct value in the docstring should be, or if it should really even specify a maximum size.

Hope this helps,

Lukáš

Debian package for nemea-common

Add necessary files and make neccessary changes to generate the following deb package:

• nemea-common (in https://github.com/CESNET/Nemea-Framework/tree/master/common)

For inspiration, have a look into https://github.com/CESNET/Nemea-Framework/tree/master/libtrap/debian and https://github.com/CESNET/Nemea-Framework/blob/master/libtrap/Makefile.am#L47

Fix coding style of ip_prefix_search

See CodingStyle and fix the files from the PR #22

BUGFIX CPU load increases to 100%

Probably due to a bug in libtrap, cpu load rises to 100% of a core when there is TIME_WAIT on an IFC and nobody is connected. Both TCP and UNIX output IFCs are affected.

This issue is very important for embedded devices such as OpenWrt routers...

unirec array

Newly added support for unirec array needs to be tested.

https://github.com/CESNET/Nemea-Framework/tree/unirec_array

Feature request: Logger shouldn't discard nonprintable characters

Currenlty, logger module (or rather unirec2csv functionality in Nemea-Framework) doesn't show non-printable characters in string-type fields. They are quietly discarded (see line 160). Similarly, new-line characters (\n) are converted to spaces (line 149).

I think this behavior is quite bad for debugging, as it hides a potentially important information (we're just dealing with a bug where there is \r at the end of some fields and it would help a lot if logger showed it).

I propose to print such characters using \x00 format or something silimar. (Maybe \u0000 if I remember correctly that string fields are expected to be in UTF-8 encoding?)

It needs some discussion though, as it's a change of behavior which can potentially break something (although I think it's not likely). And a corresponding change in logreplay may be necessary, if we want to keep that anything logged by logger can be replayed by logreplay.

Drop "multi_recv" feature from lbtrap

The multi_recv function aimed to provide a simple API to get a list of messages among multiple interfaces. Due to low flexibility, NEMEA modules probably handle receiving from multiple interfaces on their own (using own threads or openmp etc.).
Since there are "reader" threads in libtrap internals that are created during initialization, it is a useless resource consumption when the feature is not used - is it really unused in all modules? How can we find out that there is no NEMEA module that uses multi_recv?

It was proposed by @janskto1 to drop this feature from the libtrap API.

I generally agree because it will clean up code a bit.
Any comments?

Ring buffer and signalling in libtrap

Finish the implementation of ring buffer and signalling features and integrate them into libtrap.
The current work-in-progress is in the signalling branch.

The Ring buffer is tested by libtrap/tests/test_trap_buffer.c and implemented in libtrap/src/trap_buffer.c and libtrap/src/trap_buffer.h

There is already a finished part of TCP IFC implementation: https://github.com/CESNET/Nemea-Framework/blob/signalling/libtrap/src/ifc_tcpip.c#L1071 and the sending is done in https://github.com/CESNET/Nemea-Framework/blob/signalling/libtrap/src/ifc_tcpip.c#L1600

The receiving part is missing, it should use the new buffer too.

Current implementation of Ring buffer for push_mess and FLUSH probably won't work for size of 1 block... It should be solved somehow.

unirec/ipaddr_cpp.h: wrong copy constructor

Piece commented in unirec/ipaddr_cpp.h as copy constructor is not one. Valid copy constructor is needed. This piece of code is used so far only here in hoststats.

BUGFIX segfault when DEFAULTSOCKETPATH doesn't exist

Bug can be easily simulated by configuring with --with-defaultsocketdir=/ and run e.g. a basic test:

./basic_test -i "t:localhost:50001"

as a normal user.
This path leads to permission denied error and failure during the creation of socket.

ERROR: Socket could not be opened on given port 'service_19534'.
ERROR: Error while creating service IFC.

Docs: Syntax of rules in Reporter config is not documented

I can't find any documentation of rules syntax in reporter configuration. The corresponding documentation section only states it is as "supported by Mentat filter (MFilter)" - which is wrong (the library is called pynspect) and there is no link.

Please, at least fix the name and add a link to pynspect documentation.

The other issue is that pynspect itself is not well documented, but you can't fix that.

Btw: The original issue I stumbled upon is a wierd behavior of IP address matching. When I want to match Source.IP4 to a single IP address, this doesn't work:
Source.IP4 == "1.2.3.4"
If I write it this way, it suddenly works:
Source.IP4 in [ "1.2.3.4" ]

I know you probably can't fix that, but maybe it can be mentioned in the documentation as a known bug (or just unintuitive behaviour)?

libtrap: misaligned address

Hi,

I'm working on an IoT detector using a nemea-framework. The detector works fine on the x86 architecture, but when tested on a Turris router with OpenWRT and an ARM processor, it crashes on a Bus Error when receiving a message and then retrieving message items using the ur_get () macro. Crashes only occur when compiler optimization is turned on. This line of code causes crash.

ZWaveDetector.cpp:236: double event_type = ur_get(in_events_template, in_record, F_EVENT_TYPE);

I tried compilation using clang with a sanitizer. I got this output after sending UniRec message to detector:

ZWaveDetector.cpp:235:26: runtime error: load of misaligned address 0x63300001c84a for type 'ur_time_t' (aka 'unsigned long'), which requires 8 byte alignment
0x63300001c84a: note: pointer points here
00 00 f0 3f 22 db f9 1e 54 27 c5 5e 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ZWaveDetector.cpp:235:26 in

ZWaveDetector.cpp:236:24: runtime error: load of misaligned address 0x63300001c81a for type 'double', which requires 8 byte alignment
0x63300001c81a: note: pointer points here
00 00 00 00 00 00 00 00 00 00 32 40 00 00 00 00 00 00 00 00 00 00 20 49 84 32 ec 41 00 00 00 00
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ZWaveDetector.cpp:236:24 in

You can reproduce it with following sequence of commands:

$ git clone https://github.com/CESNET/NEMEA-SIoT.git
$ git checkout libtrap-bug-reproduce
$ cd zwave-detector
$ ./bootstrap.sh
$ ./configure CXX=clang++ CXXFLAGS=-fsanitize=address,undefined
$ make
$ logreplay -f test_input.csv -i u:events &
$ ./siot-zwave-detector -i u:frames,u:events,u:alerts

Strict aliasing issues may cause bus errors on ARM architecture, so it seems to me that the issue can be conected with the problems reported by ASan.

Bad return value

Modules that use the following macro:

trap.h:969 #define TRAP_DEFAULT_GET_DATA_ERROR_HANDLING(ret_code, timeout_cmd, error_cmd) \

prints:

Error: trap_get_data() returned 24 (No error)

when format is changed to "mismatch" (a required field is missing).

This error message is very confusing and should be improved as fast as possible -> trap_get_data() is obsoleted, 24 is not clear for users, No error increases confusement.

Python Module Commandline Arguments

How can Python Module commandline arguments be defined and used?

I see in README.md...

NOTE: ./python_example.py claims that it has additional parameters -f and -q. However, these parameters have no effect. They are listed just for a demonstration of the optparse.OptionParser class that can be used in python Nemea modules for unified help output.

... but looking at the source, there is no mention of the additional parameters.

Are additional parameters possible?

libtrap: tls interface: check possible issues with TLS setup

In setting up TLS connection between modules return values should be better checked for errors. Especially around https://github.com/CESNET/Nemea-Framework/blob/master/libtrap/src/ifc_tls.c#L1039

Also should check again if everything works as expected (and does not leak) since I have improved my openSSL knowledge since last year.

TLS interface: remove useless and confusing CHECK_AND_FREE macro

CHECK_AND_FREE macro in TLS interface only decreases code readability while not doing anything usefull. I will remove it.

Revision of B+ tree docs

Source file contains typos and missing params of doxy

Missing description of File IFC parameters

Rotation of files using file IFC is not described in README (and in help string that is generated from the same file).
New feature was proposed in 2c6163f

report2idea: Update `--dry` to print info about smtp_connections etc

These sections are out-of-date:

https://github.com/CESNET/Nemea-Framework/blob/report_template/pycommon/reporter_config/actions/Email.py#L155
https://github.com/CESNET/Nemea-Framework/blob/report_template/pycommon/reporter_config/Config.py#L175

due to #87.

File IFC creates files with _ suffix

Using:

/usr/bin/nemea/flow_meter -I eth0 -i f:/tmp/data.trapcap:w:time=2

I got the following file: data.trapcap.201607251739_ and data.trapcap.201607251741_

I am not sure if the trailing _ is a good idea.

Bug in setting of autoflush timeout

I reviwed the code and I think there should be timeout_fixed, not datatimeout_fixed on the following line:

Nemea-Framework/libtrap/src/trap.c

Line 2075 in 0ac1029

ifc->datatimeout_fixed = 1;

Review "ifcspec" help

Text in README.ifcspec.md, which is also written by "-h trap", could be written better (e.g. explain that the first parameter is ifc type, other params depends on the type, etc.; english could be improved as well).

Also, remove unnecessary line-breaks. Libtrap have code which breaks lines automatically according to terminal width and manual line-breaking collides with that.

python3 support

We need to improve python wrapper code in python/ in order to support python3.
Some work is already done in #2 in the CESNET/Nemea-Framework/python3 branch.
It should be tested using the python_example.py module.

report2idea ignores name from warden config file

The reporter name from warden config file is ignored by all report2idea modules. The reason is here:

Nemea-Framework/pycommon/report2idea.py

Line 159 in 6b65653

config['name'] = args.name

The name of the reporter module (parameter --name) is used instead. It should either be documented, or the name from warden config should be used.

Add --without-libxml2 parameter into configure

Debian package for UniRec

Add necessary files and make neccessary changes to generate the following deb package:

• unirec (in https://github.com/CESNET/Nemea-Framework/tree/master/unirec)

For inspiration, have a look into https://github.com/CESNET/Nemea-Framework/tree/master/libtrap/debian and https://github.com/CESNET/Nemea-Framework/blob/master/libtrap/Makefile.am#L47

libtrap: multi-file read errors --> `strncpy` misuse?

Nemea-Framework/libtrap/src/ifc_file.c

Line 378 in a57307a

strncpy(config->filename, config->files[config->file_index], strlen(config->files[config->file_index]));

For reading multiple files from a directory by globbing, won't this miss the null termination when a shorter filename follows a longer filename?
Ref: CESNET/LiST#10

From strncpy(3) man page:

Warning: If there is no null byte among the first n bytes of src, the string placed in dest will not be null-terminated.

Fix doxygen for UniRec

e.g. https://github.com/CESNET/Nemea-Framework/blob/master/unirec/ipaddr.h is totally missing doxygen for functions - format of comments is not recognized.