redborder / f2k Goto Github PK
View Code? Open in Web Editor NEWnetflow 2 kafka translator
License: GNU Affero General Public License v3.0
netflow 2 kafka translator
License: GNU Affero General Public License v3.0
Non POSIX expression on Makefile causes unexpected behavior where all tests are running in the background.
Using standard assert.h
causes tests to silently fail.
we are attempting to run f2k on centos 7.
f2k immediately segfault on start. this is not entirely deterministic as it will continue to run every once in a while:
16/Oct/2017 16:33:35 [rb_listener.c:154] Creating listening socket in port 9996
16/Oct/2017 16:33:35 [rb_kafka.c:58] Applying socket.keepalive.enable=true to rdkafka
16/Oct/2017 16:33:35 [rb_kafka.c:58] Applying socket.max.fails=3 to rdkafka
16/Oct/2017 16:33:35 [rb_kafka.c:58] Applying socket.keepalive.enable=true to rdkafka
16/Oct/2017 16:33:35 [rb_kafka.c:58] Applying socket.max.fails=3 to rdkafka
Segmentation fault (core dumped)
the segfault is due to readOnlyGlobals.rb_databases.sensors_info being null:
#0 get_sensor (database=0x0, ip=181607651) at rb_sensor.c:1329
#1 0x00000000004136e7 in netFlowCollectLoop0 (collector=<optimized out>, collector=<optimized out>) at rb_listener.c:87
#2 netFlowCollectLoop (_port_collector=0x625460) at rb_listener.c:136
#3 0x00007ffff7bc6dc5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007ffff671d28d in clone () from /lib64/libc.so.6
the commandline we use:
f2k --debug --kafka=10.212.225.48:[email protected] --collector-port=9996 --rb-config=config_basic.json -b99 --template ../templates/ --event-log=test.log
any help advise appreciated.
I have compiled an f2k and installed dependencies on a centos 7 system. When I execute it I get the following output:
./f2k
10/Jan/2017 11:18:22 [f2k.c:1614] Welcome to f2k v.6.13.170110 (1.0.1-61-g2e1b16) for Linux
10/Jan/2017 11:18:22 [f2k.c:2307] Welcome to f2k v.6.13.170110 for Linux
10/Jan/2017 11:18:22 [f2k.c:2356] Flows ASs will not be computed
10/Jan/2017 11:18:22 [util.c:828] nProbe changed user to 'nobody'
Violación de segmento
Add integration tests. This should be done by running:
Then send NetFlow traffic to f2k and check data readed from Kafka.
The repository should be integrated with Travis CI for automated tests, builds and coverage.
Integration tests involves some services that may be not available during tests, so they should be disabled by default and could be enabled using:
./configure --enable-integration-tests
Currently, the netflow from sensors with unknown IP addresses is being discarded, but not data from sensors with known IP address and unknown observation domain.
f2k should print L1 and L2 domain to Kafka.
I have detect some errors in script called by make manuf, which is tools/manuf.py. Shebang is incorrect, and when script tries to write mac_vendors file fails due to incorrect file codifications.
I have fixed this errors and I have create the following pull request: #39
The following functions on export.c
are not tested. They should be tested:
The following error is detected by Helgrind on an external library:
==29300== Thread #2: pthread_cond_{timed}wait called with un-held mutex
==29300== at 0x4C35954: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==29300== by 0x4C35A19: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==29300== by 0x4E43C5C: rd_fifoq_pop0 (in /usr/local/lib/librd.so)
==29300== by 0x410C55: popPacketFromQueue_timedwait (util.h:90)
==29300== by 0x410C55: netFlowConsumerLoop (collect.c:1456)
==29300== by 0x4C34DB6: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==29300== by 0x50556F9: start_thread (pthread_create.c:333)
==29300== by 0x6471B5C: clone (clone.S:109)
This would be a possible suppression:
{
librd rdqueue
Helgrind:Misc
obj:/usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so
obj:/usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so
fun:rd_fifoq_pop0
fun:popPacketFromQueue_timedwait
fun:netFlowConsumerLoop
obj:/usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so
fun:start_thread
fun:clone
}
The following functions on export.c
are not tested. They should be tested or removed in case they are dead code.
All the data received from unknown IP addresses should be dumped to a Kafka topic.
Currently builds on Travis CI are not reporting coverage to Coveralls (always send 0%).
Field names for the PTR DNS resolution should be renamed:
client_name
-> lan_ip_name
target_name
-> wan_ip_name
The application should be able to allow unregistered IP address if the sensors send an option template with a valid serial number.
Use case (current behavior):
listener
gets the message and forwards it to a worker.Use case:
listener
gets the message and forwards it to a worker with unknown_ip=1
.option template
exists with a serial number
field.It's necessary to implement a Kafka consumer to be able to get raw netflow data from a Kafka queue instead from an UDP socket.
I am trying to install f2k in my ubuntu14 machine by cloning the git repo and executing the the step "./configure && make && make install". However, I'm getting lots of "no such header file errors". Please find below the snippet that I got after executing the step. There were other errors too, which I removed by removing some of dependent packages. But, for below errors, I'm not able to find any proper solution.
root@ubuntu:/home/devops/f2k# ./configure
checking for OS or distribution... ok (Ubuntu)
checking for C compiler from CC env... failed
checking for gcc (by command)... ok
checking executable ld... ok
checking executable nm... ok
checking executable objdump... ok
checking executable strip... ok
checking for pkgconfig (by command)... ok
checking for install (by command)... ok
checking for __atomic_32 (by compile)... ok
checking for __atomic_64 (by compile)... ok
checking for socket (by compile)... ok
checking for librd (by pkg-config)... failed
checking for librd (by compile)... failed (fail)
checking for pcap (by pkg-config)... failed
checking for pcap (by compile)... failed (fail)
checking for librdkafka (by pkg-config)... failed
checking for librdkafka (by compile)... ok
checking for rb_mac_vendor (by pkg-config)... failed
checking for rb_mac_vendor (by compile)... failed (fail)
checking for geoip (by pkg-config)... ok
checking for zookeeper (by pkg-config)... failed
checking for zookeeper (by compile)... ok
checking for udns (by pkg-config)... failed
checking for udns (by compile)... failed (fail)
checking for HAVE_JSON (by pkg-config)... failed
checking for HAVE_JSON (by compile)... ok
checking for optreset (by compile)... failed (disable)
checking for pthread (by pkg-config)... failed
checking for pthread (by compile)... ok
checking for pthread_setaffinity_np (by compile)... failed (disable)
checking for sin6_len (by compile)... failed (disable)
checking for netfilter (by pkg-config)... failed
checking for netfilter (by compile)... failed (disable)
checking for sctp (by compile)... failed (disable)
checking for pcap_next_ex (by compile)... failed (disable)
checking for pf_ring (by pkg-config)... failed
checking for pf_ring (by compile)... failed (disable)
librd ()
module: f2k
action: fail
reason:
compile check failed:
CC: CC
flags: -lrd -lpthread -lz -lrt
gcc -Wno-missing-field-initializers -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -O2 -g -Wcast-qual -Wunused -Wextra -Wdisabled-optimization -Wshadow -Wmissing-declarations -Wundef -Wswitch-default -Wmissing-include-dirs -Wstrict-overflow=5 -Winit-self -Wlogical-op -Wcast-align -Wdisabled-optimization -DNDEBUG -D_GNU_SOURCE -DFORTIFY_SOURCE=2 -Wall -Werror -lrd -lpthread -lz -lrt _mkltmp8AkgWk.c -o _mkltmp8AkgWk.c.o :
_mkltmp8AkgWk.c:1:22: fatal error: librd/rd.h: No such file or directory
#include <librd/rd.h>
^
compilation terminated.
source: #include <librd/rd.h>
pcap ()
module: f2k
action: fail
reason:
compile check failed:
CC: CC
flags: -lpcap
gcc -Wno-missing-field-initializers -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -O2 -g -Wcast-qual -Wunused -Wextra -Wdisabled-optimization -Wshadow -Wmissing-declarations -Wundef -Wswitch-default -Wmissing-include-dirs -Wstrict-overflow=5 -Winit-self -Wlogical-op -Wcast-align -Wdisabled-optimization -DNDEBUG -D_GNU_SOURCE -DFORTIFY_SOURCE=2 -Wall -Werror -lpcap _mkltmpDRRB09.c -o _mkltmpDRRB09.c.o :
/usr/bin/ld: cannot find -lpcap
collect2: error: ld returned 1 exit status
source:
rb_mac_vendor (HAVE_RB_MAC_VENDORS)
module: f2k
action: fail
reason:
compile check failed:
CC: CC
flags: -lrb_mac_vendors
gcc -Wno-missing-field-initializers -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -O2 -g -Wcast-qual -Wunused -Wextra -Wdisabled-optimization -Wshadow -Wmissing-declarations -Wundef -Wswitch-default -Wmissing-include-dirs -Wstrict-overflow=5 -Winit-self -Wlogical-op -Wcast-align -Wdisabled-optimization -DNDEBUG -D_GNU_SOURCE -DFORTIFY_SOURCE=2 -Wall -Werror -lrb_mac_vendors _mkltmpmtvaLo.c -o _mkltmpmtvaLo.c.o :
_mkltmpmtvaLo.c:1:28: fatal error: rb_mac_vendors.h: No such file or directory
#include <rb_mac_vendors.h>
compilation terminated.
source: #include <rb_mac_vendors.h>
udns (HAVE_UDNS)
module: f2k
action: fail
reason:
compile check failed:
CC: CC
flags: -ludns
gcc -I/usr/include/ -Wno-missing-field-initializers -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -O2 -g -Wcast-qual -Wunused -Wextra -Wdisabled-optimization -Wshadow -Wmissing-declarations -Wundef -Wswitch-default -Wmissing-include-dirs -Wstrict-overflow=5 -Winit-self -Wlogical-op -Wcast-align -Wdisabled-optimization -DNDEBUG -D_GNU_SOURCE -DFORTIFY_SOURCE=2 -Wall -Werror -ludns _mkltmpNkVGfP.c -o _mkltmpNkVGfP.c.o :
/tmp/ccA5w4FZ.o: In functionf': /home/devops/f2k/_mkltmpNkVGfP.c:2: undefined reference to
dns_init'
collect2: error: ld returned 1 exit status
source: #include <udns.h>
void *f();void *f(){return dns_init;}
It's necessary to recreate memory errors to tests the application robustness.
The tests 0019 has some data race making tests fail sometimes
There are deadcode that should be find a removed.
Add more tests to increase code coverage.
There is a missing wrapper for __strdup
that breaks compilation on gcc 5.4
Currently f2k can guess which is the client and which is the target mac addresses. It should be capable to guess also the client_ip
/target_ip
and client_port
/target_port
.
Parse the option to specify a kafka broker and topic to dump the discarded flow.
--kafka-discarder=<kafka-broker-ip>@<kafka-topic>
f2k
has an internal database with the IP addresses of the sensors (this is readed from the configuration file). When Netflow data is received, f2k
checks if the data comes from the IP address of a sensor stored on this database. The data received from unknowns IP is ignored.
It's necessary to implemente a way to dump this ignored data from unknown sensors to a Kafka topic so it can be processed.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.