Open source components and extensions for nProbe
Home Page: http://ntop.org
License: GNU General Public License v2.0
This repository contains open source components and extensions for nProbe available at http://www.ntop.org/products/netflow/nprobe/
Here you can find:
Enjoy!
If you want to learn more about nProbe please visit the User's Guide.
For more information about nProbe, please visit http://ntop.org.
Overview:
"[gtpv2Plugin.c:797] WARNING: Missing support for Bearer Context type 0x61" appears when processing GTPv2 traffic
Actual Result:
"[gtpv2Plugin.c:797] WARNING: Missing support for Bearer Context type 0x61" warning message
Expected Result:
No warning message
Build date&Hardware:
nProbe v.7.3.160118 (r4799) on Debian GNU/Linux 8.2 (jessie)
nProbe Command used:
/usr/local/bin/nprobe -n 192.168.10.198:9145 -i 0x61.pcap -u 5 -Q 0 -d 5 -t 10 -V 9 -o 10 -U 700
--tunnel
-cpu-affinity 1
--dump-bad-packets /home/genie/pcap/bad-pkt-eth3.pcap
--timestamp-format 1
--event-log /home/genie/nprobe-event-eth3.log
-T
"
%FIRST_SWITCHED %LAST_SWITCHED
%FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS
%IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS
%L7_PROTO %APPLICATION_ID
%UPSTREAM_TUNNEL_ID %DOWNSTREAM_TUNNEL_ID %UNTUNNELED_IPV4_SRC_ADDR %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_PROTOCOL
%DNS_QUERY %DNS_QUERY_ID %DNS_QUERY_TYPE %DNS_RET_CODE
%GTPV1_REQ_MSG_TYPE %GTPV1_RSP_MSG_TYPE %GTPV1_C2S_TEID_DATA %GTPV1_C2S_TEID_CTRL %GTPV1_S2C_TEID_DATA %GTPV1_S2C_TEID_CTRL %GTPV1_END_USER_IP %GTPV1_END_USER_IMSI %GTPV1_END_USER_MSISDN %GTPV1_END_USER_IMEI %GTPV1_APN_NAME %GTPV1_RAI_MCC %GTPV1_RAI_MNC %GTPV1_RAI_LAC %GTPV1_RAI_RAC %GTPV1_ULI_MCC %GTPV1_ULI_MNC %GTPV1_ULI_CELL_LAC %GTPV1_ULI_CELL_CI %GTPV1_ULI_SAC %GTPV1_RESPONSE_CAUSE %GTPV1_RAT_TYPE
%GTPV2_REQ_MSG_TYPE %GTPV2_RSP_MSG_TYPE %GTPV2_S5_S8_GTPC_TEID %GTPV2_C2S_S5_S8_GTPU_TEID %GTPV2_S2C_S5_S8_GTPU_TEID %GTPV2_C2S_S5_S8_GTPU_IP %GTPV2_S2C_S5_S8_GTPU_IP %GTPV2_END_USER_IMSI %GTPV2_END_USER_MSISDN %GTPV2_APN_NAME %GTPV2_ULI_MCC %GTPV2_ULI_MNC %GTPV2_ULI_CELL_TAC %GTPV2_ULI_CELL_ID %GTPV2_RESPONSE_CAUSE %GTPV2_RAT_TYPE %GTPV2_PDN_IP %GTPV2_END_USER_IMEI "
Hi,
I'm tring to monitor SIP and RTP traffic
For the following command
nprobe -i eth4 -i eth3 --elastic "sip;nprobe-%Y.%m.%d;http://localhost:9200/_bulk;" -T "%IPV4_SRC_ADDR %L4_SRC_PORT %IPV4_DST_ADDR %L4_DST_PORT %PROTOCOL %IN_BYTES %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %IN_PKTS %OUT_PKTS %IP_PROTOCOL_VERSION %APPLICATION_ID %L7_PROTO_NAME %ICMP_TYPE %SRC_IP_COUNTRY %DST_IP_COUNTRY %APPL_LATENCY_MS %HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE %SIP_CALL_ID SIP %SIP_CALLING_PARTY %SIP_CALLED_PARTY %SIP_RTP_CODECS %SIP_INVITE_TIME %SIP_TRYING_TIME %SIP_RINGING_TIME %SIP_INVITE_OK_TIME %SIP_INVITE_FAILURE_TIME SIP %SIP_BYE_TIME %SIP_BYE_OK_TIME %SIP_CANCEL_TIME %SIP_CANCEL_OK_TIME SIP %SIP_RTP_IPV4_SRC_ADDR %SIP_RTP_L4_SRC_PORT %SIP_RTP_IPV4_DST_ADDR %SIP_RTP_L4_DST_PORT %SIP_FAILURE_CODE %SIP_REASON_CAUSE %RTP_FIRST_SSRC %RTP_FIRST_TS %RTP_LAST_SSRC %RTP_LAST_TS %RTP_IN_JITTER %RTP_OUT_JITTER %RTP_IN_PKT_LOST %RTP_OUT_PKT_LOST %RTP_IN_PAYLOAD_TYPE %RTP_OUT_PAYLOAD_TYPE %RTP_IN_MAX_DELTA %RTP_OUT_MAX_DELTA %RTP_SIP_CALL_ID %RTP_SIP_CALL_ID %RTP_MOS %RTP_R_FACTOR %RTP_RTT"
I received warrning about discarding afew templates
21/Feb/2016 13:46:16 [exportPlugin.c:292] Using ElasticSearch for data dump [sip][nprobe-%Y.%m.%d][http://localhost:9200/_bulk] 21/Feb/2016 13:46:16 [nprobe.c:6793] Welcome to nProbe v.7.3.160221 for x86_64-unknown-linux-gnu 21/Feb/2016 13:46:16 [nprobe.c:6051] Using NetFlow Packet Payload Len: 1472 21/Feb/2016 13:46:16 [template.c:1438] WARNING: Geo-location requires --city-list to be specified: ignored SRC_IP_COUNTRY 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'SIP_FAILURE_CODE'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'RTP_FIRST_SSRC'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'RTP_LAST_SSRC'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'SIP_FAILURE_CODE'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'RTP_FIRST_SSRC'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'RTP_LAST_SSRC'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'SIP_RTP_IPV6_SRC_ADDR'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'SIP_RTP_IPV6_DST_ADDR'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'SIP_FAILURE_CODE'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'RTP_FIRST_SSRC'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'RTP_LAST_SSRC'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'SIP_RTP_IPV6_SRC_ADDR'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'SIP_RTP_IPV6_DST_ADDR'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'SIP_FAILURE_CODE'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'RTP_FIRST_SSRC'. Discarded. 21/Feb/2016 13:46:16 [template.c:1526] WARNING: Unable to locate template 'RTP_LAST_SSRC'. Discarded.
Why?
Is the nProbe documentation actual?
Or maybe I'm doing something wrong?
I'm currently working with nProbe PRO version with HTTP Plugin.
I was wondering how HTTP2.0 was processed by nProbe+HTTP Plugin. It is classified as HTTP, SSL or Unknown? which "protocol" is dumped in HTTP plugin output (https|https?).
Regards,
Zied
Hi,
For the time beeing. It's apper twice.
Welcome to nProbe v.7.3.160227 (r4881) for x86_64-unknown-linux-gnu
with native PF_RING acceleration.
Copyright 2002-16 ntop.org
Build OS: Debian GNU/Linux 8.2 (jessie)
SystemID: 3C0E6232xxxxx
Edition: nProbe Pro
License: xxxxxxxxxxxxxxxxxxxxxxxxxxx [valid license]
License Type: Permanent License
Maintenance: Until Sat Feb 25 15:54:36 2017 [364 days left]
My config nprobe.conf
-n=none
-i=eth1,eth5
-b=1
-s=128
-t=60
-d=30
-a=0
-e=1
-S=1:1
-g=/var/run/nprobe-eth1eth5.pid
--dont-nest-dump-dirs
--elastic=flows;nprobe-%Y.%m.%d;http://localhost:9200/_bulk;
--dump-stats=/var/log/nprobe/eth1eth5-0_flows_stats.txt
--timestamp-format=2
-T=%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %OUT_PKTS %L4_DST_PORT %L4_SRC_PORT %IN_BYTES %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %PROTOCOL %IN_SRC_MAC %OUT_DST_MAC %TCP_FLAGS %APPL_LATENCY_MS %SRC_IP_COUNTRY %DST_IP_COUNTRY %L7_PROTO %L7_PROTO_NAME %ICMP_TYPE %IP_PROTOCOL_VERSION %NUM_PKTS_UP_TO_128_BYTES %NUM_PKTS_128_TO_256_BYTES %NUM_PKTS_256_TO_512_BYTES %NUM_PKTS_512_TO_1024_BYTES %NUM_PKTS_1024_TO_1514_BYTES %NUM_PKTS_OVER_1514_BYTES %DNS_QUERY %DNS_QUERY_TYPE %DNS_RET_CODE %DNS_NUM_ANSWERS %DNS_TTL_ANSWER %DNS_RESPONSE %HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE %DOT1Q_SRC_VLAN %DOT1Q_DST_VLAN %SRC_IP_CITY %DST_IP_CITY
--json-labels=
-A=/usr/local/nprobe/GeoIPASNum.dat
--city-list=/usr/local/nprobe/GeoLiteCity.dat
I got an ERROR
27/Feb/2016 06:45:06 [nprobe.c:2683] ---------------------------------
27/Feb/2016 07:45:06 [nprobe.c:2684] Average traffic: [293.12 K pps][1.66 Gb/sec]
27/Feb/2016 07:45:06 [nprobe.c:2691] Current traffic: [292.29 K pps][1.66 Gb/sec]
27/Feb/2016 07:45:06 [nprobe.c:2697] Current flow export rate: [12948.0 flows/sec]
27/Feb/2016 07:45:06 [nprobe.c:2700] Flow drops: [export queue too long=0][too many flows=0]
27/Feb/2016 07:45:06 [nprobe.c:2704] Export Queue: 9636/512000 [1.9 %]
27/Feb/2016 07:45:06 [nprobe.c:2709] Flow Buckets: [active=419525][allocated=429160][toBeExported=9635]
27/Feb/2016 07:45:06 [cache.c:1224] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
27/Feb/2016 07:45:06 [cache.c:1722] LRUCacheUnit L7Cache [current_hash_size: 12288][max_cache_node_len: 4][mem_size: 2112.4 MB/2112.4 MB]
27/Feb/2016 07:45:06 [cache.c:1731] LRUCache L7Cache [find: 4280021 operations/12030.6 find/sec][cache miss 360916/100.0 %][add: 1561346 operations/5204.9 add/sec][tot: 12288][mem_size: 2112.4 MB]
27/Feb/2016 07:45:06 [nprobe.c:2551] Processed packets: 106108333 (max bucket search: 24)
27/Feb/2016 07:45:06 [nprobe.c:2534] Fragment queue length: 1547
27/Feb/2016 07:45:06 [nprobe.c:2557] WARNING: Your bucket search is too slow (24): expect drops
27/Feb/2016 07:45:06 [nprobe.c:2560] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
27/Feb/2016 07:45:06 [nprobe.c:2570] Flow drop stats: [0 bytes/0 pkts][0 flows]
27/Feb/2016 07:45:06 [nprobe.c:2575] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
27/Feb/2016 07:45:30 [util.c:1942] WARNING: flowBufferPrintf(HTTP_UA): INTERNAL ERROR [len: 1483][i: 2772]
27/Feb/2016 07:45:30 [util.c:1920] WARNING: INTERNAL ERROR on flowBufferPrintf() [len=4255][line_buffer_len=4095]
Flow data from a Juniper MX router is not being captured by nProbe running in Proxy mode. Nothing is passed to nTopng via ZMQ.
.pcap file of flow data hitting the nProbe server during the following nProbe run:
tcpdump -n -l -i eth4 -w cflow.pcap port 2055
(rename to cflow.pcap. Github keeps rejecting .zip files)
cflow.txt
root@uncsnbox:/etc/nprobe# nprobe -3 2055 -zmq=tcp://*:5556 -n none -b 2
04/Dec/2015 17:40:32 [nprobe.c:3130] Valid nProbe Pro license found
04/Dec/2015 17:40:32 [plugin.c:166] No plugins found in ./plugins
04/Dec/2015 17:40:32 [plugin.c:174] Loading 22 plugins [.so] from /usr/local/lib/nprobe/plugins
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin DHCP Protocol: missing license [/etc/nprobe.license.dhcp]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Diameter Protocol: missing license [/etc/nprobe.license.diameter]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin DNS Protocol: missing license [/etc/nprobe.license.dns]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Export Plugin: missing license [/etc/nprobe.license.export]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin FTP Protocol: missing license [/etc/nprobe.license.ftp]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin GTPv0 Signaling Protocol: missing license [/etc/nprobe.license.gtpv0]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin GTPv1 Signaling Protocol: missing license [/etc/nprobe.license.gtpv1]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin GTPv2 Signaling Protocol: missing license [/etc/nprobe.license.gtpv2]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin HTTP Protocol: missing license [/etc/nprobe.license.http]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin IMAP Protocol: missing license [/etc/nprobe.license.email]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Netflow-Lite Plugin: missing license [/etc/nprobe.license.nflite]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Oracle Protocol: missing license [/etc/nprobe.license.oracle]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin POP3 Protocol: missing license [/etc/nprobe.license.email]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin System process information: missing license [/etc/nprobe.license.process]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Radius Protocol: missing license [/etc/nprobe.license.radius]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin RTP Plugin: missing license [/etc/nprobe.license.voippro]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin S1AP Protocol: missing license [/etc/nprobe.license.S1AP]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin SIP Plugin: missing license [/etc/nprobe.license.voippro]
04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin SMTP Protocol: missing license [/etc/nprobe.license.email]
04/Dec/2015 17:40:32 [nprobe.c:4488] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
04/Dec/2015 17:40:32 [nprobe.c:4491] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
04/Dec/2015 17:40:32 [nprobe.c:4552] Welcome to nProbe Pro v.7.2.151202 (
04/Dec/2015 17:40:32 [nprobe.c:4562] Running on Ubuntu 14.04.2 LTS
04/Dec/2015 17:40:32 [nprobe.c:4573] [LICENSE] nProbe SystemId: FA623D157104A1D2
04/Dec/2015 17:40:32 [nprobe.c:4620] Tracing enabled
04/Dec/2015 17:40:32 [bgpPlugin.c:375] BGP plugin is disabled (--bgp-port has not been specified)
04/Dec/2015 17:40:32 [dbPlugin.c:49] Initializing DB plugin
04/Dec/2015 17:40:32 [mysqlPlugin.c:111] Initialized MySQL plugin
04/Dec/2015 17:40:32 [plugin.c:248] 3 plugin(s) loaded [3 delete][2 packet].
04/Dec/2015 17:40:32 [nprobe.c:6526] Welcome to nprobe v.7.2.151202 for x86_64-unknown-linux-gnu
04/Dec/2015 17:40:32 [nprobe.c:5752] Compiling flow templates...
04/Dec/2015 17:40:32 [plugin.c:851] Scanning plugin BGP Update Listener [bgp]
04/Dec/2015 17:40:32 [plugin.c:851] Scanning plugin MySQL DB [db]
04/Dec/2015 17:40:32 [plugin.c:851] Scanning plugin MySQL Plugin [mysql]
04/Dec/2015 17:40:32 [plugin.c:1000] 0 plugin(s) enabled
04/Dec/2015 17:40:32 [nprobe.c:6203] Non IPv4/v6 traffic is discarded according to the template
04/Dec/2015 17:40:32 [util.c:287] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
04/Dec/2015 17:40:32 [util.c:296] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
04/Dec/2015 17:40:32 [nprobe.c:6698] IPv6 traffic will NOT be exported/accounted by this probe
04/Dec/2015 17:40:32 [nprobe.c:6699] due to configuration options (e.g. use NetFlow v9)
04/Dec/2015 17:40:32 [nprobe.c:6702] The flows hash has 131072 buckets
04/Dec/2015 17:40:32 [nprobe.c:6704] Flows older than 120 seconds will be exported
04/Dec/2015 17:40:32 [nprobe.c:6707] Flows inactive for at least 30 seconds will be exported
04/Dec/2015 17:40:32 [nprobe.c:6710] Expired flows will not be queued for more than 30 seconds
04/Dec/2015 17:40:32 [nprobe.c:6717] Exported flows with engineType 0 and engineId 112
04/Dec/2015 17:40:32 [nprobe.c:6739] TCP TOS will be ignored and set to 0.
04/Dec/2015 17:40:32 [nprobe.c:6757] After 1 flow packets are sent, we'll delay at least 1 ms
04/Dec/2015 17:40:32 [nprobe.c:6777] Flows will be emitted in NetFlow 5 format
04/Dec/2015 17:40:32 [nprobe.c:6807] Flow input interface index is set to 0
04/Dec/2015 17:40:32 [nprobe.c:6813] Flow output interface index is set to 0
04/Dec/2015 17:40:32 [util.c:2892] WARNING: Don't dropping privileges (required by NetFilter)
04/Dec/2015 17:40:32 [plugin.c:813] Disabling plugin BGP Update Listener (no template is using it)
04/Dec/2015 17:40:32 [plugin.c:813] Disabling plugin MySQL DB (no template is using it)
04/Dec/2015 17:40:32 [plugin.c:813] Disabling plugin MySQL Plugin (no template is using it)
04/Dec/2015 17:40:32 [collect.c:86] Created UDP sockets
04/Dec/2015 17:40:32 [collect.c:90] Created a SCTP socket (53)
04/Dec/2015 17:40:32 [collect.c:145] Flow collector listening on port 2055 (IPv4/v6)
04/Dec/2015 17:40:32 [nprobe.c:6947] Starting 1 packet fetch thread(s)
04/Dec/2015 17:40:32 [nprobe.c:7035] nProbe started successfully
04/Dec/2015 17:40:32 [engine.c:3210] Starting bucket dequeue thread
04/Dec/2015 17:43:25 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
04/Dec/2015 17:43:25 [nprobe.c:386] Received shutdown request... [signal: 2]
04/Dec/2015 17:43:26 [nprobe.c:4716] nProbe is shutting down...
04/Dec/2015 17:43:26 [nprobe.c:4752] Exporting pending buckets...
04/Dec/2015 17:43:26 [nprobe.c:4773] Pending buckets have been exported...
04/Dec/2015 17:43:28 [engine.c:3293] Export thread terminated [exportQueue=0]
04/Dec/2015 17:43:28 [nprobe.c:4839] Flushing queued flows...
04/Dec/2015 17:43:28 [nprobe.c:4842] Freeing memory...
04/Dec/2015 17:43:28 [plugin.c:277] Terminating plugins.
04/Dec/2015 17:43:28 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
04/Dec/2015 17:43:28 [nprobe.c:4934] Still allocated 0 hash buckets
04/Dec/2015 17:43:28 [nprobe.c:2457] Processed packets: 0 (max bucket search: 0)
04/Dec/2015 17:43:28 [nprobe.c:2440] Fragment queue length: 0
04/Dec/2015 17:43:28 [nprobe.c:2466] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
04/Dec/2015 17:43:28 [nprobe.c:2473] Flow collection: [collected pkts: 0][processed flows: 0]
04/Dec/2015 17:43:28 [nprobe.c:2476] Flow drop stats: [0 bytes/0 pkts][0 flows]
04/Dec/2015 17:43:28 [nprobe.c:2481] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
04/Dec/2015 17:43:28 [nprobe.c:4947] Cleaning globals
04/Dec/2015 17:43:28 [nprobe.c:4967] nProbe terminated.
Overview:
Customers would like to know Diameter Cancel-Location Request related info. Therefore, two fields "diameter.Cancellation-Type" and "diameter.CLR-Flags" need to be exported.
Actual Result:
Currently there is no template of the related fields.
Expected Result:
New templates %DIAMETER_CLR_CANCEL_TYPE and %DIAMETER_CLR_FLAGS available for diameter.Cancellation-Type and diameter.CLR-Flags fields.
e.g. In the pcap packets attached, the %DIAMETER_CLR_CANCEL_TYPE=2 and %DIAMETER_CLR_FLAGS=1
Build Date&Hardware:
nProbe v.7.3.160315 (r4939) on Ubuntu 14.04.3 LTS
Additional Info:
diameter_CLR.pcap.zip
Hello,
I want to test nprobe stable on CentOS6 (v.7.2.151211) and I have an issue with nprobe and L2TP tunnelled traffic. Here is the command I launch :
[root@netflow-linux ~]# nprobe -i eth1 -d 60 -P
[capture_nprobe.zip](https://github.com/ntop/nProbe/files/65629/capture_nprobe.zip)
/tmp/flows -D t -I sfr -T "%IN_BYTES %IN_PKTS %L4_SRC_PORT %PROTOCOL %IPV4_SRC_ADDR %L4_DST_PORT %IPV4_DST_ADDR %IPV6_SRC_ADDR %IPV6_DST_MASK %UPSTREAM_TUNNEL_ID %DOWNSTREAM_TUNNEL_ID" -V 9 --smart-udp-frags -N 0 --tunnel
I'd expect to get records like
122|1|53|17|IP_IN_TUNNEL|13217|IP_IN_TUNNEL|::|0|000054B5|0000B5AB|
117|2|443|6|IP_IN_TUNNEL|53820|IP_IN_TUNNEL|::|0|00006304|0000BB56|
I get some of them, but most of my records are not correctly decapsulated and I usually get records like that :
52|1|30753|17|L2TP_IP|49752|L2TP_IP|::|0|00000000|00000000|
52|1|4560|17|L2TP_IP|34232|L2TP_IP|::|0|00000000|00000000|
As you can see, L4_SRC_PORT and L4_DST_PORT are correctly decapsulated. However, I neither get the tunneled IP address or the tunnel informations (I obfuscated IP informations, replacing them with IP_IN_TUNNEL and L2TP_IP). ~75% of flows are concerned.
I am pretty sure the problem comes from the decapsulation and it's not a false positive as if it was, src port and dest port would be 1701.
When I try to use it in debug mode I get a segfault (which I don't get without the --tunnel option). :
[root@netflow-linux ~]# nprobe -i eth1 -d 60 -P /tmp/flows -D t -I sfr -T "%IN_BYTES %IN_PKTS %L4_SRC_PORT %PROTOCOL %IPV4_SRC_ADDR %L4_DST_PORT %IPV4_DST_ADDR %IPV6_SRC_ADDR %IPV6_DST_MASK %UPSTREAM_TUNNEL_ID %DOWNSTREAM_TUNNEL_ID %UNTUNNELED_IPV4_SRC_ADDR" -V 9 --smart-udp-frags -N 0 --debug --tunnel
17/Dec/2015 16:19:38 [nprobe.c:3114] ERROR: Invalid nProbe license (/etc/nprobe.license) [Missing license file]
17/Dec/2015 16:19:38 [nprobe.c:3121] ERROR: *****************************************************
17/Dec/2015 16:19:38 [nprobe.c:3122] ERROR: ** **
17/Dec/2015 16:19:38 [nprobe.c:3123] ERROR: ** Switching to DEMO MODE (missing valid license) **
17/Dec/2015 16:19:38 [nprobe.c:3124] ERROR: ** **
17/Dec/2015 16:19:38 [nprobe.c:3125] ERROR: ** Create your nProbe license at **
17/Dec/2015 16:19:38 [nprobe.c:3126] ERROR: ** http://www.nmon.net/mklicense/ **
17/Dec/2015 16:19:38 [nprobe.c:3127] ERROR: ** **
17/Dec/2015 16:19:38 [nprobe.c:3128] ERROR: *****************************************************
17/Dec/2015 16:19:38 [nprobe.c:6508] ERROR: ***************************************************************
17/Dec/2015 16:19:38 [nprobe.c:6509] ERROR: * NOTE: This is a DEMO version limited to 25000 flows export. *
17/Dec/2015 16:19:38 [nprobe.c:6510] ERROR: ***************************************************************
17/Dec/2015 16:19:38 [plugin.c:166] No plugins found in ./plugins
17/Dec/2015 16:19:38 [plugin.c:174] Loading 22 plugins [.so] from /usr/local/lib/nprobe/plugins
datagramSourceIP 0.0.0.0
datagramSize 48
unixSecondsUTC 1450365578
datagramVersion 5
agentSubId 0
agent 192.168.1.1
packetSequenceNo 1084445
sysUpTime 2429093100
samplesInPacket 4
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 187645
sourceId 0:1
counterBlock_tag 2176:0
skipping unknown counters_sample_element: 2176:0 len=0
counterBlock_tag 568615:598
skipping unknown counters_sample_element: 568615:598 len=0
endSample ----------------------
unexpected end of datagram after sample 1 of 4
datagramSourceIP 0.0.0.0
datagramSize 48
unixSecondsUTC 1450365578
datagramVersion 5
agentSubId 0
agent 192.168.1.1
packetSequenceNo 1084446
sysUpTime 2429093100
samplesInPacket 10
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 11443
sourceId 0:2
meanSkipCount 50
samplePool 8912896
dropEvents 0
inputPort multiple 181563990
outputPort 0
flowBlock_tag 0:0
skipping unknown flow_sample_element: 0:0 len=-2147483648
Segmentation fault
The output without --debug :
[root@netflow-linux nprobe]# nprobe -i eth1 -d 60 -P /tmp/flows -D t -T "%IN_BYTES %IN_PKTS %L4_SRC_PORT %PROTOCOL %IPV4_SRC_ADDR %L4_DST_PORT %IPV4_DST_ADDR %IPV6_SRC_ADDR %IPV6_DST_MASK %UPSTREAM_TUNNEL_ID %DOWNSTREAM_TUNNEL_ID" -V 9 --smart-udp-frags -N 0 --tunnel
17/Dec/2015 18:36:29 [nprobe.c:3114] ERROR: Invalid nProbe license (/etc/nprobe.license) [Missing license file]
17/Dec/2015 18:36:29 [nprobe.c:3121] ERROR: *****************************************************
17/Dec/2015 18:36:29 [nprobe.c:3122] ERROR: ** **
17/Dec/2015 18:36:29 [nprobe.c:3123] ERROR: ** Switching to DEMO MODE (missing valid license) **
17/Dec/2015 18:36:29 [nprobe.c:3124] ERROR: ** **
17/Dec/2015 18:36:29 [nprobe.c:3125] ERROR: ** Create your nProbe license at **
17/Dec/2015 18:36:29 [nprobe.c:3126] ERROR: ** http://www.nmon.net/mklicense/ **
17/Dec/2015 18:36:29 [nprobe.c:3127] ERROR: ** **
17/Dec/2015 18:36:29 [nprobe.c:3128] ERROR: *****************************************************
17/Dec/2015 18:36:29 [nprobe.c:6508] ERROR: ***************************************************************
17/Dec/2015 18:36:29 [nprobe.c:6509] ERROR: * NOTE: This is a DEMO version limited to 25000 flows export. *
17/Dec/2015 18:36:29 [nprobe.c:6510] ERROR: ***************************************************************
17/Dec/2015 18:36:29 [plugin.c:166] No plugins found in ./plugins
17/Dec/2015 18:36:29 [plugin.c:174] Loading 22 plugins [.so] from /usr/local/lib/nprobe/plugins
17/Dec/2015 18:36:29 [nprobe.c:4488] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
17/Dec/2015 18:36:29 [nprobe.c:4491] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
17/Dec/2015 18:36:29 [nprobe.c:4552] Welcome to nProbe Pro v.7.2.151211 ($Revision: 4471 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration
17/Dec/2015 18:36:29 [nprobe.c:4562] Running on CentOS release 6.6 (Final)
17/Dec/2015 18:36:29 [nprobe.c:4573] [LICENSE] nProbe SystemId: 76A0E91411B1B8A2
17/Dec/2015 18:36:29 [nprobe.c:4653] Dumping flow files every 60 sec into directory /tmp/flows
17/Dec/2015 18:36:29 [nprobe.c:4658] WARNING: -n parameter is missing. 127.0.0.1:2055 will be used.
17/Dec/2015 18:36:29 [dbPlugin.c:49] Initializing DB plugin
17/Dec/2015 18:36:29 [exportPlugin.c:239] Initializing Export plugin
17/Dec/2015 18:36:29 [nprobe.c:6526] Welcome to nprobe v.7.2.151211 for x86_64-unknown-linux-gnu
17/Dec/2015 18:36:29 [nprobe.c:5789] Using NetFlow Packet Payload Len: 1472
17/Dec/2015 18:36:29 [plugin.c:1000] 0 plugin(s) enabled
17/Dec/2015 18:36:29 [nprobe.c:6183] Each flow is 54 bytes long
17/Dec/2015 18:36:29 [nprobe.c:6184] The # packets per flow has been set to 26
17/Dec/2015 18:36:29 [nprobe.c:6203] Non IPv4/v6 traffic is discarded according to the template
17/Dec/2015 18:36:29 [util.c:287] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
17/Dec/2015 18:36:29 [util.c:296] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
17/Dec/2015 18:36:29 [nprobe.c:5121] Using packet capture length 128
17/Dec/2015 18:36:29 [pro/pf_ring.c:358] Using PF_RING in-kernel accelerated packet parsing
17/Dec/2015 18:36:29 [pro/pf_ring.c:362] Dumping traffic statistics on /proc/net/pf_ring/stats/28920-eth1.61
17/Dec/2015 18:36:29 [nprobe.c:6834] Smart fragment rebuild enabled (no fragments are rebuilt)
17/Dec/2015 18:36:29 [nprobe.c:6837] Enabled tunnel decoding (e.g. IPSEC/GTP)
17/Dec/2015 18:36:29 [util.c:2919] nProbe changed user to 'nobody'
17/Dec/2015 18:36:29 [nprobe.c:7035] nProbe started successfully
17/Dec/2015 18:36:29 [pro/pf_ring.c:172] [PF_RING] Reading packets in 0 copy mode
17/Dec/2015 18:37:03 [engine.c:2163] WARNING: Too many (524288) active flows [threadId=0][limit=524288] (see -M)
^C17/Dec/2015 18:37:28 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
17/Dec/2015 18:37:28 [nprobe.c:386] Received shutdown request... [signal: 2]
17/Dec/2015 18:37:28 [pro/pf_ring.c:300] Terminated PF_RING packet processing
17/Dec/2015 18:37:28 [nprobe.c:4722] Waiting for PF_RING termination
17/Dec/2015 18:37:28 [nprobe.c:4731] PF_RING terminated
17/Dec/2015 18:37:28 [engine.c:2673] About to flush hash (threadId 0)
17/Dec/2015 18:37:29 [engine.c:2675] Completed hash walk (thread 0)
17/Dec/2015 18:37:29 [export.c:380] ERROR: ***************************************************************************
17/Dec/2015 18:37:29 [export.c:381] ERROR: * NOTE: You have reached the max demo 25000 flows export: no more exports *
17/Dec/2015 18:37:29 [export.c:383] ERROR: * NOTE: no additional flows will be exported by this nProbe instance *
17/Dec/2015 18:37:29 [export.c:384] ERROR: ***************************************************************************
17/Dec/2015 18:37:32 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
17/Dec/2015 18:37:32 [nprobe.c:2457] Processed packets: 4415181 (max bucket search: 16)
17/Dec/2015 18:37:32 [nprobe.c:2440] Fragment queue length: 0
17/Dec/2015 18:37:32 [nprobe.c:2463] WARNING: Your bucket search is too slow (16): expect drops
17/Dec/2015 18:37:32 [nprobe.c:2466] Flow export stats: [1055151133 bytes/1728265 pkts][25001 flows/1150 pkts sent]
17/Dec/2015 18:37:32 [nprobe.c:2476] Flow drop stats: [8300751 bytes/25458 pkts][12288 flows]
17/Dec/2015 18:37:32 [nprobe.c:2481] Total flow stats: [1063451884 bytes/1753723 pkts][37289 flows/1150 pkts sent]
When I compare with what I get in a pcap, I can see that in my pcap file I almost don't get any packet
Is there a performance issue (it doesn't seem so, CPU stays low) ? Is there a fix somewhere, or did I miss something ?
In attachment, a pcap file which corresponds to the traffic.
Thank you very much,
Regards,
Grégoire
nProbe is changing some bytes at the end of the packet as safety boundary
for packet processing, this is causing issues on cards (i.e. Accolade) working
with segments where there is no padding (or just some alignment) at the end
of each packet.
In order to reproduce the issue, send packets with random size to nProbe
using an Accolade card for as packet capture card.
Temporary fix in pro/pf_ring.c:
196 if (strstr(readOnlyGlobals.captureDev, "anic") != NULL)
197 allocate_buffer = 1;
I installed nProbe from http://packages.ntop.org/Ubiquity/ and when I run:
nprobe --zmq "tcp://10.0.1.1:5556" -i eth0 -n none
I get:
[nprobe.c:4495] WARNING: Unrecognized option '--zmq'
Is this version of nProbe not compiled to support the zmq interface? I have ntopng running on a dedicated machine and want to be able to interface with nProbe that is running on the EdgeRouter.
The following issues were found after VoLTE PCAP processed by nProbe
The found issues are:
• Pcaps with multiple VoLTE calls does not show any stats at all. Only basic IP stats
• IPv6 address in the GTP tunnel cannot be shown.
• No MOS is calculated (license issue).
Hi
I'm trying to push nprobe into a mysql db, but it's not being inserted into the db. The ntopng historical is.
This is the nprobe conf file
-n=10.164.0.1:6343
-n=10.165.0.1:6343
-n=10.166.0.253:6343
-i=none
-t=60
-d=60
-a=0
-e=1
-B=10
-w=128000
-z=0
-S=1:1
-E=0:0
-g=/var/run/nprobe-file.pid
-3=6343
--zmq=tcp://127.0.0.1:5556
--vlanid-as-iface-idx=none
-T=%IN_BYTES %IN_PKTS %PROTOCOL %SRC_TOS %TCP_FLAGS %L4_SRC_PORT %IPV4_SRC_ADDR %INPUT_SNMP %L4_DST_PORT %IPV4_DST_ADDR %OUTPUT_SNMP %LAST_SWITCHED %FIRST_SWITCHED
-V=9
--mysql=127.0.0.1:nrpobe:flows:nprobe:nprobe
--dump-stats=/var/log/nprobe/file-0_flows_stats.txt
Hi,
Overview:
I would like to use nProbe only as netflow collector on port 9996 and send the collected flows to ntopng. Not using "-i" nprobe fails with "Segmentation Fault".
Are you able to reproduce my issue?
(There is also a second issue if I choose to run nprobe with "-i none" (complete command: $ nprobe --zmq "tcp://127.0.0.1:5556" -b 2 --collector-port 9996 -i none) it will start and run but do sniffing on eth0 for packets and not listening on port 9996 for netflow. should I open a second Issue for that?)
Actual Result:
$ nprobe --zmq "tcp://127.0.0.1:5556" -b 2 --collector-port 9996
30/Mar/2016 15:10:26 [nprobe.c:3273] ERROR: Invalid nProbe license (/etc/nprobe.license) [Missing license file]
30/Mar/2016 15:10:26 [nprobe.c:3280] ERROR: *****************************************************
30/Mar/2016 15:10:26 [nprobe.c:3281] ERROR: ** **
30/Mar/2016 15:10:26 [nprobe.c:3282] ERROR: ** Switching to DEMO MODE (missing valid license) **
30/Mar/2016 15:10:26 [nprobe.c:3283] ERROR: ** **
30/Mar/2016 15:10:26 [nprobe.c:3284] ERROR: ** Purchase your nProbe license at **
30/Mar/2016 15:10:26 [nprobe.c:3285] ERROR: ** https://shop.ntop.org/ **
30/Mar/2016 15:10:26 [nprobe.c:3286] ERROR: ** **
30/Mar/2016 15:10:26 [nprobe.c:3287] ERROR: *****************************************************
30/Mar/2016 15:10:26 [nprobe.c:6845] ERROR: ***************************************************************
30/Mar/2016 15:10:26 [nprobe.c:6846] ERROR: * NOTE: This is a DEMO version limited to 25000 flows export. *
30/Mar/2016 15:10:26 [nprobe.c:6847] ERROR: ***************************************************************
30/Mar/2016 15:10:26 [plugin.c:168] No plugins found in ./plugins
30/Mar/2016 15:10:26 [plugin.c:176] Loading 24 plugins [.so] from /usr/local/lib/nprobe/plugins
30/Mar/2016 15:10:26 [nprobe.c:4705] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
30/Mar/2016 15:10:26 [nprobe.c:4708] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
30/Mar/2016 15:10:26 [nprobe.c:4802] Welcome to nProbe Pro v.7.3.160329 (
30/Mar/2016 15:10:26 [nprobe.c:4812] Running on CentOS Linux release 7.1.1503 (Core)
30/Mar/2016 15:10:26 [nprobe.c:4823] [LICENSE] nProbe SystemId: 688A0F0A5905A17E
30/Mar/2016 15:10:26 [nprobe.c:4874] Tracing enabled
30/Mar/2016 15:10:26 [nprobe.c:4912] WARNING: -n parameter is missing. 127.0.0.1:2055 will be used.
30/Mar/2016 15:10:26 [nprobe.c:3102] Exporting flows towards 127.0.0.1:2055 using UDP
30/Mar/2016 15:10:26 [bgpPlugin.c:375] BGP plugin is disabled (--bgp-port has not been specified)
30/Mar/2016 15:10:26 [dbPlugin.c:49] Initializing DB plugin
30/Mar/2016 15:10:26 [dhcpPlugin.c:305] Initialized DHCP plugin
30/Mar/2016 15:10:26 [diameterPlugin.c:98] Initialized Diameter plugin
30/Mar/2016 15:10:26 [dnsPlugin.c:109] Initialized DNS plugin
30/Mar/2016 15:10:26 [exportPlugin.c:253] Initializing Export plugin
30/Mar/2016 15:10:26 [ftpPlugin.c:80] Initialized FTP plugin
30/Mar/2016 15:10:26 [gtpv0Plugin.c:92] Initialized GTPv0 plugin
30/Mar/2016 15:10:26 [gtpv1Plugin.c:122] Initialized GTPv1 plugin
30/Mar/2016 15:10:26 [gtpv2Plugin.c:136] Initialized GTPv2 plugin
30/Mar/2016 15:10:26 [httpPlugin.c:498] HTTP log files will be dumped each 60 seconds or each 10000 lines
30/Mar/2016 15:10:26 [httpPlugin.c:505] Initialized HTTP plugin
30/Mar/2016 15:10:26 [imapPlugin.c:127] Initialized IMAP plugin
30/Mar/2016 15:10:26 [mysqlPlugin.c:111] Initialized MySQL plugin
30/Mar/2016 15:10:26 [netbiosPlugin.c:79] Initialized NETBIOS plugin
30/Mar/2016 15:10:26 [nflitePlugin.c:914] [NFLite] Initialized NetFlow-Lite plugin
30/Mar/2016 15:10:26 [oraclePlugin.c:172] Initialized Oracle plugin
30/Mar/2016 15:10:26 [popPlugin.c:118] Initialized POP plugin
30/Mar/2016 15:10:26 [processPlugin.c:394] Initialized process plugin
30/Mar/2016 15:10:26 [radiusPlugin.c:123] Initialized Radius plugin
30/Mar/2016 15:10:26 [rtpPlugin.c:168] Initializing RTP plugin [argc: 7]
30/Mar/2016 15:10:26 [s1apPlugin.c:1407] Initialized S1AP plugin
30/Mar/2016 15:10:26 [sipPlugin.c:258] Initialized SIP plugin
30/Mar/2016 15:10:26 [sipPlugin.c:287] Initialized SIP plugin
30/Mar/2016 15:10:26 [smtpPlugin.c:119] Initialized SMTP plugin
30/Mar/2016 15:10:26 [ssdpPlugin.c:48] Initialized SSDP plugin
30/Mar/2016 15:10:26 [plugin.c:250] 24 plugin(s) loaded [22 delete][21 packet].
30/Mar/2016 15:10:26 [nprobe.c:6863] Welcome to nProbe v.7.3.160329 for x86_64-unknown-linux-gnu
30/Mar/2016 15:10:26 [nprobe.c:5991] Compiling flow templates...
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin BGP Update Listener [bgp]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin MySQL DB [db]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin DHCP Protocol [dhcp]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin Diameter Protocol [diameter]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin DNS/LLMNR Protocol [dns]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin Export Plugin [export]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin FTP Protocol [ftp]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin GTPv0 Signaling Protocol [gtpv0]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin GTPv1 Signaling Protocol [gtpv1]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin GTPv2 Signaling Protocol [gtpv2]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin HTTP Protocol [http]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin IMAP Protocol [imap]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin MySQL Plugin [mysql]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin NETBIOS Protocol [netbios]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin Netflow-Lite Plugin [nflite]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin Oracle Protocol [oracle]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin POP3 Protocol [pop3]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin System process information [process]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin Radius Protocol [radius]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin RTP Plugin [rtp]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin S1AP Protocol [S1AP]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin SIP Plugin [sip]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin SMTP Protocol [smtp]
30/Mar/2016 15:10:26 [plugin.c:872] Scanning plugin SSDP Protocol [ssdp]
30/Mar/2016 15:10:26 [plugin.c:1029] 0 plugin(s) enabled
30/Mar/2016 15:10:26 [nprobe.c:6446] Non IPv4/v6 traffic is discarded according to the template
30/Mar/2016 15:10:26 [util.c:434] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
30/Mar/2016 15:10:26 [util.c:445] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
30/Mar/2016 15:10:26 [nprobe.c:7033] IPv6 traffic will NOT be exported/accounted by this probe
30/Mar/2016 15:10:26 [nprobe.c:7034] due to configuration options (e.g. use NetFlow v9)
30/Mar/2016 15:10:26 [nprobe.c:7037] The flows hash has 131072 buckets
30/Mar/2016 15:10:26 [nprobe.c:7039] Flows older than 120 seconds will be exported
30/Mar/2016 15:10:26 [nprobe.c:7042] Flows inactive for at least 30 seconds will be exported
30/Mar/2016 15:10:26 [nprobe.c:7045] Expired flows will not be queued for more than 30 seconds
30/Mar/2016 15:10:26 [nprobe.c:7052] Exported flows with engineType 0 and engineId 66
30/Mar/2016 15:10:26 [nprobe.c:7074] TCP TOS will be ignored and set to 0.
30/Mar/2016 15:10:26 [nprobe.c:7092] After 1 flow packets are sent, we'll delay at least 1 ms
30/Mar/2016 15:10:26 [nprobe.c:7112] Flows will be emitted in NetFlow 5 format
30/Mar/2016 15:10:26 [nprobe.c:7142] Flow input interface index is set to 0
30/Mar/2016 15:10:26 [nprobe.c:7148] Flow output interface index is set to 0
30/Mar/2016 15:10:26 [util.c:4069] Initializing ZMQ as server
30/Mar/2016 15:10:26 [util.c:4112] Succesfully created ZMQ endpoint tcp://127.0.0.1:5556
30/Mar/2016 15:10:26 [util.c:3162] nProbe changed user to 'nobody'
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin BGP Update Listener (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin MySQL DB (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin DHCP Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin Diameter Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin DNS/LLMNR Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin Export Plugin (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin FTP Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin GTPv0 Signaling Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin GTPv1 Signaling Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin GTPv2 Signaling Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin HTTP Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin IMAP Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin MySQL Plugin (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin NETBIOS Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin Netflow-Lite Plugin (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin Oracle Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin POP3 Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin System process information (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin Radius Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin RTP Plugin (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin S1AP Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin SIP Plugin (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin SMTP Protocol (no template is using it)
30/Mar/2016 15:10:26 [plugin.c:834] Disabling plugin SSDP Protocol (no template is using it)
30/Mar/2016 15:10:26 [collect.c:86] Created UDP sockets
30/Mar/2016 15:10:26 [collect.c:145] Flow collector listening on port 9996 (IPv4/v6)
30/Mar/2016 15:10:26 [nprobe.c:7275] Starting 1 packet fetch thread(s)
30/Mar/2016 15:10:26 [nprobe.c:7361] nProbe started successfully
30/Mar/2016 15:10:26 [engine.c:3104] Starting bucket dequeue thread
Segmentation fault
Expected Result:
No segmentation fault and the process is up fine
**
Build & OS:**
$ nprobe --version
Welcome to nProbe v.7.3.160329 (r4985) for x86_64-unknown-linux-gnu
with native PF_RING acceleration.
Build OS: CentOS Linux release 7.1.1503 (Core)
License: Invalid nProbe license (/etc/nprobe.license) [Missing license file]
nProbe Command used:
$ nprobe --zmq "tcp://127.0.0.1:5556" -b 2 --collector-port 9996
When using nProbe with Elastic search plugin, the nprobe log file ([email protected]) is becoming very big, the nProbe is writing to the log file each flow that is being send to elastic.
We are using nprobe version 7.3.151129
i have attached an example of the log file in the following link.
https://dl.dropboxusercontent.com/u/27973370/nprobe-eth0%400.log
Kibana examples container - ignore this issue.
I installed nProbe from http://packages.ntop.org/Ubiquity/ and when I run:
nprobe --zmq "tcp://10.0.1.1:5556" -i eth0 -n none
I get:
[nprobe.c:4495] WARNING: Unrecognized option '--zmq'
Is this version of nProbe not compiled to support the zmq interface? I have ntopng running on a dedicated machine and want to be able to interface with nProbe that is running on the EdgeRouter.
Is there any posibility to store GeoPoint data of SRC and DST IP Location?
My nprobe config file is following
-n=none
-i=eth1,eth5
-b=1
-s=128
-t=60
-d=30
-a=0
-e=1
-S=1:1
-g=/var/run/nprobe-eth1eth5.pid
--dont-nest-dump-dirs
--elastic=flows;nprobe-%Y.%m.%d;http://localhost:9200/_bulk;
--dump-stats=/var/log/nprobe/eth1eth5-0_flows_stats.txt
--timestamp-format=2
-T=%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %OUT_PKTS %L4_DST_PORT %L4_SRC_PORT %IN_BYTES %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %PROTOCOL %IN_SRC_MAC %OUT_DST_MAC %TCP_FLAGS %APPL_LATENCY_MS %SRC_IP_COUNTRY %DST_IP_COUNTRY %L7_PROTO %L7_PROTO_NAME %ICMP_TYPE %IP_PROTOCOL_VERSION %NUM_PKTS_UP_TO_128_BYTES %NUM_PKTS_128_TO_256_BYTES %NUM_PKTS_256_TO_512_BYTES %NUM_PKTS_512_TO_1024_BYTES %NUM_PKTS_1024_TO_1514_BYTES %NUM_PKTS_OVER_1514_BYTES %DNS_QUERY %DNS_QUERY_TYPE %DNS_RET_CODE %DNS_NUM_ANSWERS %DNS_TTL_ANSWER %DNS_RESPONSE %HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE %DOT1Q_SRC_VLAN %DOT1Q_DST_VLAN %SRC_IP_CITY %DST_IP_CITY
--json-labels=
-A=/usr/local/nprobe/GeoIPASNum.dat
--city-list=/usr/local/nprobe/GeoLiteCity.dat
#--zmq=tcp://*:5556
With interaction with ntopng there is possible to display location on Geo Map but there is no data in ES.
In case of storing data directly from ntopng to ES there was such possibility.
I do not find any Tamplates for this.
How to do it in case of nprobe?
New arg to report traffic in or out on an interface properly in v9 using IN_ and DIRECTION values
Consider --host
For ethernet traffic inbound on the NIC:
For ethernet traffic outbound to the NIC:
Hi,
I have noticed that there is no SSL_CERTIFICATE data stored to DB like in case of ntopng.
Is there any plans to add this feature to HTTP module?
when i run nprobe with the following options:
nprobe --zmq "tcp://*:5556" -i none -n none --collector-port 6343
i am able to collect sflow and provide it to ntopng but it seems like the L2 headers containing the src/dst mac addresses in the flow record are not processed
that is based on my limited understanding of using strace and looking at the zmq code here
Overview:
In recent nProbe builds, one process dealing with Diameter traffic experiences segmentation fault right after it initiates.
Actual Result:
16/Feb/2016 18:04:33 [nprobe.c:3199] Valid nProbe Pro license found
16/Feb/2016 18:04:33 [plugin.c:169] No plugins found in ./plugins
16/Feb/2016 18:04:33 [plugin.c:177] Loading 22 plugins [.so] from /usr/local/lib/nprobe/plugins
16/Feb/2016 18:04:33 [plugin.c:750] Unable to enable plugin DHCP Protocol: missing license [/etc/nprobe.license.dhcp]
16/Feb/2016 18:04:33 [plugin.c:750] Unable to enable plugin Export Plugin: missing license [/etc/nprobe.license.export]
16/Feb/2016 18:04:33 [plugin.c:750] Unable to enable plugin FTP Protocol: missing license [/etc/nprobe.license.ftp]
16/Feb/2016 18:04:33 [plugin.c:750] Unable to enable plugin IMAP Protocol: missing license [/etc/nprobe.license.email]
16/Feb/2016 18:04:33 [plugin.c:750] Unable to enable plugin Oracle Protocol: missing license [/etc/nprobe.license.oracle]
16/Feb/2016 18:04:33 [plugin.c:750] Unable to enable plugin POP3 Protocol: missing license [/etc/nprobe.license.email]
16/Feb/2016 18:04:33 [plugin.c:750] Unable to enable plugin S1AP Protocol: missing license [/etc/nprobe.license.S1AP]
16/Feb/2016 18:04:33 [plugin.c:750] Unable to enable plugin SMTP Protocol: missing license [/etc/nprobe.license.email]
16/Feb/2016 18:04:33 [nprobe.c:3642] WARNING: Ignored -c: it must be specified after -L
16/Feb/2016 18:04:33 [nprobe.c:3781] WARNING: Sorry: the -p parameter has an invalid format
16/Feb/2016 18:04:33 [nprobe.c:4550] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
16/Feb/2016 18:04:33 [nprobe.c:4647] Welcome to nProbe Pro v.7.3.160216 (
16/Feb/2016 18:04:33 [nprobe.c:4657] Running on Ubuntu 14.04.3 LTS
16/Feb/2016 18:04:33 [nprobe.c:4668] [LICENSE] nProbe SystemId: A194887A9105A1EF
16/Feb/2016 18:04:33 [nprobe.c:4679] [LICENSE] nProbe License: 0453A945C52766B8D1A7286B3EBDCD9E14628610128FB9EE34
16/Feb/2016 18:04:33 [nprobe.c:4682] [LICENSE] nProbe Edition: Professional with PF_RING Acceleration
16/Feb/2016 18:04:33 [nprobe.c:4712] [LICENSE] Maintenance is available until Tue May 10 14:16:52 2016 [83 days left]
16/Feb/2016 18:04:33 [nprobe.c:6664] Welcome to nProbe v.7.3.160216 for x86_64-unknown-linux-gnu
16/Feb/2016 18:04:33 [nprobe.c:5922] Using NetFlow Packet Payload Len: 1472
16/Feb/2016 18:04:33 [plugin.c:1009] 1 plugin(s) enabled
16/Feb/2016 18:04:33 [nprobe.c:6319] Each flow is 290 bytes long
16/Feb/2016 18:04:33 [nprobe.c:6320] The # packets per flow has been set to 4
16/Feb/2016 18:04:33 [nprobe.c:6339] Non IPv4/v6 traffic is discarded according to the template
16/Feb/2016 18:04:33 [util.c:431] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
16/Feb/2016 18:04:33 [util.c:442] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
16/Feb/2016 18:04:33 [nprobe.c:5211] Using packet capture length 1600
16/Feb/2016 18:04:33 [plugin.c:825] Enabling plugin Diameter Protocol
Segmentation fault (core dumped)
Expected Result:
No segmentation fault and the process is up fine
Build Date & Hardware:
nProbe v.7.3.160216 (r4839) on Ubuntu 14.04.3 LTS
nProbe Command used:
/usr/local/bin/nprobe -n 192.168.10.198:9145 -i TEST.pcap -u 4 -Q 0 -d 5 -t 10 -V 9 -o 100 -U 600 -cpu-affinity 2 --export-thread-affinity 7 --dump-bad-packets /home/genie/probe_log/bad-pkt-eth2.pcap --event-log /home/genie/nprobe-event-eth2.log --bi-directional -T "%FIRST_SWITCHED %LAST_SWITCHED %IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS
%BIFLOW_DIRECTION
%L7_PROTO %APPLICATION_ID
%SRC_VLAN %DST_VLAN
%DIAMETER_REQ_MSG_TYPE %DIAMETER_RSP_MSG_TYPE %DIAMETER_REQ_ORIGIN_HOST %DIAMETER_RSP_ORIGIN_HOST %DIAMETER_REQ_USER_NAME %DIAMETER_RSP_RESULT_CODE %DIAMETER_EXP_RES_VENDOR_ID %DIAMETER_EXP_RES_RESULT_CODE %DIAMETER_HOP_BY_HOP_ID
"
Hi
nProbe v.7.3.160313 (r4932) for x86_64-unknown-linux-gnu
with native PF_RING acceleration.
Copyright 2002-16 ntop.org
Build OS: Debian GNU/Linux 8.2 (jessie)
SystemID: 3C0E6232B206AB23
Edition: nProbe Pro
[...]
With active HTTP module and nprobe.conf option where
-T=%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %OUT_PKTS %L4_DST_PORT %L4_SRC_PORT %IN_BYTES %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %PROTOCOL %IN_SRC_MAC %OUT_DST_MAC %TCP_FLAGS %APPL_LATENCY_MS %SRC_IP_COUNTRY %DST_IP_COUNTRY %L7_PROTO %L7_PROTO_NAME %ICMP_TYPE %IP_PROTOCOL_VERSION %NUM_PKTS_UP_TO_128_BYTES %NUM_PKTS_128_TO_256_BYTES %NUM_PKTS_256_TO_512_BYTES %NUM_PKTS_512_TO_1024_BYTES %NUM_PKTS_1024_TO_1514_BYTES %NUM_PKTS_OVER_1514_BYTES %DNS_QUERY %DNS_QUERY_TYPE %DNS_RET_CODE %DNS_NUM_ANSWERS %DNS_TTL_ANSWER %DNS_RESPONSE %HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE %DOT1Q_SRC_VLAN %DOT1Q_DST_VLAN %SRC_IP_CITY %DST_IP_CITY
During exporting data to Elasticsearch there are a lot of notes / errors in ES logs.
[2016-03-13 22:39:01,029][DEBUG][action.bulk ] [node-1] [nprobedev-2016.03.13][0] failed to execute bulk item (index) index {[nprobedev-2016.03.13][flows][AVNx6wUkGYu_AAvmVAjQ], source[{"IPV4_SRC_ADDR":"5.134.213.61","IPV4_DST_ADDR":"77.114.78.4","IN_PKTS":6,"OUT_PKTS":1,"L4_DST_PORT":49837,"L4_SRC_PORT":80,"IN_BYTES":4452,"OUT_BYTES":467,"FIRST_SWITCHED":1457905119,"LAST_SWITCHED":1457905120,"PROTOCOL":6,"IN_SRC_MAC":"50:C5:8D:1E:8F:C4","OUT_DST_MAC":"02:03:00:11:98:00","TCP_FLAGS":18,"APPL_LATENCY_MS":17.310,"SRC_IP_COUNTRY":"PL","DST_IP_COUNTRY":"PL","L7_PROTO":7,"L7_PROTO_NAME":"HTTP","ICMP_TYPE":0,"IP_PROTOCOL_VERSION":4,"NUM_PKTS_UP_TO_128_BYTES":3,"NUM_PKTS_128_TO_256_BYTES":0,"NUM_PKTS_256_TO_512_BYTES":2,"NUM_PKTS_512_TO_1024_BYTES":0,"NUM_PKTS_1024_TO_1514_BYTES":3,"NUM_PKTS_OVER_1514_BYTES":0,"HTTP_URL":"img16.staticclassifieds.com/images_tablicapl/350409411_1_261x203_grubosciowkastrugarkaheblarka-zywiec.jpg^Q�Dj^?","HTTP_METHOD":"GET","HTTP_RET_CODE":200,"HTTP_REFERER":"olx.pl/dom-ogrod/narzedzia/q-grubo%C5%9Bci%C3%B3wkaotwierdza-plotki-zrobila-to-ku-przera,nId,2161801-m-n-ts-rd-u-co-re-r-kg-g.","HTTP_UA":"Mozilla/5.0 (Windows NT 6.2; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0iowkastrugarkaheblarka-zywiec.jpg^Q�Dj^?","HTTP_MIME":"image/jpeg-ogrod/narzedzia/q-grubo%C5%9Bci%C3%B3wkaotwierdza-plotki-zrobila-to-ku-przera,nId,2161801-m-n-ts-rd-u-co-re-r-kg-g.","HTTP_HOST":"img16.staticclassifieds.comq-grubo%C5%9Bci%C3%B3wkaotwierdza-plotki-zrobila-to-ku-przera,nId,2161801-m-n-ts-rd-u-co-re-r-kg-g.","HTTP_SITE":"staticclassifieds.comNT 6.2; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0iowkastrugarkaheblarka-zywiec.jpg^Q�Dj^?","DOT1Q_SRC_VLAN":1480,"DOT1Q_DST_VLAN":1480,"SRC_IP_CITY":"Grupa","DST_IP_CITY":"Warsaw","@version":"1","@timestamp":"2016-03-13T21:38:39Z", "EXPORTER_IPV4_ADDRESS":"10.221.1.44"}]}
MapperParsingException[failed to parse [HTTP_URL]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 17)): has to be escaped using backslash to be included in string value
at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@1ebbe99e; line: 1, column: 749]];
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:343)
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:318)
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:445)
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:271)
at org.elasticsearch.index.mapper.DocumentParser.innerParseDocument(DocumentParser.java:131)
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:79)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:304)
at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:500)
at org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:481)
at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:214)
at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:595)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:263)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 17)): has to be escaped using backslash to be included in string value
at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@1ebbe99e; line: 1, column: 749]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
Overview:
nProbe provides %GTPV1_END_USER_IMEI for GTPv1 IMEI, but not for GTPv2.
Actual Result:
GTPv2 IMEI cannot be exported.
Expected Result:
New element %GTPV2_END_USER_IMEI for gtpv2.mei field. In the pcap attached, the value is 3578670615796200
Build Date & Hardware:
nProbe v.7.3.151202 (r4728) on Ubuntu 14.04.2 LTS
Attached pcap File:
GTP-tunnel.pcap.zip
Does these features work under Windows? I can't get the country flags to show in ntopng when I browse my flows in a Windows install.
I am using the most recent version and running nprobe with these options.
--as-list GeoIPASNum.dat --city-list GeoLiteCity.dat
Thanks!
Hi
I have a Cisco ASR 1k.
When i send Netflow v9 to nprobe it works well but when i change to ipfix i get 0 bps in "Actual Thpt" in ntopng. And the "info" always give "This flow cannot be found (expired ?)" but total bytes is showing but only low KB/byte.
I was hoping to use ipfix %HTTP_HOST to show site accessed thru HTTP proxy.
But i read some thing 6 mounts old saying it was not supported.
11/Dec/2015 23:54:40 [nprobe.c:3167] ERROR: *****************************************************
11/Dec/2015 23:54:40 [nprobe.c:3168] ERROR: ** **
11/Dec/2015 23:54:40 [nprobe.c:3169] ERROR: ** Switching to DEMO MODE (missing valid license) **
11/Dec/2015 23:54:40 [nprobe.c:3170] ERROR: ** **
11/Dec/2015 23:54:40 [nprobe.c:3171] ERROR: ** Create your nProbe license at **
11/Dec/2015 23:54:40 [nprobe.c:3172] ERROR: ** http://www.nmon.net/mklicense/ **
11/Dec/2015 23:54:40 [nprobe.c:3173] ERROR: ** **
11/Dec/2015 23:54:40 [nprobe.c:3174] ERROR: *****************************************************
11/Dec/2015 23:54:40 [nprobe.c:6681] ERROR: ***************************************************************
11/Dec/2015 23:54:40 [nprobe.c:6682] ERROR: * NOTE: This is a DEMO version limited to 25000 flows export. *
11/Dec/2015 23:54:40 [nprobe.c:6683] ERROR: ***************************************************************
11/Dec/2015 23:54:40 [plugin.c:169] No plugins found in ./plugins
11/Dec/2015 23:54:40 [plugin.c:177] Loading 22 plugins [.so] from /usr/local/lib/nprobe/plugins
11/Dec/2015 23:54:40 [nprobe.c:4576] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
11/Dec/2015 23:54:40 [nprobe.c:4579] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
11/Dec/2015 23:54:40 [nprobe.c:4671] Welcome to nProbe Pro v.7.3.151211 (
11/Dec/2015 23:54:40 [nprobe.c:4681] Running on CentOS Linux release 7.1.1503 (Core)
11/Dec/2015 23:54:40 [nprobe.c:4692] [LICENSE] nProbe SystemId: 689677AB82072B13
11/Dec/2015 23:54:40 [nprobe.c:6699] Welcome to nProbe v.7.3.151211 for x86_64-unknown-linux-gnu
11/Dec/2015 23:54:40 [plugin.c:1009] 0 plugin(s) enabled
11/Dec/2015 23:54:40 [nprobe.c:6374] Non IPv4/v6 traffic is discarded according to the template
11/Dec/2015 23:54:40 [util.c:431] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
11/Dec/2015 23:54:40 [util.c:441] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
11/Dec/2015 23:54:40 [nprobe.c:5243] Using packet capture length 128
11/Dec/2015 23:54:40 [nprobe.c:6872] IPv6 traffic will NOT be exported/accounted by this probe
11/Dec/2015 23:54:40 [nprobe.c:6873] due to configuration options (e.g. use NetFlow v9)
11/Dec/2015 23:54:40 [nprobe.c:7001] Not capturing packet from interface (collector mode)
11/Dec/2015 23:54:40 [util.c:4011] Succesfully created ZMQ endpoint tcp://127.0.0.1:5556
11/Dec/2015 23:54:40 [collect.c:145] Flow collector listening on port 2055 (IPv4/v6)
11/Dec/2015 23:54:40 [nprobe.c:7213] nProbe started successfully
^C11/Dec/2015 23:56:48 [cache.c:1210] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
11/Dec/2015 23:56:48 [nprobe.c:394] Received shutdown request... [signal: 2]
11/Dec/2015 23:56:48 [engine.c:2639] About to flush hash (threadId 0)
11/Dec/2015 23:56:48 [engine.c:2641] Completed hash walk (thread 0)
11/Dec/2015 23:56:51 [cache.c:1210] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
11/Dec/2015 23:56:51 [nprobe.c:2503] Processed packets: 0 (max bucket search: 0)
11/Dec/2015 23:56:51 [nprobe.c:2486] Fragment queue length: 0
11/Dec/2015 23:56:51 [nprobe.c:2512] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
11/Dec/2015 23:56:51 [nprobe.c:2519] Flow collection: [collected pkts: 543][processed flows: 2395]
11/Dec/2015 23:56:51 [nprobe.c:2522] Flow drop stats: [0 bytes/0 pkts][0 flows]
11/Dec/2015 23:56:51 [nprobe.c:2527] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
regards
Christer
Hi,
I was checking how nprobe will be count RTP and quality. (I'm using demo mode , but as I understand all plugins should works, yes?)
I used such command:
nprobe -i eth0 -P /user -T "%IPV4_SRC_ADDR %L4_SRC_PORT %IPV4_DST_ADDR %L4_DST_PORT %L7_PROTO_NAME %RTP_SSRC %RTP_FIRST_SEQ %RTP_FIRST_TS %RTP_LAST_SEQ %RTP_LAST_TS %RTP_IN_JITTER %RTP_OUT_JITTER %RTP_IN_PKT_LOST %RTP_OUT_PKT_LOST %RTP_IN_PAYLOAD_TYPE %RTP_OUT_PAYLOAD_TYPE %RTP_IN_MAX_DELTA %RTP_OUT_MAX_DELTA %RTP_SIP_CALL_ID %RTP_MOS %RTP_IN_MOS %RTP_R_FACTOR %RTP_RTT"
I received eg:
192.168.1.1|7070|192.168.1.2|11780|RTP|563533831|83|32000|702|131040|1353|-1|0|0|8|-1|3395|0||0|0|0|1.031
and very similar for
192.168.1.2|11780|192.168.1.1|7070|.....
first issue - "-1" - this is only in-traffic so out is unavailable. I received two separate lines in log file.
How it can be join? Is it any possibility? One line for both RTP streams?
second issue - why MOS, R-Factor is 0? Did I ommit something in command?
third issue - RTP and SIP flows are together, so it should be possible to join SIP flow and RTP, but in my case it not happened. RTP_SIP_CALL_ID is empty
Additionaly at log file I see also SIP flow:
192.168.1.1|5060|192.168.1.2|5062|SIP|0|0|0|0|0|0|0|0|0|-1|0|0||0|0|0|0
what is wrong??
nProbe in interactive mode stops working (or writing records to MySQL) after 25.001 records. While in console mode (nprobe /c) all work fine (with the same command as in interactive mode)
We have IPFIX flow data coming from a Juniper MX router to the nProbe server. nProbe is not capturing the data or forwarding to other collectors.
.pcap file attached of flow data received while the following nProbe instance was running. As you can see below, nProbe does not capture any flows.
root@uncsnbox:~# tcpdump -n -l -i eth4 port 2055 -w cflow.pcap
(rename to .pcap, github kept rejected a .zip of this)
cflow.txt
nbox@uncsnbox:~$ nprobe -3 2055 --zmq=tcp://:5556 -n none -i none -b 2
04/Dec/2015 14:12:34 [nprobe.c:3130] Valid nProbe license found
04/Dec/2015 14:12:34 [nprobe.c:4488] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
04/Dec/2015 14:12:34 [nprobe.c:4491] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
04/Dec/2015 14:12:34 [nprobe.c:4552] Welcome to nProbe v.7.2.151202 ($Revision: 4471 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration
04/Dec/2015 14:12:34 [nprobe.c:4562] Running on Ubuntu 14.04.2 LTS
04/Dec/2015 14:12:34 [nprobe.c:4573] [LICENSE] nProbe SystemId: FA623D157104A1D2
04/Dec/2015 14:12:34 [nprobe.c:4584] [LICENSE] nProbe License: D96F0F134B77B12B046C75AE0CF3BD0D1480454057C265E036
04/Dec/2015 14:12:34 [nprobe.c:4587] [LICENSE] nProbe Edition: Standard [without PF_RING Acceleration]
04/Dec/2015 14:12:34 [nprobe.c:4614] [LICENSE] Maintenance is available until Tue Nov 29 15:14:17 2016 [361 days left]
04/Dec/2015 14:12:34 [nprobe.c:4620] Tracing enabled
04/Dec/2015 14:12:34 [plugin.c:248] 0 plugin(s) loaded [0 delete][0 packet].
04/Dec/2015 14:12:34 [nprobe.c:6526] Welcome to nprobe v.7.2.151202 for x86_64-unknown-linux-gnu
04/Dec/2015 14:12:34 [nprobe.c:5752] Compiling flow templates...
04/Dec/2015 14:12:34 [plugin.c:1000] 0 plugin(s) enabled
04/Dec/2015 14:12:34 [nprobe.c:6203] Non IPv4/v6 traffic is discarded according to the template
04/Dec/2015 14:12:34 [util.c:287] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
04/Dec/2015 14:12:34 [util.c:296] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
04/Dec/2015 14:12:34 [nprobe.c:5121] Using packet capture length 128
04/Dec/2015 14:12:34 [nprobe.c:6698] IPv6 traffic will NOT be exported/accounted by this probe
04/Dec/2015 14:12:34 [nprobe.c:6699] due to configuration options (e.g. use NetFlow v9)
04/Dec/2015 14:12:34 [nprobe.c:6702] The flows hash has 131072 buckets
04/Dec/2015 14:12:34 [nprobe.c:6704] Flows older than 120 seconds will be exported
04/Dec/2015 14:12:34 [nprobe.c:6707] Flows inactive for at least 30 seconds will be exported
04/Dec/2015 14:12:34 [nprobe.c:6710] Expired flows will not be queued for more than 30 seconds
04/Dec/2015 14:12:34 [nprobe.c:6717] Exported flows with engineType 0 and engineId 178
04/Dec/2015 14:12:34 [nprobe.c:6739] TCP TOS will be ignored and set to 0.
04/Dec/2015 14:12:34 [nprobe.c:6757] After 1 flow packets are sent, we'll delay at least 1 ms
04/Dec/2015 14:12:34 [nprobe.c:6777] Flows will be emitted in NetFlow 5 format
04/Dec/2015 14:12:34 [nprobe.c:6807] Flow input interface index is set to 0
04/Dec/2015 14:12:34 [nprobe.c:6813] Flow output interface index is set to 0
04/Dec/2015 14:12:34 [nprobe.c:6827] Not capturing packet from interface (collector mode)
04/Dec/2015 14:12:34 [util.c:3840] Succesfully created ZMQ endpoint tcp://:5556
04/Dec/2015 14:12:34 [collect.c:86] Created UDP sockets
04/Dec/2015 14:12:34 [collect.c:90] Created a SCTP socket (22)
04/Dec/2015 14:12:34 [collect.c:145] Flow collector listening on port 2055 (IPv4/v6)
04/Dec/2015 14:12:34 [nprobe.c:6947] Starting 1 packet fetch thread(s)
04/Dec/2015 14:12:34 [engine.c:3210] Starting bucket dequeue thread
04/Dec/2015 14:12:34 [nprobe.c:7035] nProbe started successfully
^C04/Dec/2015 14:14:15 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
04/Dec/2015 14:14:15 [nprobe.c:386] Received shutdown request... [signal: 2]
04/Dec/2015 14:14:15 [nprobe.c:4716] nProbe is shutting down...
04/Dec/2015 14:14:15 [nprobe.c:4752] Exporting pending buckets...
04/Dec/2015 14:14:15 [nprobe.c:4773] Pending buckets have been exported...
04/Dec/2015 14:14:17 [engine.c:3293] Export thread terminated [exportQueue=0]
04/Dec/2015 14:14:17 [nprobe.c:4839] Flushing queued flows...
04/Dec/2015 14:14:17 [nprobe.c:4842] Freeing memory...
04/Dec/2015 14:14:17 [plugin.c:277] Terminating plugins.
04/Dec/2015 14:14:17 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
04/Dec/2015 14:14:17 [nprobe.c:4934] Still allocated 0 hash buckets
04/Dec/2015 14:14:17 [nprobe.c:2457] Processed packets: 0 (max bucket search: 0)
04/Dec/2015 14:14:17 [nprobe.c:2440] Fragment queue length: 0
04/Dec/2015 14:14:17 [nprobe.c:2466] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
04/Dec/2015 14:14:17 [nprobe.c:2473] Flow collection: [collected pkts: 0][processed flows: 0]
04/Dec/2015 14:14:17 [nprobe.c:2476] Flow drop stats: [0 bytes/0 pkts][0 flows]
04/Dec/2015 14:14:17 [nprobe.c:2481] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
04/Dec/2015 14:14:17 [nprobe.c:4947] Cleaning globals
04/Dec/2015 14:14:17 [nprobe.c:4967] nProbe terminated.
Summary:
The template %GTPV2_RAT_TYPE is available but not for GTPv1
Actual result:
Not able to retrieve GTPv1 RAT type
Expected result:
New template %GTPV1_RAT_TYPE is available for use
Build Date & Hardware:
nProbe v.7.3.160104 (r4767) on Ubuntu 14.04.3 LTS
This feature request is similar to the sample application pfcount, Which allows the user to select which direction to probe.
printf("-e 0=RX+TX, 1=RX only, 2=TX only\n");
This will be a very beneficial feature for users who only monitor unidirectional traffic.
A good example is that when pf_ring is loaded, ntopng only listens on the RX queues of the interface. However, if a user is listening to a vnet interface, the guest's RX interface is actually vnet's TX interface.
Currently, in our setup, i have disabled pf_ring so ntopng would listen to both RX and TX interfaces, but the added benefit of pf_ring is not being utilized
I've also made a feature request for ntopng
Overview:
Missing decoder warning message appears with GTP traffic
Actual Result:
03/Mar/2016 20:26:30 [nprobe.c:4650] Welcome to nProbe Pro v.7.3.160217 (
03/Mar/2016 20:26:30 [nprobe.c:4660] Running on Ubuntu 14.04.3 LTS
03/Mar/2016 20:26:30 [nprobe.c:4671] [LICENSE] nProbe SystemId: 39ECC22D76066B0A
03/Mar/2016 20:26:30 [nprobe.c:6667] Welcome to nProbe v.7.3.160217 for x86_64-unknown-linux-gnu
03/Mar/2016 20:26:30 [nprobe.c:5925] Using NetFlow Packet Payload Len: 1472
03/Mar/2016 20:26:30 [plugin.c:1009] 2 plugin(s) enabled
03/Mar/2016 20:26:30 [nprobe.c:6322] Each flow is 298 bytes long
03/Mar/2016 20:26:30 [nprobe.c:6323] The # packets per flow has been set to 3
03/Mar/2016 20:26:30 [nprobe.c:6342] Non IPv4/v6 traffic is discarded according to the template
03/Mar/2016 20:26:30 [util.c:431] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
03/Mar/2016 20:26:30 [util.c:442] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
03/Mar/2016 20:26:30 [nprobe.c:5214] Using packet capture length 1600
03/Mar/2016 20:26:30 [nprobe.c:6979] Enabled tunnel decoding (e.g. IPSEC/GTP)
03/Mar/2016 20:26:30 [plugin.c:825] Enabling plugin GTPv1 Signaling Protocol
03/Mar/2016 20:26:30 [plugin.c:825] Enabling plugin GTPv2 Signaling Protocol
03/Mar/2016 20:26:30 [gtpv2Plugin.c:808] WARNING: Missing decoder for GTP type: 0xAA [offset: 34/0022][GTPv2 packet_id: 1]
Expected Result:
Should be no warning messages.
nProbe Command used:
/usr/local/bin/nprobe -n 192.168.10.198:8888 -i GTPv2_0xAA_Missing.pcap -u 5 -Q 0 -d 10 -t 15 -V 9 -o 10 -U 710
-cpu-affinity 6 --export-thread-affinity 12
-f "udp dst port 2123"
-b 0
--tunnel
--dump-bad-packets /home/genie/bad-pkt-eth3-gtpc.pcap
--timestamp-format 1
--event-log /home/genie/genie-event-eth3-gtpc.log
--bi-directional
-T
"
%FIRST_SWITCHED %LAST_SWITCHED
%FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS
%IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS
%BIFLOW_DIRECTION
%L7_PROTO %APPLICATION_ID
%UPSTREAM_TUNNEL_ID %DOWNSTREAM_TUNNEL_ID %UNTUNNELED_IPV4_SRC_ADDR %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_PROTOCOL
%GTPV1_REQ_MSG_TYPE %GTPV1_RSP_MSG_TYPE %GTPV1_C2S_TEID_DATA %GTPV1_C2S_TEID_CTRL %GTPV1_S2C_TEID_DATA %GTPV1_S2C_TEID_CTRL %GTPV1_END_USER_IP %GTPV1_END_USER_IMSI %GTPV1_END_USER_MSISDN %GTPV1_END_USER_IMEI %GTPV1_APN_NAME %GTPV1_RAI_MCC %GTPV1_RAI_MNC %GTPV1_RAI_LAC %GTPV1_RAI_RAC %GTPV1_ULI_MCC %GTPV1_ULI_MNC %GTPV1_ULI_CELL_LAC %GTPV1_ULI_CELL_CI %GTPV1_ULI_SAC %GTPV1_RESPONSE_CAUSE %GTPV1_RAT_TYPE
%GTPV2_REQ_MSG_TYPE %GTPV2_RSP_MSG_TYPE %GTPV2_S5_S8_GTPC_TEID %GTPV2_C2S_S5_S8_GTPU_TEID %GTPV2_S2C_S5_S8_GTPU_TEID %GTPV2_C2S_S5_S8_GTPU_IP %GTPV2_S2C_S5_S8_GTPU_IP %GTPV2_END_USER_IMSI %GTPV2_END_USER_MSISDN %GTPV2_APN_NAME %GTPV2_ULI_MCC %GTPV2_ULI_MNC %GTPV2_ULI_CELL_TAC %GTPV2_ULI_CELL_ID %GTPV2_RESPONSE_CAUSE %GTPV2_RAT_TYPE %GTPV2_PDN_IP %GTPV2_END_USER_IMEI
"
Build Date&Hardware:
nProbe v.7.3.160217 (r4844) on Ubuntu 14.04.3 LTS
Additional Info:
pcap file
GTPv2_0xAA_Missing.pcap.zip
Which versions of Windows OSs are supported by nProbe on Windows?
Hello,
Running with cento, I found that directory and file permission is not secure enough which is using 777 for directory permission and files for 666.
Would you please change it more secure way - example) 755 for directory and 644 for files ?
Thanks,
i used a command like --black-list= or --collection-filter but when i look at traffic profiles or flow i always see the networks in the list.
How can i remove networks from netflow?
thanks
Sven
If one uses the following -T string you will have very low values for in_pkts and in_bytes. Removing %OUT_PKTS, %OUT_BYTES, %RETRANSMITTED_OUT_PKTS and %OOORDER_OUT_PKTS fixes the issue and results in proper data.
-T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %IPV6_SRC_ADDR %IPV6_DST_ADDR %IN_SRC_MAC %OUT_DST_MAC %FRAGMENTS %CLIENT_NW_DELAY_USEC %CLIENT_NW_DELAY_MS %SERVER_NW_DELAY_USEC %SERVER_NW_DELAY_MS %APPL_LATENCY_MS %RETRANSMITTED_IN_PKTS %RETRANSMITTED_OUT_PKTS %OOORDER_IN_PKTS %OOORDER_OUT_PKTS”
Hello team,
After reading through the following blog post:
http://mrncciew.com/2013/02/13/who-really-support-wlc-netflow/
It seems that Cisco sends these unique fields in their Wireless LAN Controller netflow v9 exports:
• applicationTag
• ipDiffServCodePoint
• octetDeltaCount
• packetDeltaCount
• postIpDiffServCodePoint
• staIPv4Address
• staMacAddress
• wlanSSID
• wtpMacAddress
Could you please help me in getting these fields incorporated into nprobe? Currently, ntopng shows zero data for the received netflow packets coming from the WLC.
I'm attaching the pcap file for a netflow capture taken on the nprobe server where I used the following: "tcpdump -n host 137.158.248.10 -w WLC -s 0" :
The nprobe and ntopng commands that I used are as follows:
nprobe --zmq tcp://127.0.0.1:2055 --collector-port 9991 -i none -n none -b 2 &
ntopng -i tcp://127.0.0.1:2055 &
Best regards,
Yasser
We are using nProbe (version v.7.2.151021 (
here are the links for both file:
in the config file of the nProbe it clearly states --disable-l7-protocol-guess.
thanks
Ohad
Overview:
Recent we found out that for one interface of 41.03 kpps, 205.53 Mbps traffic, nProbe (without enabling L7 dissection) experiences quite amount of packet loss. (the total traffic of all interfaces tested in lab is 42.21 Kpps, 211.70 Mbps)
Actual Result:
root@GenieProbe-6400:/home/genie# date ; cat /proc/net/pf_ring/stats/4590-eth3.6328
Thu Jan 28 16:02:26 CST 2016
Duration: 0:00:01:34:008
Bytes: 2211376777
Packets: 3548844
Dropped: 446036
root@GenieProbe-6400:/home/genie# date ; cat /proc/net/pf_ring/stats/4590-eth3.6328
Thu Jan 28 16:02:33 CST 2016
Duration: 0:00:01:41:008
Bytes: 2386219167
Packets: 3824021
Dropped: 468422
Expected Result:
The packet loss should not be that significant
Build Date & Hardware:
nProbe v.7.3.160118 (r4799) on Debian GNU/Linux 8.2 (jessie)
nProbe Command used:
/usr/local/bin/nprobe -n 192.168.10.198:9145 -i eth3 -u 5 -Q 0 -d 5 -t 10 -V 9 -o 10 -U 700
--tunnel
-cpu-affinity 1
--dump-bad-packets /home/genie/pcap/bad-pkt-eth3.pcap
--timestamp-format 1
--event-log /home/genie/nprobe-event-eth3.log
--bi-directional
-T
"
%FIRST_SWITCHED %LAST_SWITCHED
%FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS
%IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS
%BIFLOW_DIRECTION
%L7_PROTO %APPLICATION_ID
%UPSTREAM_TUNNEL_ID %DOWNSTREAM_TUNNEL_ID %UNTUNNELED_IPV4_SRC_ADDR %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_PROTOCOL
%DNS_QUERY %DNS_QUERY_ID %DNS_QUERY_TYPE %DNS_RET_CODE
%GTPV1_REQ_MSG_TYPE %GTPV1_RSP_MSG_TYPE %GTPV1_C2S_TEID_DATA %GTPV1_C2S_TEID_CTRL %GTPV1_S2C_TEID_DATA %GTPV1_S2C_TEID_CTRL %GTPV1_END_USER_IP %GTPV1_END_USER_IMSI %GTPV1_END_USER_MSISDN %GTPV1_END_USER_IMEI %GTPV1_APN_NAME %GTPV1_RAI_MCC %GTPV1_RAI_MNC %GTPV1_RAI_LAC %GTPV1_RAI_RAC %GTPV1_ULI_MCC %GTPV1_ULI_MNC %GTPV1_ULI_CELL_LAC %GTPV1_ULI_CELL_CI %GTPV1_ULI_SAC %GTPV1_RESPONSE_CAUSE %GTPV1_RAT_TYPE
"
Additional Info:
CPU loading (Intel Xeon CPU E5620 @ 2.40GHz)-
root@GenieProbe-6400:/home/genie# top
top - 16:40:37 up 23:48, 2 users, load average: 0.14, 0.16, 0.21
Tasks: 233 total, 1 running, 232 sleeping, 0 stopped, 0 zombie
%Cpu0 : 10.7 us, 1.4 sy, 0.0 ni, 88.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 1.4 us, 1.7 sy, 0.0 ni, 96.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 1.4 us, 2.4 sy, 0.0 ni, 96.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 3.8 us, 2.1 sy, 0.0 ni, 94.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 1.4 us, 0.7 sy, 0.0 ni, 97.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.7 us, 1.8 sy, 0.0 ni, 96.1 id, 1.1 wa, 0.0 hi, 0.4 si, 0.0 st
%Cpu6 : 1.0 us, 0.3 sy, 0.0 ni, 98.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.7 us, 1.4 sy, 0.0 ni, 97.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 1.4 us, 1.1 sy, 0.0 ni, 97.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 1.7 us, 1.3 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 2.1 us, 2.1 sy, 0.0 ni, 95.1 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu12 : 1.4 us, 1.1 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
%Cpu13 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu14 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu15 : 1.0 us, 0.7 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 8165952 total, 1385312 used, 6780640 free, 275416 buffers
KiB Swap: 8377340 total, 0 used, 8377340 free. 605316 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11290 nobody 20 0 388052 72760 12384 S 23.9 0.9 0:12.63 nprobe
1641 nobody 20 0 1314980 60112 15884 S 11.0 0.7 158:51.34 ntopng
11292 nobody 20 0 388008 21284 12332 S 11.0 0.3 0:05.55 nprobe
11291 nobody 20 0 322532 16980 12148 S 10.0 0.2 0:04.78 nprobe
11293 nobody 20 0 322472 16952 12148 S 9.3 0.2 0:04.92 nprobe
153 root 39 19 0 0 0 S 0.3 0.0 0:08.22 khugepaged
232 root 20 0 0 0 0 S 0.3 0.0 0:17.73 kworker/9:1
1181 root 20 0 18180 1712 1236 S 0.3 0.0 0:15.52 check_failure_a
1 root 20 0 33488 2860 1472 S 0.0 0.0 0:03.31 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:01.96 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
Memory usage -
root@GenieProbe-6400:/home/genie# cat /proc/meminfo
MemTotal: 8165952 kB
MemFree: 6779336 kB
Buffers: 275472 kB
Cached: 605420 kB
SwapCached: 0 kB
My project is – to implement own DPI (l7 application reckognition) as a part of router functionality.
I want to send reports with L7 application names(codes) to nTopng, in order to display DPI results.
Is that possible?
I run nprobe+ntopng on CentOS 7.2
/etc/ntopng/ntopng.conf:
-G=/var/tmp/ntopng.pid --community --dont-change-user -i tcp://127.0.0.1:5557
nprobe command line:
nprobe --zmq "tcp://127.0.0.1:5557" -i none -n none --collector-port 2055 -T "%FIRST_SWITCHED %LAST_SWITCHED %PROTOCOL %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IN_BYTES %IN_PKTS %APPLICATION_ID %APPLICATION_NAME %L7_PROTO %L7_PROTO_NAME" --disable-l7-protocol-guess -V 9
Now I send CFLOW packets to port 2055. I define set of NBAR applications, and I send NBAR ID + L7_PROTO fields (see attachment) in hope to see or NBAR name, or L7 name
Result – I can see NBAR ID inside flow report, but I don't see desired application name. No NBAR, no L7
Question – is it possible to display results of "third party DPI" (my router) using nTopng UI?
Thanks,
Igor.
Create 2 new args that configure flow to be sent to the Kentik platform:
--kentik-host: (Any flag below can be overridden with another command-line arg except -n, that cannot be overridden)
--kentik-sensor (Any flag below can be overridden with another command-line arg except -n, that cannot be overridden)
Overview:
nProbe process experiences segmentation fault dealing with DNS packets.
Actual Result:
03/Mar/2016 21:02:55 [nprobe.c:6667] Welcome to nProbe v.7.3.160217 for x86_64-unknown-linux-gnu
03/Mar/2016 21:02:55 [nprobe.c:5925] Using NetFlow Packet Payload Len: 1472
03/Mar/2016 21:02:55 [plugin.c:1009] 1 plugin(s) enabled
03/Mar/2016 21:02:55 [nprobe.c:6322] Each flow is 203 bytes long
03/Mar/2016 21:02:55 [nprobe.c:6323] The # packets per flow has been set to 6
03/Mar/2016 21:02:55 [nprobe.c:6342] Non IPv4/v6 traffic is discarded according to the template
03/Mar/2016 21:02:55 [util.c:431] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
03/Mar/2016 21:02:55 [util.c:442] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
03/Mar/2016 21:02:55 [nprobe.c:5214] Using packet capture length 1600
03/Mar/2016 21:02:55 [nprobe.c:6979] Enabled tunnel decoding (e.g. IPSEC/GTP)
03/Mar/2016 21:02:55 [plugin.c:825] Enabling plugin DNS/LLMNR Protocol
Segmentation fault (core dumped)
Expected Result:
No segmentation fault should appear.
nProbe Command used:
/usr/local/bin/nprobe -n 192.168.10.198:8888 -i GTP_DNS_Segmentation_fault.pcap -u 5 -Q 0 -d 10 -t 30 -V 9 -o 10 -U 700
-cpu-affinity 3 --export-thread-affinity 8
-f "udp dst port 2152"
-b 0
--tunnel
--dump-bad-packets /home/genie/bad-pkt-eth3.pcap
--timestamp-format 1
--event-log /home/genie/genie-event-eth3.log
--bi-directional
-T
"
%FIRST_SWITCHED %LAST_SWITCHED
%FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS
%IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS
%BIFLOW_DIRECTION
%UPSTREAM_TUNNEL_ID %DOWNSTREAM_TUNNEL_ID %UNTUNNELED_IPV4_SRC_ADDR %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_PROTOCOL
%L7_PROTO %APPLICATION_ID
%DNS_QUERY %DNS_QUERY_ID %DNS_QUERY_TYPE %DNS_RET_CODE
"
Build Date&Hardware:
nProbe v.7.3.160217 (r4844) on Ubuntu 14.04.3 LTS
Additional Info:
pcap file
GTP_DNS_Segmentation fault.pcap.zip
I sent the pcap file in email to Luca, as requested.
I am sending Netflow packets to nprobe, containing the application and L7 protocol which are determined by my DPI. Those values are not presented by the ntopng.
I run nprobe with –T including L7_PROTO and L7_PROTO_NAME
I send to nprobe netflow packets with APPLICATION_ID, L7_PROTO and L7_PROTO_NAME, set to values from my own DPI (I do not wish to use the nDPI)
I see the nprobe priniting the L7_PROTO as Unknown
In the ntopng I see the L7 protocol as Unknown and the Application is set by the port of the flow
How do I make the ntopng present my own values of DPI?
The output of the nprobe is:
[ZMQ] {"22":4290673048,"21":1446400270,"4":6,"8":"10.1.20.174","12":"212.179.180.82","7":55546,"11":443,"1":592,"2":6,"95":"70:80","57590":0,"57591":"Unknown","42":4}
01/Nov/2015 19:51:10 [engine.c:2541] Emitting Flow: [->][tcp] 10.1.20.174:55546 -> 212.179.180.82:443 [6 pkt/592 bytes][ifIdx 0->0][0.0 sec][Unknown.Unknown/0][init Unknown][AS: 0 -> 8551]
Thanks,
Anat
We have installed nProbe v.7.2.150922 (r4468) on ubuntu 14.04 (64 but) and we are trying to monitor a network with IP cameras.
We are using the nProbe to send network information into elastic search along with dump files into folder and to analyze the information, with the http plugin we also dump logs into a folder
in some of the http requests relevant information is located in line-based text data, this is also relevant in http/xml format where important info is located in the body\envelope section.
it would be very useful to include this data in the dump file.
thanks.
Ohad
As we know that nDPI integrates a Quic dissector.
A Question arise for some entries resulting after processing a large pcap file.
I'm using nprobe and http plugin.
For some flows the App Name is Only "Google". At http plugin output side, this flows was not processed by the plugin. Do you agree with considering this flow as Quic.Google as no protocols (SSL, HTTP, DNS , etc) was detected and HTTP plugin didn't recognize the packet as "parsable".
Same question arises for others cases such as "CloudFlare". As we know Cloudflare integrates SPDY while delivering some contents. Is is correct to consider it as SPDY?
As part of the Kafka option in nprobe:
--kafka :: | Send flows to the specified Apache Kafka broker. Example --kafka localhost:test:1
You have to specify a static partition to be used. However an actual Kafka producer implementation should (or maybe even must) be able to discover (using metadata) the right partition to send to instead of it being specified in the configuration.
The Kafka option should only ask for a list of one or many producers and the partition („key“ could be an option value if static partioning is desired). Then it should make a call to fetch the metadata for the topic from the first broker (or another broker if the first is unavailable). This metadata will contain:
The current implementation „speaks“ the Kafka protocol, but its limitations defeat the purpose of Kafka (distributed and highly available)
Overview:
The flow traffic statistics will drop when enabling HTTP templates (e.g. 100Mbps traffic but flow shows < 60Mbps). Removing HTTP templates will come back to normal.
Actual Result:
---pf_ring status---
root@GenieProbe-6400:/home/genie# cat /proc/net/pf_ring/stats/24614-eth3.87
Duration: 0:00:23:21:119
Bytes: 37046105281
Packets: 58196558
Dropped: 2248715
Expected Result:
Adding HTTP templates should not be much discrepancy in traffic flow counting.
Build Date & Hardware:
nProbe v.7.3.151213 (r4738) on Ubuntu 14.04.2 LTS
nProbe Command Used:
/usr/local/bin/nprobe -n 192.168.10.168:9145 -i eth3 -u 5 -Q 0 -d 10 -t 10 -V 9 -o 10 -U 700
--tunnel
--dump-bad-packets /home/genie/pcap/bad-pkt-eth3.pcap
--timestamp-format 1
-T
"
%FIRST_SWITCHED %LAST_SWITCHED
%FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS
%IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS
%L7_PROTO
%UPSTREAM_TUNNEL_ID %DOWNSTREAM_TUNNEL_ID %UNTUNNELED_IPV4_SRC_ADDR %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_PROTOCOL
%HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %HTTP_UA %HTTP_HOST
%DNS_QUERY %DNS_QUERY_ID %DNS_QUERY_TYPE %DNS_RET_CODE
%GTPV1_REQ_MSG_TYPE %GTPV1_RSP_MSG_TYPE %GTPV1_C2S_TEID_DATA %GTPV1_C2S_TEID_CTRL %GTPV1_S2C_TEID_DATA %GTPV1_S2C_TEID_CTRL %GTPV1_END_USER_IP %GTPV1_END_USER_IMSI %GTPV1_END_USER_MSISDN %GTPV1_END_USER_IMEI %GTPV1_APN_NAME %GTPV1_RAI_MCC %GTPV1_RAI_MNC %GTPV1_RAI_LAC %GTPV1_RAI_RAC %GTPV1_ULI_MCC %GTPV1_ULI_MNC %GTPV1_ULI_CELL_LAC %GTPV1_ULI_CELL_CI %GTPV1_ULI_SAC %GTPV1_RESPONSE_CAUSE
%GTPV2_REQ_MSG_TYPE %GTPV2_RSP_MSG_TYPE %GTPV2_S5_S8_GTPC_TEID %GTPV2_C2S_S5_S8_GTPU_TEID %GTPV2_S2C_S5_S8_GTPU_TEID %GTPV2_C2S_S5_S8_GTPU_IP %GTPV2_S2C_S5_S8_GTPU_IP %GTPV2_END_USER_IMSI %GTPV2_END_USER_MSISDN %GTPV2_APN_NAME %GTPV2_ULI_MCC %GTPV2_ULI_MNC %GTPV2_ULI_CELL_TAC %GTPV2_ULI_CELL_ID %GTPV2_RESPONSE_CAUSE %GTPV2_RAT_TYPE %GTPV2_PDN_IP %GTPV2_END_USER_IMEI
" > /home/genie/nprobe-eth3.log &
starting nprobe with export to ES on windows with command:
nprobe /c --collector-port 6343 --elastic "flows;nprobe;http://192.168.97.222:9200/_bulk"
it runs for a few seconds, collecting incoming sflows (when use debug you can see them) but doesn't export them to ES, with netstat -abn we don't see any connection to 192.168.97.222.
It always stop with about 6mb debug data but in the data no errors are found.
running version : nProbe v.7.2.151019 (r4597)
os: windows 2012
Hi,
nProbe segfaults for us when the SIP and RTP plugins are enabled. From a coredump, i got the following backtrace:
(gdb) bt
#0 0x00007fbf0d4ec3a8 in parse_rtp_codecs () from /usr/local/lib/nprobe/plugins/libsipPlugin-7.3.160108.so
#1 0x00007fbf0d4ed520 in sipPlugin_packet () from /usr/local/lib/nprobe/plugins/libsipPlugin-7.3.160108.so
#2 0x00007fbf1a08dc8b in pluginCallback () from /usr/local/lib/libnprobe-7.3.160108.so
#3 0x00007fbf1a07b86e in processFlowPacket () from /usr/local/lib/libnprobe-7.3.160108.so
#4 0x000000000040a1c7 in ?? ()
#5 0x000000000040a98a in decodePacket ()
#6 0x0000000000413499 in ?? ()
#7 0x0000000000417388 in ?? ()
#8 0x00007fbf14cdaaf5 in __libc_start_main () from /lib64/libc.so.6
#9 0x0000000000405019 in ?? ()
I was able to reduce the problem to a specific SIP packet which contains a rather large SDP body:
m=audio 53456 RTP/AVP 0 8 98 97 2 96 4 18 15 100 106 114 254 254 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:98 G726-16/8000
a=rtpmap:97 G726-24/8000
a=rtpmap:2 G726-32/8000
a=rtpmap:96 G726-40/8000
a=rtpmap:4 G723/8000
a=rtpmap:18 G729/8000
a=rtpmap:15 G728/8000
a=rtpmap:100 G729E/8000
a=rtpmap:106 BV16/8000
a=rtpmap:114 iLBC/8000
a=rtpmap:254 encaprtp/8000
a=rtpmap:254 rtploopback/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:18 annexb=no
a=fmtp:114 mode=20
a=fmtp:101 0-15,144,149,159
a=ptime:20
a=sendrecv
a=silenceSupp:off - - - -
I will send an example pcap, which will result in a segfault, to [email protected].
Tested with centos and centos-stable repos from packages.ntop.org.
When i add the check field dump alle traffic nothing happens.
0 packets dumped. Why?
Overview:
Only the first set of Diameter info is collected by nProbe if a packet contains multiple Diameter sections.
Actual Result:
Only the part1 is processed and exported by nProbe.
Expected Result:
In the example above, both Diameter sections should be exported.
Build Date & Hardware:
nProbe v.7.3.160119 (r4799) on Ubuntu 14.04.3 LTS
Additional Info:
Diameter pcap file
diameter0121.pcap.zip
This BUG appears only for** nProbe versions greater than 7.3.151126 (r4714)** and consists on a Segmentation fault.
When processing a pcap file, and saving flows in sqlite format.
For a pcap file resulting on 50000000 flows dump. nProbe crashes at the end of the processing time with segmentation fault error.
The bug doesn’t appear when dumping 5000000 flows only.
This bug appear ONLY for recent versions of nProbe as explained above.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.