Open source components and extensions for n2disk
This repository contains open source components and extensions for n2disk available at http://www.ntop.org/products/traffic-recording-replay/n2disk/
Here you can find:
Enjoy!
If you want to learn more about n2disk please visit the User's Guide.
For more information about n2disk, please visit http://ntop.org
As reported here: https://askubuntu.com/questions/997820/unable-to-uninstall-package
Removing n2disk (3.0.180626-4915) ...
Stopping n2disk services...
update-rc.d: /etc/init.d/n2disk exists during rc.d purge (use -f to force)
dpkg: error processing package n2disk (--remove):
subprocess installed pre-removal script returned error exit status 1
OS: Redhat 8.2 Kernel: 4.18.0-193.1.2.el8_2.x86_64
N2DISK from http://packages.ntop.org/centos/
Version: n2disk-3.5.200528-5204.x86_64
Identifying a N2DISK PCAP (successful)
npcapextract -t /var/log/he/timeline -b "2020-06-02 13:25:43" -e "2020-06-02 13:25:59" -l
/var/log/he/1591118416.721609/1591118759.507696.pcap
I then want to move this pcap to another directory (starts to be problematic) :
npcapmove /var/log/he/1591118416.721609/1591118759.507696.pcap /var/log/he/pp/ /var/log/he/pp/timeline/
Files moved to:
/var/log/he/pp//1591118759.507696.pcap /var/log/he/pp//1591118759.507696.pcap.idx.timeline
/var/log/he/pp//1591118759.507696.pcap.idx /var/log/he/pp//1591118759.507696.pcap.timeline
/var/log/he/pp/timeline//2020/06/02/13/20/1591118759.507696.pcap /var/log/he/pp/timeline//2020/06/02/13/20/1591118759.507696.pcap.idx
Links are all relative paths, n2disk saves links to timelines with absolute paths:
-rw-r-----. 1 n2disk n2disk 4294967928 Jun 2 13:26 1591118759.507696.pcap
-rw-r-----. 1 n2disk n2disk 54292009 Jun 2 13:26 1591118759.507696.pcap.idx
drwxr-xr-x. 3 n2disk n2disk 26 Jun 3 12:24 timeline
lrwxrwxrwx. 1 n2disk n2disk 48 Jun 3 12:24 1591118759.507696.pcap.timeline -> timeline/2020/06/02/13/20/1591118759.507696.pcap
lrwxrwxrwx. 1 n2disk n2disk 52 Jun 3 12:24 1591118759.507696.pcap.idx.timeline -> timeline/2020/06/02/13/20/1591118759.507696.pcap.idx
./timeline/2020/06/02/13/20:
lrwxrwxrwx. 1 n2disk n2disk 40 Jun 3 12:24 1591118759.507696.pcap -> ../../../../../../1591118759.507696.pcap
lrwxrwxrwx. 1 n2disk n2disk 44 Jun 3 12:24 1591118759.507696.pcap.idx -> ../../../../../../1591118759.507696.pcap.idx
Run test to make sure npcapextract can see the pcap (successful):
npcapextract -t /var/log/he/pp/timeline -b "2020-06-02 13:25:43" -e "2020-06-02 13:25:59" -l
../../../../../../1591118759.507696.pcap
Next run npcapmange to delete the pcap, indexes and timeline files (fails):
npcapmanage -t /var/log/he/pp/timeline -b "2020-06-02 13:25:43" -e "2020-06-02 13:25:59" -d -v 4
03/Jun/2020 12:37:07 [npcapmanage.c:395] Welcome to npcapmanage - (C) 2016 ntop.org
03/Jun/2020 12:37:07 [npcapmanage.c:397] Begin time: 2020-06-02 13:25:43, End time 2020-06-02 13:25:59
03/Jun/2020 12:37:07 [npcapmanage.c:188] Scanning /var/log/he/pp/timeline/2020/06/02/13/20
03/Jun/2020 12:37:07 [npcapmanage.c:222] Checking epoch for /var/log/he/pp/timeline/2020/06/02/13/20/1591118759.507696.pcap: 1591118743 < 1591118759 <= 1591118759
03/Jun/2020 12:37:07 [npcapmanage.c:412] 0 PCAP files deleted
03/Jun/2020 12:37:07 [npcapmanage.c:413] Total processing time: 0.000 sec.
Fails to detect the PCAP even though npcapextract sees the pcap with the exact same filter.
Rerun with slightly larger window:
npcapmanage -t /var/log/he/pp/timeline -b "2020-06-02 13:25:43" -e "2020-06-02 13:26:00" -d -v 4
03/Jun/2020 12:43:31 [npcapmanage.c:395] Welcome to npcapmanage - (C) 2016 ntop.org
03/Jun/2020 12:43:31 [npcapmanage.c:397] Begin time: 2020-06-02 13:25:43, End time 2020-06-02 13:26:00
03/Jun/2020 12:43:31 [npcapmanage.c:188] Scanning /var/log/he/pp/timeline/2020/06/02/13/20
03/Jun/2020 12:43:31 [npcapmanage.c:222] Checking epoch for /var/log/he/pp/timeline/2020/06/02/13/20/1591118759.507696.pcap: 1591118743 < 1591118759 <= 1591118760
03/Jun/2020 12:43:31 [npcapmanage.c:236] rm ../../../../../../1591118759.507696.pcap
03/Jun/2020 12:43:31 [npcapmanage.c:241] rm ../../../../../../1591118759.507696.pcap.timeline
03/Jun/2020 12:43:31 [npcapmanage.c:245] rm /var/log/he/pp/timeline/2020/06/02/13/20/1591118759.507696.pcap
03/Jun/2020 12:43:31 [npcapmanage.c:258] rm ../../../../../../1591118759.507696.pcap.idx
03/Jun/2020 12:43:31 [npcapmanage.c:263] rm ../../../../../../1591118759.507696.pcap.idx.timeline
03/Jun/2020 12:43:31 [npcapmanage.c:267] rm /var/log/he/pp/timeline/2020/06/02/13/20/1591118759.507696.pcap.idx
03/Jun/2020 12:43:31 [npcapmanage.c:412] 1 PCAP files deleted
03/Jun/2020 12:43:31 [npcapmanage.c:413] Total processing time: 0.001 sec.
However, it does not delete all the files, just the timeline ones:
-rw-r-----. 1 n2disk n2disk 4294967928 Jun 2 13:26 1591118759.507696.pcap
-rw-r-----. 1 n2disk n2disk 54292009 Jun 2 13:26 1591118759.507696.pcap.idx
lrwxrwxrwx. 1 n2disk n2disk 48 Jun 3 12:24 1591118759.507696.pcap.timeline -> timeline/2020/06/02/13/20/1591118759.507696.pcap
lrwxrwxrwx. 1 n2disk n2disk 52 Jun 3 12:24 1591118759.507696.pcap.idx.timeline -> timeline/2020/06/02/13/20/1591118759.507696.pcap.idx
(links are bad since timeline directory doesn't exist. Not sure why it deletes the timeline directory as I would want to move other pcaps into it without recreating it)
The .pcap, .idx and the links should have been deleted.
Currently n2disk uses 0750 for created folders and 0640 for files. It is requested to make them configurable.
#define MKDIR_MODE 0750
#define OPEN_MODE 0640
I can not find the stats file you mentioned. Thanks.
And "ip netns exec 154 ls /proc/net/pf_ring/stats" also shows nothing. I forgot to mention it last time. sorry for that. However,
ip netns exec 154 ls /proc/net/pf_ring/dev/ will show
eth4 lo
Now I want to use n2disk to export the stream to ntopng for monitoring, but starting n2disk fails and reports the following error：
[email protected] - n2disk ultra-high-speed traffic recorder with realtime indexing on test
Loaded: loaded (/etc/systemd/system/[email protected]; disabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: signal) since Wed 2021-07-21 14:35:36 +08; 1s ago
Process: 3234 ExecStopPost=/bin/sh -c /bin/echo "$(/bin/date) n2disk test StopPost" >> /var/log/ntop-systemd.log (code=exited, status=0/SUCCESS)
Process: 3233 ExecStopPost=/bin/rm -rf /run/n2disk-test.conf (code=exited, status=0/SUCCESS)
Process: 3232 ExecStopPost=/bin/rm -rf /run/n2disk-test.env (code=exited, status=0/SUCCESS)
Process: 3215 ExecStartPost=/bin/sh -c /bin/echo "$(/bin/date) n2disk test StartPost" >> /var/log/ntop-systemd.log (code=exited, status=0/SUCCESS)
Process: 3214 ExecStart=/usr/bin/stdbuf -oL /usr/bin/${N2DISK_BINARY} /run/n2disk-test.conf (code=killed, signal=KILL)
Process: 3212 ExecStartPre=/bin/sh -c /bin/sed "/-P.$|--daemon.|--pid.*/s/^/#/" /etc/n2disk/n2disk-test.conf > /run/n2disk-test.conf (code=exited, st
Process: 3178 ExecStartPre=/bin/sh -c /usr/bin/n2disk --check-license | /bin/grep "Ok|Time-Limited" && /bin/echo "N2DISK_BINARY=n2disk" > /run/n2disk-te
Process: 3120 ExecStartPre=/bin/sh -c /usr/bin/n2disk5g --check-license | /bin/grep "Ok|Time-Limited" && /bin/echo "N2DISK_BINARY=n2disk5g" > /run/n2dis
Process: 3111 ExecStartPre=/bin/sh -c /usr/bin/n2disk1g --check-license | /bin/grep "Ok|Time-Limited" && /bin/echo "N2DISK_BINARY=n2disk1g" > /run/n2dis
Process: 3109 ExecStartPre=/bin/sh -c /bin/echo "N2DISK_BINARY=n2disk" > /run/n2disk-test.env (code=exited, status=0/SUCCESS)
Process: 3106 ExecStartPre=/bin/sh -c /bin/echo "$(/bin/date) n2disk test StartPre" >> /var/log/ntop-systemd.log (code=exited, status=0/SUCCESS)
Main PID: 3214 (code=killed, signal=KILL)
n2disk.conf
--interface=ens33
--dump-directory=/storage/n2disk/pcap
--timeline-dir=/storage/n2disk/timeline
--disk-limit=512
--max-file-len=1000
--buffer-len=4000
--max-file-duration=60
--index
--snaplen=1536
--writer-cpu-affinity=0
--reader-cpu-affinity=1
--compressor-cpu-affinity=2,3
--index-on-compressor-threads
-u=ntopng
--zmq=tcp://127.0.0.1:5556
--zmq-probe-mode
--zmq-export-flows
ntopng.conf
-i tcp://127.0.0.2:5556
-w=3001
-F=nindex
-m="192.168.0.0/24,192.168.1.0/24"
-G=/var/run/ntopng.pid
I recently got a license but its only supporting the 1g binary yet i have 10g binary. how do i change the binary to 1g
Hi,
Wanted to ask the question of whether n2disk would consider supporting the community-id spec as seen here - https://github.com/corelight/community-id-spec
Multiple network flow or analysis sensors support this whcih improves analysis workflow. having the same seed value configured in n2disk to generate the same hash across tools is beneficial to analysis workflow. Is this possible at all?
Also may assist with npcapextract search and extract functionality instead of defining 5 tuple bpf syntax it could be npcapextract community-id to pin point the same stream as other tools observe.
Thanks,
Nathan
Currently, we're intending to use n2disk for saving packet and use npcapextract for extracting a pcap file with l7proto, something looks like: npcapextract -a 9.pcap -i 9.pcap.idx -o test111.pcap -f "l7proto 159". But it won't work if we don't enable --zmq-export-flows. However, enabling export flows via zmq is seem redundant in our case. So it would be great if n2disk support index pcap file with l7proto as default.
Currently I am experiencing the following.
Using the -o options to send data to multiple different sized destination partitions, n2disk will fill up the first to the limit, then start rotating the information on all drives Below is an example of the output drives after /data4 attained 97% limit set by the execution options
/dev/sde1 12T 12T 411G 97% /data4
/dev/sdf1 29T 12T 17T 42% /data5
/dev/sdc1 53T 12T 41T 23% /data2
/dev/sdd1 53T 12T 41T 23% /data3
I found that data[2,3,5] will never increase in storage over the sizes shown - all will maintain the 12T limit imposed by /data4 - I reached this conclusion as the data shown is quite a while after data4 reached 97%, as well as by executing df and observing bytes deleted then written on the disks that were not yet full
this is how I start it /usr/bin/stdbuf -oL /usr/bin/n2disk --syslog --daemon -i fbcard:0:b00 -P /var/run/n2disk.pid -o /data3/pcap/ -o /data2/pcap -o /data4/pcap -o /data5/pcap -A /data2/timeline --disk-limit 97% -b 16384 -p 2048 -C 16384 -q 1 -c 34 -w 36,38,44,46 --index -3 -Z -z 48,50,52,54
n2disk v.3.6.230113 (r5273)
I have registered a ntopng Enterprise L license. According to the license generator it should also include a license for n2disk. However, it keeps running in demo and will only capture for 5 min. Did I do something wrong? I'm running ntopng Enterprise L v.5.0.211117 rev.15952 on Ubuntu 20.04.3 LTS.
This is probably more of an nBox issue than an n2disk issue, but this was the most relevant place I could find to open an issue.
We rename interfaces via udev rules:
root@pcap00:/storage/n2disk/xmr-ptc-tap# cat /etc/udev/rules.d/70-persistent-net.rules | grep xmr
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="a0:36:9f:8b:a0:dc", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="xmr-ptc-tap"
root@pcap00:/storage/n2disk/xmr-ptc-tap# ifconfig xmr-ptc-tap
xmr-ptc-tap Link encap:Ethernet HWaddr a0:36:9f:8b:a0:dc
inet6 addr: fe80::a236:9fff:fe8b:a0dc/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:9710 Metric:1
RX packets:313948571 errors:0 dropped:0 overruns:0 frame:0
TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:284011121856 (284.0 GB) TX bytes:324 (324.0 B)
Memory:91d00000-91e00000
The dashboard and PF_RING/ZC pages of nBox seem to detect the renamed interfaces fine:
The n2disk page, however, does not recognize it as an interface. The enp5s0f* interfaces were the names of the interfaces before they were renamed. The xmr-ptc-tap you see below is from trying to manually edit that non-existent interface via just replacing the bad ifname with that in the URL. That cause it to "create" in some capacity, as you can see; however, it still thinks that it is down and unsuable.
n2disk works just fine with the renamed interface when invoked from CLI. It just seems to be an nBox thing -- at least as far as I can identify.
It would be nice to be able to specify the configuration file for n2disk rather than relying on /etc/n2disk/n2disk-${INTERFACE_NAME}.conf.
I try to start n2disk with the napatech card. I followed the Appendix F from n2disk. I edited the /opt/napatech3/config/ntservice.ini following the documentation. I started the napatech driver and startup the pfring driver.
I give the following command:
[root@localhost scripts]# n2disk -o /data -p 1000 -b 4000 -i nt:1 -q 1 -C 1024 -c 1 -z 2 -w 3 -n 50 -m 100
The result is:
28/Aug/2018 10:49:32 [n2disk.c:6150] Welcome to n2disk v.3.0.180712 (r4916) [CPU 5065]
28/Aug/2018 10:49:32 [n2disk.c:6180] Running on 1 node(s) system with 8 core(s). NUMA affinity set to node 1.
28/Aug/2018 10:49:32 [n2disk.c:6202] ERROR: The interface you are using is not compatible with the 'Light' n2disk license
I don't know how to solve this error.
n2disk --checklicense give LicenseOK.
Please help.
Drean
hi, when i install ntopng for my company i cant start ntop, someone help me pls
when i start with "service ntopng start "
/usr/local/etc/rc.d/ntopng: WARNING: /usr/local/etc/ntopng.conf is not readable.
/usr/local/etc/rc.d/ntopng: WARNING: failed precmd routine for ntopng
thank for your help
MetaWatch devices provide Port and Device ID. This is supported by PF_RING.
It is requested to export those information as part of flows over ZMQ.
Hello,
I'm using the following command to run n2disk:
n2disk -I -A /var/index_folder -p 1024 -b 1024 -i nt:1 -n 1000 -m 1000 -t 15 -O /tmp -o /disco04
For some reason the pcaps erased from /disco04 dir are not being saved on the tmp folder specified on -O option. Did I forget some argument?
Thanks in advance
Updated system to RPM:
n2disk-3.5.200609-5207.x86_64
Testing fix from Issue #28
npcapextract -t /var/log/he/timeline -b "2020-06-06 13:25:43" -e "2020-06-06 13:27:55" -l
/var/log/he/1591462478.559411/1591464452.656624.pcap
npcapmove /var/log/he/1591462478.559411/1591464452.656624.pcap /var/log/he/pp /var/log/he/pp/timeline
Files moved to:
/var/log/he/pp/1591464452.656624.pcap
/var/log/he/pp/1591464452.656624.pcap.idx
/var/log/he/pp/1591464452.656624.pcap.idx.timeline
/var/log/he/pp/1591464452.656624.pcap.timeline
/var/log/he/pp/timeline/2020/06/06/13/20/1591464452.656624.pcap
/var/log/he/pp/timeline/2020/06/06/13/20/1591464452.656624.pcap.idx
npcapextract -t /var/log/he/timeline -t /var/log/he/pp/timeline -b "2020-06-06 13:25:43" -e "2020-06-06 13:28:55" -l
/var/log/he/1591462478.559411/1591464487.478214.pcap
/var/log/he/1591462478.559411/1591464522.296704.pcap
../../../../../../1591464452.656624.pcap
npcapmove still creates relative paths for the links.
Trying to use npcapextract to reference these files is difficult because the output between n2disk created timelines and npcapmove timeline returns is different.
Thanks for looking into this again.
hello everyone.
i just started using licensed ntop products. the ntop and nprobe service is running on my system smoothly. but i'm unable to start n2disk. the log is showing WARNING: Unable to lock memory [Cannot allocate memory]
can anyone help me with the issue.
Red Hat Enterprise Linux release 8.2 (Ootpa)
Kernel: 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Fri Oct 16 13:38:49 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
Versions (installing from https://packages.ntop.org/centos/) :
n2disk x86_64 3.7.201117-5265
pfring x86_64 7.9.0-3273
pfring-dkms noarch 7.9.0-3273
pfring-drivers-zc-dkms noarch 7.9.0-3273
ixgbe-zc noarch 5.5.3.3273-dkms
trace.log
We are running with licensed N2DISK and PF_RING ZC (ixgbe driver on Intel X520 10G Fiber Card),
Anytime running n2disk with the --disk-limit when the limit is reached n2disk doesn't process any more packets/dump any more pcaps.
We've have run the command many ways, with different --disk-limits and other options (like CPU thread binding, timeline creation etc) and saving the output to a different disk partition all with the same results.
Test command run:
n2disk -i zc:enp5s0f0 -o /var/log/n2disk -Q /var/log/n2disk-logs/n2disk-events.log --disk-limit 2% -v --trace-log /tmp/trace.log
Also have to "kill -9" the n2disk process to end it after it reaches/exceeds the 2% limit.
Including the trace output.
Where can I find the options for the --packet-slicing flag? Does this take a TCP/IP layer, like "4" or a specific layer like "udp"?
I'm trying to slice packets after the tcp or udp header (with a -F of ip) without running a secondary "cleaner script". Right now I'm just using snaplen 200.
n2disk -h shows：
Welcome to n2disk v.3.1.180604 (r4961) [Nehalem]
Copyright 2009-2018 ntop.org
SystemID: CB78668F7104A212
Then I use CB78668F7104A212 to generate the license file. However when I run
ip netns exec 154 n2disk -i eth4 -o /storage -b 1024 -p 512 --di sk-limit 50% -I -A /storage
05/Jun/2018 20:59:08 [n2disk:5664] WARNING: Invalid license found for 339B9EE771 04A212
05/Jun/2018 20:59:08 [n2disk:5665] WARNING: (See /etc/n2disk.license) [Missing l icense file]
05/Jun/2018 20:59:08 [n2disk:6177] WARNING: ************************************ ***************
05/Jun/2018 20:59:08 [n2disk:6178] WARNING: ** **
05/Jun/2018 20:59:08 [n2disk:6179] WARNING: ** Switching to DEMO MODE due to li cense error **
05/Jun/2018 20:59:08 [n2disk:6180] WARNING: ** **
05/Jun/2018 20:59:08 [n2disk:6181] WARNING: ************************************ ***************
05/Jun/2018 20:59:08 [n2disk:6182] WARNING: ** Dumping will stop after 5 min **
05/Jun/2018 20:59:08 [n2disk:6184] WARNING: ************************************ ***************
05/Jun/2018 20:59:08 [n2disk:6188] Welcome to n2disk v.3.1.180604 (r4961) [Nehal em]
05/Jun/2018 20:59:08 [n2disk:6218] Running on 1 node(s) system with 4 core(s). N UMA affinity set to node -1.
05/Jun/2018 20:59:08 [n2disk:6284] Using PF_RING for packet capture
05/Jun/2018 20:59:08 [n2disk:6313] Multithread support enabled
05/Jun/2018 20:59:08 [n2disk:6426] Storage limit set to 71.07 GB, 0.00 GB are in use, disk size is 142.13 GB
05/Jun/2018 20:59:08 [n2disk:6479] Dump files max size is set to 512 MB
05/Jun/2018 20:59:08 [n2disk:6494] Buffer memory is set to 1024 MB (x 2 pcap fil es)
05/Jun/2018 20:59:08 [n2disk:6534] Using directory /storage for dump files
05/Jun/2018 20:59:08 [n2disk:6552] Up to 100 files will be written per folder
05/Jun/2018 20:59:08 [n2disk:6558] Dump files max duration is set to 600 sec
05/Jun/2018 20:59:08 [n2disk:6598] Dumping data in 0.1 MB chunks
05/Jun/2018 20:59:08 [n2disk:6643] Index processing memory is set to 424 MB (x 2 index files)
Killed
Invalid license for 339B9EE77104A212, the system ID is CB78668F7104A212, it is not the same. And at the end, the process is killed.
Thanks a lot.
n2disk computes flows by scanning files as soon se they are ready in memory, this creates bursts in exported flows when the file size is high and there are many flows. This should be improved to avoid message loss on the collector.
After updating n2disk for 3.2.191128-5095_amd64 version, the index files are not being persisted anymore when using -O (archive) option
This should work similar to --stop-on-limit which is based on --max-file-len/--max-file-duration/--max-file-packets, but n2disk should stop when it usually starts overwriting old data.
The npcaprepair script which is used to rebuild the timeline does not support 'npcap' compresed files at the moment, it is required to add support for those.
The archiving mode should be extended to either:
Statistics in /proc/net/pf_ring/stats/* for each interval show throughput of an arbitrary amount (a spike, then back to 0.00Mpps and 0.00Gbps). The dumpedBytes counter reliably increments but the throughput doesn't.
Note: at the moment n2disk computes packets/bytes in the capture loop in packet mode only, while in segment mode (which is what Fiberblaze uses) statistics are read from the adapter to avoid scanning all segments. This seems to cause this issue (it seems statistics are not constantly updated by the adapter, this need to be verified).
Hi,
We are planning to have n2disk(10G) deployed for our customer site. Two questions we are having are:
Is it suitable to have both nProbe and n2disk install and function in one server appliance? If yes, what would the minimum hardware spec you recommend?
Does the n2disk parameter --index-tunnel-content support GRE outer/inner IP index?
Please advise. Thank you.
n2disk is able to export flows over ZMQ to ntopng, however there is no statistics about exported flows, it would be useful to add them to /proc/net/pf_ring/stats/*
Hi. I'm using n2disk and ntopng, followed this article https://www.ntop.org/n2disk/combining-traffic-recording-with-visibility-at-100-gbps/ . It work very well but when I set snaplen to 256 to reduce pcap size, the ntopng cannot detect Speedtest Application flow.
I'm facing the following problem with n2disk1g v.3.4.200207 (r5184) :
For some reason I still don't know, part of my traffic read by n2disk is coming with invalid timestamps, for example: "4102363817.1799190". I confirmed this is the received timestamp with wireshark.
This dues to some n2disk indexes being created on a wrong way, with a future date:
ls -hal index_folder
total 52K
drwxr-xr-x 13 user root 4.0K May 4 00:14 .
drwxr-xr-x 3 user docker 4.0K Dec 16 16:00 ..
drwxr-x--- 3 user docker 4.0K Dec 16 18:02 2019
drwxr-x--- 5 user docker 4.0K May 1 00:00 2020
drwxr-x--- 3 user docker 4.0K May 2 21:48 2023
drwxr-x--- 3 user docker 4.0K May 4 00:14 2026
drwxr-x--- 3 user docker 4.0K May 2 22:01 2029
drwxr-x--- 3 user docker 4.0K May 3 22:01 2030
drwxr-x--- 3 user docker 4.0K May 2 22:01 2031
drwxr-x--- 3 user docker 4.0K May 3 03:27 2043
drwxr-x--- 3 user docker 4.0K May 3 22:02 2046
drwxr-x--- 3 user docker 4.0K May 3 22:37 2068
drwxr-x--- 3 user docker 4.0K May 3 00:02 2099
At first, this wouldn't be a problem and I could just ignore those invalid indexes, since they rarely happen. But, sometimes, all my pcaps (42TB of data) are deleted and I believe this is being done by n2disk during it's automatic rotate. I'd like to know if these pcaps being deleted could be related with the timestam problem. I imagine that depending on the way this rotate is done, if n2disk deletes all indexes previous to 2099, for example, this could cause this lost of data.
I'm using the following command to run n2disk:
n2disk1g -I -P /var/run/n2disk/n2disk.pid -A index_folder -p 1024 -b 1024 -i nt:stream0 -n 5000 -m 5000 --disk-limit 93% -t 15 -o /disco05 -o /disco06 -o /disco07
My ntservice configuration file:
NumWorkerThreads = 3 # 1 .. 100
SDRAMFillLevelWarning = 80, 100 # X1, X2, X3, X4
TimestampFormat = PCAP # NATIVE - NATIVE_NDIS - NATIVE_UNIX* - PCAP - PCAP_NS
TimestampMethod = EOF # UNKNOWN - SOF - EOF*
TimeSyncOsTimeReference = None # None* - adapter-0 - adapter-1 - adapter-2 - adapter-3 - adapter-4 - adapter-5 - adapter-6 - adapter-7
[Logging]
LogBufferWrap = wrap # wrap* - nowrap
LogFileName = /tmp/Log3G_%s.log # String
LogMask = 7 #
LogToFile = false # true/false
LogToSystem = true # true/false
[Adapter0]
AdapterType = NT20E3_2_PTP # NT4E - NT20E - NT4E_STD - NTPORT4E - NT20E2 - NT40E2_1 - NT40E2_4 - NT4E2_BP - NT4E2_PTP - NT20E2_PTP - NT20E3_2_PTP - NT40E3_4_PTP - NT100E3_1_PTP
DiscardSize = 16 # 16 .. 63
HostBufferHandlerAffinity = -2 # -2 .. 31
HostBufferPollInterval = default # default* - 100 - 250 - 500 - 1000
HostBufferSegmentTimeOut = default # default* - 100 - 250 - 500 - 1000
IfgMode = NS # NS* - BYTE
MaxFrameSize = 9018 # 1518 .. 10000
OnBoardMemorySplit = Even # Even* - Proportional
HostBuffersRx = [14,32,-1] # [x1, x2, 0], ...
HostBuffersTx = [2,32,-1] # [x1, x2, 0], ...
PacketDescriptor = PCAP # PCAP - NT* - Ext7 - Ext8 - Ext9
SofLinkSpeed = 10G # 100M - 1G - 10G
Profile = Capture # None* - Capture - Inline - CaptureReplay - TrafficGen
TimeSyncConnectorExt1 = NttsIn # None - NttsIn* - PpsIn - NttsOut - PpsOut - RepeatInt1 - RepeatInt2
TimeSyncConnectorInt1 = None # None* - NttsIn - PpsIn - NttsOut - PpsOut - RepeatExt1 - RepeatInt2
TimeSyncConnectorInt2 = None # None* - NttsIn - PpsIn - NttsOut - PpsOut - RepeatExt1 - RepeatInt1
TimeSyncNTTSInSyncLimit = 5000 # 1 .. 4294967295
TimeSyncOSInSyncLimit = 50000 # 1 .. 4294967295
TimeSyncPPSInSyncLimit = 5000 # 1 .. 4294967295
TimeSyncReferencePriority = Ext1, FreeRun # FreeRun* - PTP - Int1 - Int2 - Ext1 - OSTime
TimeSyncTimeOffset = 0 # 0 .. 1000000```
Thanks in advance.
I am running n2disk on a super-micro server running Ubuntu 20.04, linux kernel 5.8.0-55-generic. When I try to specify multiple reader threads with core affinities, n2disk crashes and produces a segmentation fault. I have provided the log below.
sudo n2disk -R 0,1,2,3 -S 4 -zI 5,6 -C 256 -b 65536 -p 10240 -L -i ntxs1 -o /mnt/storage -e 1
17/Jun/2021 16:47:28 [n2disk.c:6250] [DEBUG] Simulating packet capture
17/Jun/2021 16:47:28 [n2disk.c:6718] Welcome to n2disk v.3.6.210604 (r5261) [CPU 830F1]
17/Jun/2021 16:47:28 [n2disk.c:6751] Running on 2 node(s) system with 64 core(s). NUMA affinity set to node -1.
17/Jun/2021 16:47:28 [n2disk.c:6811] Using PF_RING for packet capture
17/Jun/2021 16:47:28 [n2disk.c:6833] WARNING: If you are using standard drivers (packet capture via kernel) please disable time-pulse thread
17/Jun/2021 16:47:28 [n2disk.c:6836] Multithread support enabled
PS. Is there documentation on specifying multiple reader threads?
Packets are captured with Napatech in chunk mode. A chunk can contain packets belonging to two different slots, in that case the slot is currently copied in the pcap belonging to the first slot. This should be handled when dumping to disk by splitting a chunk (or when extracting traffic).
What is the difference between the -o (save) and -O (archive) flag? Is there a forum or better place to see answers like this?
I'm currently running n2disk without a license, just for testing, so the service goes down at every 5 minutes. I have napatech running inside a container and n2disk running inside another container, both services are being orchestrated by docker swarm. At the beginning n2disk works very well, capturing all my network traffic without dropping any packet, after 5 minutes the n2disk service goes down (as expected) and swarm brings the service back, this process is repeated indefinitely. After some time (it may takes minutes or hours), without any special event or throughput peak, napatech buffer reaches 100% and n2disk stops not only recording packets but also stops restarting at every 5 minutes (The service keeps up until I manually kill the process). Restart n2disk service doesn't solve the problem, as soon as n2disk is up, napatech buffer reaches 100% again and the problem remains. The services only gets back to the expected behavior after killing napatech AND n2disk service.
Below the output of /proc/net/pf_ring/stats/16004-none.383 file:
Duration: 0:02:46:01:446
Throughput: 0.00 Mpps 0.00 Gbps
Packets: 0
Filtered: 0
Dropped: 23371139
Bytes: 0
DumpedBytes: 0
DumpedFiles: 0
SlowSlavesLoops: 0
SlowStorageLoops: 0
CaptureLoops: 0
FirstDumpedEpoch: 0
LastDumpedEpoch: 0
It should be easy to generate .pcapng when extracting traffic from pcap files with npcapextract, this is useful to encode in the per-packet interface id metadata the dump set index in case of multiple timelines/interfaces.
We are interested in performance and health monitoring of the individual services. Unfortunately the log format for multithreaded capture is multiline. Any chance we could have a feature added to support json?
19/Oct/2016 19:24:52 [n2disk.c:863] Proc Stats:
Duration: 0:00:46:42:420
Throughput: <redacted> Mpps <redacted> Gbps
Packets: <redacted>
Filtered: <redacted>
Dropped: <redacted>
Bytes: <redacted>
DumpedBytes: <redacted>
DumpedFiles: <redacted>
SlowSlavesLoops: <redacted>
SlowStorageLoops: <redacted>
I would like to know if there is any intention or ongoing efforts to have n2disk working in FreeBSD to some capacity.
Restart n2disk and it starts processing traffic then it stops capturing traffic. The avg bps goes to 0. Using cardstat on the silicom card see then PRB starting to fill up until it gets to 100% and then it discards traffic
Duration: 0:00:01:09:008
Throughput: 0.15 Mpps 0.48 Gbps
Packets: 10612119
Filtered: 10612119
Dropped: 0
Bytes: 4293166464
DumpedBytes: 0
DumpedFiles: 0
SlowStorageLoops: 0
CaptureLoops: 4411
FirstDumpedEpoch: 1570001060
LastDumpedEpoch: 1582888323
BytesOnDisk: 608316037120923
TimelinePath: /local/isilon/pcap/capture101/index/
Duration: 0:00:01:10:008
Throughput: 0.11 Mpps 0.38 Gbps
Packets: 10726239
Filtered: 10726239
Dropped: 0
Bytes: 4341233172
DumpedBytes: 0
DumpedFiles: 0
SlowStorageLoops: 0
CaptureLoops: 4460
FirstDumpedEpoch: 1570001060
LastDumpedEpoch: 1582888323
BytesOnDisk: 609882875515012
TimelinePath: /local/isilon/pcap/capture101/index/
Duration: 0:00:01:11:008
Throughput: 0.00 Mpps 0.00 Gbps
Packets: 10726239
Filtered: 10726239
Dropped: 0
Bytes: 4341233172
DumpedBytes: 0
DumpedFiles: 0
SlowStorageLoops: 0
CaptureLoops: 4460
FirstDumpedEpoch: 1570001060
LastDumpedEpoch: 1582888323
BytesOnDisk: 611584728367675
TimelinePath: /local/isilon/pcap/capture101/index/
Original request at ntop/ntopng#3192
I am trying to write a bash script to run multiple instances of n2disk, trying to bind one port to each file system.
I have two napatech NICs with two 40G ports each, and four drives. The four ports are called ntxs0, ntxs1, ntxs2, ntxs3 and my storage drives are named storage0, storage1, storage2, storage3. I would like to save all the data coming into port ntxs0 to be saved in storage0, all the data in ntxs1 to be saved in storage1, and so on. I have provided the commands my script runs below, when I run it the data from all four ports are scattered between the four filesystems. I.E the pcap files that are saved in the filesystems all contain packets from all four ports. Why is this happening? I thought I could bind each port to a filesystem.
sudo n2disk -i $SFP_PORT_A -o $SFP_PORT_A_STORAGE -s $PACKET_SIZE -p $MAX_FILE_LENGTH -b $BUFFER_LENGTH -q $POLL_DURATION -C $CHUNK_LENGTH -S 0 -c 1 -w 2
sudo n2disk -i $SFP_PORT_B -o $SFP_PORT_B_STORAGE -s $PACKET_SIZE -p $MAX_FILE_LENGTH -b $BUFFER_LENGTH -q $POLL_DURATION -C $CHUNK_LENGTH -S 3 -c 4 -w 5
sudo n2disk -i $SFP_PORT_C -o $SFP_PORT_C_STORAGE -s $PACKET_SIZE -p $MAX_FILE_LENGTH -b $BUFFER_LENGTH -q $POLL_DURATION -C $CHUNK_LENGTH -S 6 -c 7 -w 8
sudo n2disk -i $SFP_PORT_D -o $SFP_PORT_D_STORAGE -s $PACKET_SIZE -p $MAX_FILE_LENGTH -b $BUFFER_LENGTH -q $POLL_DURATION -C $CHUNK_LENGTH -S 9 -c 10 -w 11
It seems that the disk-limit parameter ignores the -I parameter when specified.
n2disk -i zc:enp101s0f0 -I/index/enp101s0f0 -o /pcap -A /timeline/enp101s0f0 -b 8192 -p 1024 -S 0 -c 1 -w 2 -z 3,4,5 -Z -v
13/Mar/2020 11:27:19 [n2disk.c:2900] [writer][#0] Creating pcap file [idx 1] /pcap/30.pcap
13/Mar/2020 11:27:20 [n2disk.c:2970] [writer][#0] Creating index file /index/enp101s0f0/30.pcap.idx
n2disk -i zc:enp101s0f0 -I/index/enp101s0f0 -o /pcap -A /timeline/enp101s0f0 -b 8192 -p 1024 -S 0 -c 1 -w 2 -z 3,4,5 -Z -6 4098 -v
13/Mar/2020 11:31:47 [n2disk.c:2900] [writer][#0] Creating pcap file [idx 1] /pcap/1584095503.310082/1584095505.344544.pcap
13/Mar/2020 11:31:48 [n2disk.c:2970] [writer][#0] Creating index file /pcap/1584095503.310082/1584095505.344544.pcap.idx
For our customer it would be VERY helpful, if on the "n2disk Status" web-page, the Ethernet NIC names (in bold and big font, e.g. eth1) could be overwritten with a different string, that has more meaning to our customer.
Therefore we'd like to ask you you could consider such a feature in a future n2disk release.
When a capture begins, it would be great to include the initiating user in the log entry. In a multiuser environment, it would be useful to know who started the last dump.
Thanks!
I'm using two machines to reproduce this problem. One of them it's used just for sending the packets via tunnel and on the other one I'm running npcapextract and n2disk.
These are the steps to reproduce this issue:
n2disk1g -I -P /var/run/n2disk/n2disk.pid -G 1 -A index_folder -p 1024 -b 1024 -i ens18f0 --disk-limit 93% -t 15 -a -o /disco03 -o /disco04
sudo modprobe ip_gresudo ip link add name ${tunnel_name} type gretap local 192.168.10.2 remote 192.168.10.3sudo ip link set ${tunnel_name} upsudo tcpreplay -i ${tunnel_name} --mbps 10 -K --loop 1 gre-within-gre.pcapnpcapextract -u root -t index_folder -b '2020-05-25 16:18:56' -e '2020-05-25 16:19:31' -o test.pcap -f '(inner host 224.0.0.9 and inner port 520) and (inner host 3.3.3.2 and inner port 520)'npcapextract -u root -t index_folder -b '2020-05-25 16:18:56' -e '2020-05-25 16:19:31' -o test.pcap -f '(inner host 224.0.0.9)'npcapextract -u root -t index_folder -b '2020-05-25 16:18:56' -e '2020-05-25 16:19:31' -o test.pcap -f '(inner host 3.3.3.1)'npcapextract -u root -t index_folder -b '2020-05-25 16:18:56' -e '2020-05-25 16:19:31' -o test.pcap -f '(inner host 3.3.3.2)'Neither of the above extractions worked
Also, when I run npcaprintindex on the machine where n2disk is running, i get:
1917) len: 200, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, 192.168.10.2:5247 -> 192.168.10.3:5247, l7proto: Unknown/Unknown, not ip
1921) len: 200, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, 192.168.10.2:49772 -> 192.168.10.3:9995, l7proto: Unknown/Unknown, not ip
1923) len: 200, vlan: 0, vlan_qinq: 0, ipv4, proto: 6, 192.168.10.2:38666 -> 192.168.10.3:9100, l7proto: Unknown/Unknown, not ip
As you can see, n2disk is not identifying the tunneled ipv4.
I can assure you that the problem is not with the tunnel, since I'm monitoring the network on the first machine and I can see the traffic I was replaying.
This is the pcap I used for these tests:
gre-within-gre.pcap.zip
n2disk version: n2disk v.3.4.200414 (r5191)
pfring version: 7.6.0 (7.6.0-stable:0e07b27c0d593174584de79e273470366a91d307)
The timestamp of the packet are not accuarate enough.
when i check the captured file. I found a lot of the packet have the same tiemstamp. I think it is not correct.
since the packet length is 1440-byte, the NIC bandwidht is 10Gb/s. the interval between two packts are at least 1440*8/10 = 1.152 us. However, below is the timestamps of a group of 12 countiuous packets. The packet length are all 1440-byte.
09:47:42:960246
09:47:42:001590
09:47:42:001592
09:47:42:001592
09:47:42:001592
09:47:42:001593
09:47:42:001593
09:47:42:001593
09:47:42:001594
09:47:42:001594
09:47:42:001595
09:47:42:001595
3 packets are stamped with the same timestamp. which mean in 1us, three packets are sent. which is not correct.
I search from the internet.
https://www.elvidence.com.au/understanding-time-stamps-in-packet-capture-data-pcap-files/
they talk about the timestamp accuracy. It says:
The accuracy of a timestamp depends on many factors including the performance of the system used to capture packets, its operating system, configuration, and more.
"On a Microsoft Windows computer the WinPCAP library is synced with the computer clock only at the beginning of the packet capture. As a result, timestamps and the system time may differ by a few seconds or milliseconds under heavy system load, high CPU utilisation or prolonged packet captures.
On Linux and most POSIX systems where libpcap is used, the timestamps are supplied by the kernel."
How can I use n2disk to get a very acurate timestamp? or is it impossble？ thanks a lot.
In n2disk not exist any description about interface names format.
Option :
[--interface|-i] <device> | Ingress packet device.
As i understand it is possible to start n2disk using :
pcap:eth0 , tc:eth0, zc: eth0 . But not exist any place where described all such options.
Also not described how to use it without PF_RING or PF_RING ZC and how to use it with just default libpcap.
As i see docker image try to use PF_RING even with "pcap:eth0"
n2disk should use the nDPI serializer to build JSON/TLV flow data to be exported over ZMQ, as the other products are doing.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.