Comments (14)
Hi,
can you provide some more information?
- Are there any netmap-related error messages in the kernel log?
- where is tcpreplay spinning? e.g., is it continually returning from the
poll() with an error?
Original comment by giuseppe.lettieri73
on 16 Aug 2014 at 4:06
- Changed state: Started
from netmap.
Hi, I am the original reporter of this issue.
I first noticed that tcpreplay 4.0.4 would hang in netmap mode while tcpdump is
also running in netmap mode.
I have since then updated to tcpreplay version 4.0.5 beta 3. The symptoms have
changed a bit, but the use
case is still not working
Running tcpreplay 4.0.5 beta 3
Setup:
Eth16 and Eth17 are connected back to back, so traffic leaving eth16 will
arrive eth17. Goal is to play Pcap from
eth16 using netmap and have tcpdump running with netmap on eth17 (saving to
RAMdisk). Both eth16 and eth17 are ixgbe
interfaces running netmap patched drivers.
Pre-req:
I can run both tcpreplay and tcpdump with netmap seperately (i can get good
performance on both). But I can't seem to
run them at the same time.
Step 1:
Launches tcpdump with netmap first
devuser@UTraffic22:~$ sudo taskset -c 3 sudo tcpdump -s 0 -nei netmap:eth17 -w
/mnt/tmpfs/pk1.pcap -c 100000
[sudo] password for devuser:
pcap_netmap_activate device netmap:eth17 priv 0x18b6c50 fd 3 ports 0..11
pcap_netmap_ioctl eth17 ioctl 0x8913 returns 0
pcap_netmap_ioctl eth17 ioctl 0x8914 returns 0
tcpdump: listening on netmap:eth17, link-type EN10MB (Ethernet), capture size
262144 bytes
Step 2:
Launch tcpreplay 4.0.5 beta 3 playing pcap, it runs and exits
devuser@UTraffic22:~$ sudo taskset -c 11 sudo tcpreplay --netmap -i eth16 -M
8000 -K /tmp/gre_ipv4.pcap
[sudo] password for devuser:
Switching network driver for eth16 to netmap bypass mode... done!
File Cache is enabled
Actual: 1000000 packets (1020000000 bytes) sent in 1.00 seconds.
Rated: 999975490.7 Bps, 7999.80 Mbps, 980368.12 pps
Flows: 1000000 flows, 980368.12 fps, 1000000 flow packets, 0 non-flow
Statistics for network device: eth16
Attempted packets: 1000000
Successful packets: 1000000
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0
Switching network driver for eth16 to normal mode... done!
Step 3:
However, the first tcpdump instance (which would exit upcon receiving 1000000
packets with the "-c" option)
did not exit
devuser@UTraffic22:~$ sudo taskset -c 3 sudo tcpdump -s 0 -nei netmap:eth17 -w
/mnt/tmpfs/pk1.pcap -c 100000
[sudo] password for devuser:
pcap_netmap_activate device netmap:eth17 priv 0x18b6c50 fd 3 ports 0..11
pcap_netmap_ioctl eth17 ioctl 0x8913 returns 0
pcap_netmap_ioctl eth17 ioctl 0x8914 returns 0
tcpdump: listening on netmap:eth17, link-type EN10MB (Ethernet), capture size
262144 bytes
^C0 packets captured
0 packets received by filter
0 packets dropped by kernel
pcap_netmap_ioctl eth17 ioctl 0x8913 returns 0
Step 4:
Re-run step 1 and 2 but with tcpdump NOT in netmap mode (tcpreplay in netmap
mode), tcpdump logs the packets just fine
devuser@UTraffic22:~$ sudo taskset -c 3 sudo tcpdump -s 0 -nei eth17 -w
/mnt/tmpfs/pk1.pcap -c 100000
tcpdump: listening on eth17, link-type EN10MB (Ethernet), capture size 262144
bytes
100000 packets captured
100257 packets received by filter
0 packets dropped by kernel
I tried 4 various combinations between tcpreplay and tcpdump (with and without
netmap mode), below are the results
Here is another interesting observation, below is a summary of the three
combinations:
tcpreplay (netmap enabled) -> tcpdump (netmap enabled) = Does not work
tcpreplay (netmap enabled) -> tcpdump (no netmap) = Works fine
tcpreplay (no netmap) -> tcpdump (netmap enabled) = Does not work
tcpreplay (no netmap) -> tcpdump (no netmap) = Works fine
I also noticed that the TX counter for my network interface never seems to
increment when running in netmap mode.
Examming dmeg and kerner.log, I don't see anything strange, with no errors in
particular
[894792.609623] ixgbe 0000:44:00.1 eth17: detected SFP+: 6
[894792.933803] ixgbe 0000:44:00.1 eth17: NIC Link is Up 10 Gbps, Flow Control:
RX/TX
[894819.342554] 026.358522 [ 753] generic_netmap_dtor Restored native NA
(null)
[894819.377905] 026.393847 [ 753] generic_netmap_dtor Restored native NA
(null)
[894820.711718] ixgbe 0000:44:00.0 eth16: detected SFP+: 5
[894820.851576] ixgbe 0000:44:00.0 eth16: NIC Link is Up 10 Gbps, Flow Control:
RX/TX
[894828.505500] ixgbe 0000:44:00.0 eth16: detected SFP+: 5
[894828.637627] ixgbe 0000:44:00.0 eth16: NIC Link is Up 10 Gbps, Flow Control:
RX/TX
[894836.087561] ixgbe 0000:44:00.1 eth17: detected SFP+: 6
[894836.219567] ixgbe 0000:44:00.1 eth17: NIC Link is Up 10 Gbps, Flow Control:
RX/TX
Original comment by [email protected]
on 18 Aug 2014 at 6:29
from netmap.
I'm running tcpdump with netmap-libpcap
apcon@UTraffic22:~/working/tcpreplay_interactions$ sudo tcpdump --version
[sudo] password for apcon:
tcpdump version 4.6.1
libpcap version 1.6.0-PRE-GIT_2014_08_04
Netmap and netmap-libpcap are both recent clones (August 4th).
I'm running Ubuntu 14.04, running ixgbe driver patched with netmap's ixgbe.patch
apcon@UTraffic22:~/netmap$ sudo ethtool -i eth16
driver: ixgbe
version: 3.15.1-k
firmware-version: 0x80000208
bus-info: 0000:44:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
Below is dmesg in regard to the ixgbe netmap interface
[ 250.241049] 155.199290 [2259] netmap_attach success for eth16
[ 250.284602] systemd-udevd[2487]: renamed network interface eth0 to eth16
[ 250.437586] ixgbe 0000:44:00.1: irq 306 for MSI/MSI-X
[ 250.437605] ixgbe 0000:44:00.1: irq 307 for MSI/MSI-X
[ 250.437622] ixgbe 0000:44:00.1: irq 308 for MSI/MSI-X
[ 250.437638] ixgbe 0000:44:00.1: irq 309 for MSI/MSI-X
[ 250.437660] ixgbe 0000:44:00.1: irq 310 for MSI/MSI-X
[ 250.437677] ixgbe 0000:44:00.1: irq 311 for MSI/MSI-X
[ 250.437693] ixgbe 0000:44:00.1: irq 312 for MSI/MSI-X
[ 250.437709] ixgbe 0000:44:00.1: irq 313 for MSI/MSI-X
[ 250.437725] ixgbe 0000:44:00.1: irq 314 for MSI/MSI-X
[ 250.437742] ixgbe 0000:44:00.1: irq 315 for MSI/MSI-X
[ 250.437758] ixgbe 0000:44:00.1: irq 316 for MSI/MSI-X
[ 250.437775] ixgbe 0000:44:00.1: irq 317 for MSI/MSI-X
[ 250.437791] ixgbe 0000:44:00.1: irq 318 for MSI/MSI-X
[ 250.437839] ixgbe 0000:44:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx
Queue count = 12
[ 250.437974] ixgbe 0000:44:00.1: PCI Express bandwidth of 32GT/s available
[ 250.437976] ixgbe 0000:44:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[ 250.438057] ixgbe 0000:44:00.1: MAC: 2, PHY: 12, SFP+: 6, PBA No: 400900-000
[ 250.438058] ixgbe 0000:44:00.1: 00:e0:ed:49:90:99
[ 250.497171] ixgbe 0000:44:00.1: Intel(R) 10 Gigabit Network Connection
[ 250.497188] 155.455229 [2259] netmap_attach success for eth17
[ 250.539913] systemd-udevd[2487]: renamed network interface eth0 to eth17
[ 250.693937] ixgbe 0000:46:00.0: irq 319 for MSI/MSI-X
[ 250.693956] ixgbe 0000:46:00.0: irq 320 for MSI/MSI-X
[ 250.693973] ixgbe 0000:46:00.0: irq 321 for MSI/MSI-X
[ 250.693989] ixgbe 0000:46:00.0: irq 322 for MSI/MSI-X
[ 250.694005] ixgbe 0000:46:00.0: irq 323 for MSI/MSI-X
[ 250.694021] ixgbe 0000:46:00.0: irq 324 for MSI/MSI-X
[ 250.694037] ixgbe 0000:46:00.0: irq 325 for MSI/MSI-X
[ 250.694053] ixgbe 0000:46:00.0: irq 326 for MSI/MSI-X
[ 250.694072] ixgbe 0000:46:00.0: irq 327 for MSI/MSI-X
[ 250.694088] ixgbe 0000:46:00.0: irq 328 for MSI/MSI-X
[ 250.694104] ixgbe 0000:46:00.0: irq 329 for MSI/MSI-X
[ 250.694120] ixgbe 0000:46:00.0: irq 330 for MSI/MSI-X
[ 250.694136] ixgbe 0000:46:00.0: irq 331 for MSI/MSI-X
[ 250.694180] ixgbe 0000:46:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx
Queue count = 12
[ 250.694315] ixgbe 0000:46:00.0: PCI Express bandwidth of 32GT/s available
[ 250.694317] ixgbe 0000:46:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[ 250.694403] ixgbe 0000:46:00.0: MAC: 2, PHY: 12, SFP+: 5, PBA No: 400900-000
[ 250.694405] ixgbe 0000:46:00.0: 00:e0:ed:49:90:9a
[ 250.753364] ixgbe 0000:46:00.0: Intel(R) 10 Gigabit Network Connection
Original comment by [email protected]
on 18 Aug 2014 at 6:37
from netmap.
Hi morphyno, thank you for the report. I need to reproduce the failure locally,
since all the obvious culprits have been ruled out by either some of your
tests, the kernel logs, or my looking at the relevant code in tcpreplay.
Unfortunately I am out of office this week, but there is a chance I can get
remote access to a Linux machine with two ixgbe's connected back-to-back.
Giuseppe
Original comment by giuseppe.lettieri73
on 18 Aug 2014 at 10:35
from netmap.
Hi Giuseppe, thanks for looking into this, let me know if there's anything I
can do to assist
Original comment by [email protected]
on 19 Aug 2014 at 5:23
from netmap.
Hi morphyno,
I finally had access to a suitable machine. Unfortunately, I am not able to
reproduce the problem: I can send with tcpreplay --netmap on one ixgbe and
receive with tcpdump -nei netmap:... on the other.
I have compiled everything from sources, here are the relevant commit hashes:
tcpreplay: da25e37b78f39e3292ddb607a53b1703ba21949f
tcpdump: 29d83dbb61be640899d969357f2a8e1c3e02c7ee
netmap-libpcap: 0890b993dae48767af172b693f2d664181dbe943
For netmap proper, I have tried bot the current 'master' and 'next' branches
available here. Everything works fine.
We need to do some more investigation on your own system, I am afraid. Does
tcpdump with netmap loses all the packets, or just a fraction of them? (i.e.,
does tcpdump exits for some lower value of -c?)
Original comment by giuseppe.lettieri73
on 26 Aug 2014 at 12:30
from netmap.
Hi Giuseppe:
Thanks for looking into this. From my observations, when tcpreplay is using a
sending interface in netmap mode, the receiving tcpdump (in netmap mode) does
not receive any packets.
my netmap-libpcap matches yours, but I'm using official tcpdump release (4.6.1)
and tcpreplay release (4.0.5beta3). I will try again using the same versions
you did (i'm assuming you just downloaded the latest from respective GIT repo's)
Out of curiosity, was your testing conducted on Ubuntu 14.04 as well?
I also have another question, does netmap put he IXGBE in "poll-mode"? I know
for PF_RING, their IXGBE modifications were with NAPI (rx polling) enabled, and
it doesn't look netmap is doing that. Given netmap's great improvement gains,
will putting the driver in poll mode greatly increase overall performance?
Original comment by [email protected]
on 26 Aug 2014 at 6:46
from netmap.
> Out of curiosity, was your testing conducted on Ubuntu 14.04 as well?
yes, I was using Ubuntu 14.04 with kernel 3.13.0-34.
Re polling and NAPI, we are also using NAPI to wake up the receiving process
(which then does all the work in the poll() system call). NAPI does the
switching between polling and interrupts on its own, we have not changed that.
We have also not tried to tune that either, since performance was already good,
but that may change in the future.
Going back to the issue, my failure to reproduce it locally is very bad news,
but we can still try something:
- can you please attach the pcap file you are using? The one I was using was
very simple (a capture of pkt-gen) and that may have not triggered some bug,
e.g., in the handling of packet lengths.
- can you put netmap in verbose mode and see if something interesting shows up
in the kernel logs?
You can enable verbose mode with:
# echo 1 > /sys/module/netmap_lin/parameters/verbose
- is there any way to slow down the rate of the packets sent by tcpreplay, to
see if the problem is only triggered by high packet rates?
Original comment by giuseppe.lettieri73
on 29 Aug 2014 at 7:56
from netmap.
Hi Giuseppe:
For some reason, when I installed a newer version of tcpdump, and built it
against netmap-libpcap, the problem is no longer occurring.
Original comment by [email protected]
on 5 Sep 2014 at 4:38
from netmap.
Hi Giuseppe:
I have somehow recreated this failure scenario while putting together a second
system. I suspect there is an issue with netmap-libpcap and tcpdump linking as
in this case netmap mode for tcpdump (official version 4.6.1) is not working at
all.
I have enabled verbose output for netmap_lin module and I notice while the
tcpdump is in netmap mode, it continuously prints the RX and TX queues.
This is the command I'm using
apcon@UTraffic22:~$ sudo taskset -c 2 sudo tcpdump -s 0 -nei netmap:eth3 -w
/mnt/tmpfs/eth3.pcap -c 1
pcap_netmap_activate device netmap:eth3 priv 0x24dbc50 fd 3 ports 0..11
pcap_netmap_ioctl eth3 ioctl 0x8913 returns 0
pcap_netmap_ioctl eth3 ioctl 0x8914 returns 0
tcpdump: listening on netmap:eth3, link-type EN10MB (Ethernet), capture size
262144 bytes
^C0 packets captured
0 packets received by filter
0 packets dropped by kernel
pcap_netmap_ioctl eth3 ioctl 0x8913 returns 0
Attached is the dmesg collected during the tcpdump session (it prints to screen
about every 2 seconds during the process's life)
Original comment by [email protected]
on 8 Sep 2014 at 9:17
Attachments:
from netmap.
Hi Giuseppe:
I know it sounds silly, but what is the correct method of install for
netmap-libpcap? I have done three system installs, same hardware and software,
but with different results:
case 1: tcpdump and tcpreplay both works with netmap
case 2: tcprelay works with netmap but tcpdump does not (recognizes
"netmap:ethX", but does not capture packes)
case 3: tcpdump does not recognize the "netmap:ethX" interface, error message is
tcpdump: netmap:eth0: No such device exists
(SIOCGIFHWADDR: No such device)
I'm suspecting this is more of an install issue and I'm not doing something
correctly.
This is how I installed netmap-libpcap
1) make sure netmap_lin is loaded
2) glone netmap-libpcap
3) ./configure && make && make install
4) Install tcpdump with standard (configure && make && make install)
I'm also curious about how netmap maps the interfaces, because I have a large
number of interfaces (16 NIC's), I have a modified 70-net-persistent file so
the interfaces gets renamed (see dmesg example below), could this cause an
issue with netmap?
[11919.671298] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx
Queue count = 12
[11919.671433] ixgbe 0000:06:00.0: PCI Express bandwidth of 32GT/s available
[11919.671435] ixgbe 0000:06:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[11919.671522] ixgbe 0000:06:00.0: MAC: 2, PHY: 12, SFP+: 5, PBA No: 400900-000
[11919.671523] ixgbe 0000:06:00.0: 00:e0:ed:49:90:94
[11919.672877] ixgbe 0000:06:00.0: Intel(R) 10 Gigabit Network Connection
[11919.672932] 147.576627 [2259] netmap_attach success for eth0
[11919.725880] systemd-udevd[21421]: renamed network interface eth0 to eth4
Original comment by [email protected]
on 19 Sep 2014 at 12:45
from netmap.
Ok, I figured out the discrepancies. For the last case, I didn't copy the
netmap/sys/net headers to "/usr/include". As of the second case, it is directly
related to my set cpu affinity script. This issue can be closed
Original comment by [email protected]
on 23 Sep 2014 at 6:05
from netmap.
Thanks for the update, I am closing the issue now.
Original comment by giuseppe.lettieri73
on 25 Sep 2014 at 7:18
- Changed state: WontFix
from netmap.
Hi all,
my problem while installing netmap,
tcpdump does not recognize the "netmap:ethX" interface, error message is
tcpdump: netmap:eth0: No such device exists
(SIOCGIFHWADDR: No such device)
Without copying "netmap/sys/net" headers to "/usr/include",netmap support is
not there.when I try to configure in libpcap,
./configure | grep netmap
I am getting as following,
checking for net/netmap_user.h... no
configure: netmap is not supported
can anyone help me to install it??
Thanks in advance...
Original comment by [email protected]
on 9 Mar 2015 at 6:50
from netmap.
Related Issues (20)
- Running multiple pkt-gen applications for receiving packets HOT 1
- OpenWRT Makefile does not work
- [PATCH] Allow use of pipes in pkt-gen through "-E" HOT 2
- Is it possible to make netmap kernel version agnostic HOT 2
- Run Netmap in a Fedora VM running in VirtualBox HOT 2
- Netmap on CentOS 7 work abnormally HOT 1
- No packets received no the receiver side in the simple pkt-gen example HOT 1
- Something wrong with Netmap bridge HOT 2
- netmap crash on ubuntu 14.04 after weeks of continuous use
- running netmap-libpcap linked tcpdump slows pkt-gen and drops packets
- Problems installing Netmap on ARM (Nvidia Jetson TK1) HOT 4
- Conflict with Boost Threads, patch provided
- Can't build ixgbe driver for Debian 7 Wheezy HOT 3
- Kernel oops when trying to open netmap with incrorrect hardware RING number HOT 2
- Debian 7.8, e1000e plus netmap build error HOT 2
- get in touch HOT 4
- examples/bridge connect to host stack on e1000 only see net->host,no host->net HOT 2
- Intel ixgbe driver from Sourcesorge
- Optimization on linux - hugetlb pages support HOT 3
- Kernel panic with Fedora 21 (Linux 3.19 kernel) and e1000 driver
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from netmap.