pypacker stops working with large amounts of data about pypacker

[mike01]

Could be an exception in the verdict callback. I've added an exception handling in the newest master commit which should warn about such problems (packets will now be dropped in this case)

[svolpe]

I just updated my py-packer to the latest version. It still stops sending data after a period and is not giving any errors. Is there anything I need to do to test it? I'm running on a resource limited target (embedded ARM). I've monitored its RAM and CPU and have seen no corrilation to them when the pychecker stops passing data.
Is there any other way I can debug the problem?

[mike01]

Well then I assume there is a problem on the sending side. Without code I can't say anything about that.

[svolpe]

There is actually no custom code. I'm running apache on the server side hosting 100MB binary file and then use wget on the client side to pull it. Its just a stress test system to test the implementation of the pypacker on the embedded system.

[mike01]

So what's the exact Setup? Something like this?
[PC: wget -> pypacker:nfqueue] -> Network -> [Target: pypacker:nfqueue -> HTTP-server]
Did you make any basic network debugging? Checking traces via wireshark or similar?
Is the conenction reset or timed out? Any error messages anywhere? I can't say anything
without more information.

[svolpe]

The network looks like this:
server(apache)->bridge device(py-packer on it)->WAN->client
I have run wireshark and what I'm seeing is everything works great and then, for some reason, the bridge device (running py-packer) stops forwarding the ACKs from the client to the server. I'm not 100% sure if also stops transmitting data from the server to the client. I beleive it does as I see the client continually trying to send the same packet but its not coming out the other end of the bridge. There are tons of retries but nothing gets through the bridge. After stopping/starting the py-packer on the bridge it starts working again. I tried stopping/starting the py-packer script on the client but that does not recover the problem. My guess is it must somehow be caused by the limitations of resources on the embedded bridge device but can't seem to find the smoking gun yet. I was also thinking about using the multiprocessing capabilities in python as it is a quad core ARM device. My first attempt at doing that with the py-packer interceptor did not work. I will continue to dig in. Once I get the multiprocessor working I will share the code for others. Any thoughts on this would be greatly appriciated.

[svolpe]

I ran python-netfilterqueue on the same system and I do not experiance the same issue. I'm not really interested in using that as my approach because py-packer is such a nicer API to use.

Just to summerize:
I get no errors in the /proc/net/netfilter/nf_log. There is nothing in /var/log/messages. No errors are coming out of the py-packer program. I've used wireshark and it looks like communications stops. The connections are not reset and they just hang (since no packets coming out of the netfilter queues) until they timeout.

Is there anything I can do to debug deeper into py-packer and see where the problem lies?

[svolpe]

Following the instructions on "Enlarge receive/send buffers to get max performance." made a difference, now it does not stall and not recover. It still has small stalls but seems to finish 100% of the time. I've run a 100MB file through it 10 times now.
What supprises me is increasing the number of queues does not seem to have any effect on the performance.

[mike01]

Nice to hear the problem got solved. There can be a multitude of reasons for the "not increasing performance" and the max network bandwidth as bottleneck is just one of them, so it's hard to say what's going on.

[mike01]

Closing due response timeout