Non-Linear response for sequential iteration of large capture about pyshark

[chrconlo]

Not sure if you got a chance to look at this but a look through the code and some debugs it looks like the slowdown is in asyncio; more specifically the stream reader below. Are you seeing the same? Thanks

new_data = yield From(stream.read(self.DEFAULT_BATCH_SIZE))

[chrconlo]

I've backed down to using older rev of pyShark (0.2.6), prior to asyncio, and see vastly better performance although some other things seem broken like frame_info, display filter on capture and getting certain field matches to work. I was able to process 420K packets in under 8 minutes. Thanks

-- Packet Processing Progress Report --
Runtime: 0:07:21.040105 -> Packet Count: [420000]
Processing Loop Time (Single Packet): 0:00:00.001523
BPDU Accounting Dictionary Size: [6463] entries

[KimiNewt]

Yes, it seems asyncio severely reduced performance (on unix anyway, I'm
oddly seeing good performance on windows).
I'll try optimizing it over the weekend hopefully, and if push comes to
shove ill remove asyncio. Conceptually it should not have lower
performance, but we'll see.

On Wednesday, December 17, 2014, chrconlo [email protected] wrote:

I've backed down to using older rev of pyShark (0.2.6), prior to asyncio,
and see vastly better performance although some other things seem broken
like frame_info, display filter on capture and getting certain field
matches to work. I was able to process 420K packets in under 8 minutes.
Thanks

-- Packet Processing Progress Report --
Runtime: 0:07:21.040105 -> Packet Count: [420000]
Processing Loop Time (Single Packet): 0:00:00.001523
BPDU Accounting Dictionary Size: [6463] entries

—
Reply to this email directly or view it on GitHub
#48 (comment).

[KimiNewt]

I've (probably) isolated the problem to: https://github.com/KimiNewt/pyshark/blob/master/src/pyshark/capture/capture.py#L147
What seems to be happening is that a large amount of data (tshark XML) is in the subprocess stdout pipe. We read that one packet at a time, and the XML grows faster and larger as time goes on.
That line copies over what might be a very large string. That took ~40ms on a large cap file I tried.

The solution is probably to extract ALL the packets at once from the data received instead of one at-a-time (we can't use lxml for this as it does not support parsing partial XMLs). I'll try finding a solution for that.

[chrconlo]

Interesting. Any luck finding a solution for this? Thanks

[KimiNewt]

Fixed by PR #66