[Profiling] Humongous CPU with event_stream about marathon-python

[migueleliasweb]

I think the problem lies on this code:

sseclient.py:58

def _event_complete(self):
        return re.search(end_of_field, self.buf) is not None

    def __iter__(self):
        return self

    def __next__(self):
        while not self._event_complete():
            try:
                nextchar = next(self.resp_iterator)
                self.buf += nextchar
            except (StopIteration, requests.RequestException):
                time.sleep(self.retry / 1000.0)
                self._connect()

                # The SSE spec only supports resuming from a whole message, so
                # if we have half a message we should throw it out.
                head, sep, tail = self.buf.rpartition('\n')
                self.buf = head + sep
                continue

[solarkennedy]

Half a million times eh? :(

Surely there is a more efficient way to see if EOF is in the buffer!

[Rob-Johnson]

robj@robj-yelp-mb: % python -m timeit -n 10000 -s "import re; x='\r\n\r\n'; y='\r\r'; z='\n\n'; s='helloab'*100000" "x in s or y in s or z in s"
10000 loops, best of 3: 2.15 msec per loop
robj@robj-yelp-mb:% python -m timeit -n 10000 -s "import re; end_of_field = re.compile(r'\r\n\r\n|\r\r|\n\n'); s='helloab'*100000" "end_of_field.search(s)"
10000 loops, best of 3: 5.69 msec per loop

maybe I've missed something here, but seems using python's built in string matching would be faster? I took the regex from here

[migueleliasweb]

@solarkennedy Yep !
@Rob-Johnson Yep !

The caveat is that Marathon actually returns all tasks from all apps bundled into the event data so in my case where I got 200 tasks you can imagine the size of the buffer.

If we count 1 byte per character, 500k characters (or iterations of 1 char) would be...500kb of data ! OMFG !

In marathon's sse case, the last character will always be the EOF so the code from sseclient would have to iterate over the whole buffer trying to find it until it finally encounters it.

It would be even easier to read the whole buffer and use a simple (yet powerfull) string search with ".find()" or "in".

Ill try to post some benchmarks here, but I think it will be magnitudes faster than 1/2 million searches using regexp.

[migueleliasweb]

I solved this problem on #143 =]

[nuclon]

@migueleliasweb I've already made a pull-request to sseclient which solves this problem:
https://bitbucket.org/btubbs/sseclient/pull-requests/9/response-readline/diff