Flow control can, in some rare cases, not return from a child process to parent about requests-futures

[ross]

Interesting. Happy to take a 👁️ at a PR that sets a timeout, but it might get a little complicated to make sure that plays well with requests timeouts. Let me know what you find.

[asfaltboy]

@ross thanks for the feedback. I used requests.get(timeout=X) and Future.result(timeout=X + GRACE) in case requests' exception in child is thrown after a delay.

Unfortunately, my plan failed, due to my lack of understand of the problem. Even with timeout limits set on the children process, the child still "is suspended in time" and my logs show odd behaviour of the parent continuing, while the pool is not cleared 10 minutes after due finish time. ps shows the child processes in the D state - "Uninterruptible sleep (usually IO)". There must be something I'm missing.

I plan to simplify our usage by removing most callback/result to after all requests are done. This should give a good indication of where things are broken.

One other interesting thing I saw, is when I stop a test by raising KeyboardInterrupt, the exception shows this trace:

  File "my_code", line 594, in some_method
    response = res.result(timeout=CHILD_PROCESS_TIMEOUT)
  File "/home/pavel/.pyenv/versions/3.5.1/lib/python3.5/concurrent/futures/_base.py", line 400, in result
    self._condition.wait(timeout)
  File "/home/pavel/.pyenv/versions/3.5.1/lib/python3.5/threading.py", line 297, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

I wonder if concurrent.futures runs a separate thread to manage the process pool 😕

[waypointsoftware]

I have this same issue. Some of my requests are generating gateway timeouts, these exceptions are caught somewhere inside futures and control is not returned back to the calling application correctly. Meaning that in this pseudo code:

for a in assignments:
response = a.send_request(...)
responses.append(response)
wait(responses)

The "wait(responses)" never gets hit.

Is there some other way to setup exception handlers for requests?

[ross]

Sorry @waypointsoftware whatever is happening isn't likely within requests-futures's domain to handle. It just provides a framework that uses futures. The process management etc is external to it. If you can provide a simplified failing test case I can try and take a look and help figure out what's up, but otherwise I won't be able to provide much help.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.