add ProcessPoolExecutor to background_callback about requests-futures

[ross]

cool. i'll take a look at this when i get a chance now that support is there.

[ross]

after looking at this the Response object isn't the only problem. the instance method on the session object (Session.request) cannot be pickled and sent across to the ProcessPoolExecutor nor can the wrapper if a background_callback was provided.

it would be nice to be able to support it, but it looks like the Request object fix just kicked the problem down the road a bit.

i'm not familar enough with ProcessPoolExector & pickling to know what's required to solve the problem and won't have the time in the near future to look in to it. happy to accept patches, but otherwise this is likely to sit for a whlie.

[asfaltboy]

Rant

Hello guys, personally would find it very useful to opt for a ProcessPoolExecutor. I have a project where I'm executing many requests that return very big JSON responses.

In my use case, the way memory allocation in python works makes those big JSON strings eat up a huge amount of system memory, which does not get released to OS even if I periodically shutdown the executor and force garbage collection. Using processes here would make the solution trivial since we would cycle processes instead of threads, releasing memory when each process finishes.

Solution for pickling

It seems session.request "pickles" just fine today, and as for the callback function, it seems using functools.partial would do the trick. The only requirement for pickling the "partial" callback, is to make it an importable (i.e top level) function. Here is a working POC: https://gist.github.com/asfaltboy/ba33885f7fe08012d79b

P.s: I've added a PR (with tests 🎉) #31. Further discussion on that implementation is there.