Comments (3)
cool. i'll take a look at this when i get a chance now that support is there.
from requests-futures.
after looking at this the Response object isn't the only problem. the instance method on the session object (Session.request) cannot be pickled and sent across to the ProcessPoolExecutor nor can the wrapper if a background_callback was provided.
it would be nice to be able to support it, but it looks like the Request object fix just kicked the problem down the road a bit.
i'm not familar enough with ProcessPoolExector & pickling to know what's required to solve the problem and won't have the time in the near future to look in to it. happy to accept patches, but otherwise this is likely to sit for a whlie.
from requests-futures.
Rant
Hello guys, personally would find it very useful to opt for a ProcessPoolExecutor
. I have a project where I'm executing many requests that return very big JSON responses.
In my use case, the way memory allocation in python works makes those big JSON strings eat up a huge amount of system memory, which does not get released to OS even if I periodically shutdown
the executor
and force garbage collection. Using processes here would make the solution trivial since we would cycle processes instead of threads, releasing memory when each process finishes.
Solution for pickling
It seems session.request
"pickles" just fine today, and as for the callback
function, it seems using functools.partial
would do the trick. The only requirement for pickling the "partial" callback
, is to make it an importable (i.e top level) function. Here is a working POC: https://gist.github.com/asfaltboy/ba33885f7fe08012d79b
P.s: I've added a PR (with tests 🎉) #31. Further discussion on that implementation is there.
from requests-futures.
Related Issues (20)
- OSError Issues when used within celery task (gevent) HOT 2
- Adding attribute to response via hook (as seen in examples) fails HOT 1
- Cannot switch to a different thread HOT 3
- Possibility to accept old SSL versions? HOT 1
- Problem with data field in header request HOT 1
- Making a future request and letting callback work with result - some are missing HOT 2
- Error When there is a large parallel Requests go. HOT 2
- Provide wheels on PyPI HOT 1
- How do we pass arguments to the hooks? HOT 3
- AttributeError: 'Future' object has no attribute 'status_code' HOT 1
- 1.0.0: missing git tag HOT 5
- Response for batched requests HOT 6
- Add type annotations HOT 4
- Not showing as updated on PyPI HOT 3
- On exception it returns xx (Caused By None), whereas the requests module returns xx (Caused By xx) HOT 4
- The elapsed time hook example does not work HOT 3
- 1.0.0: pytest is failing in two units HOT 6
- Use pytest-httpbin? HOT 8
- Attribute set via response hooks does not work with ProcessPoolExecutor HOT 4
- RFE: preserve contextvars when performing the request HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from requests-futures.