Comments (5)
Interesting. Happy to take a 👁️ at a PR that sets a timeout, but it might get a little complicated to make sure that plays well with requests
timeouts. Let me know what you find.
from requests-futures.
@ross thanks for the feedback. I used requests.get(timeout=X)
and Future.result(timeout=X + GRACE)
in case requests' exception in child is thrown after a delay.
Unfortunately, my plan failed, due to my lack of understand of the problem. Even with timeout limits set on the children process, the child still "is suspended in time" and my logs show odd behaviour of the parent continuing, while the pool is not cleared 10 minutes after due finish time. ps
shows the child processes in the D state - "Uninterruptible sleep (usually IO)". There must be something I'm missing.
I plan to simplify our usage by removing most callback/result to after all requests are done. This should give a good indication of where things are broken.
One other interesting thing I saw, is when I stop a test by raising KeyboardInterrupt
, the exception shows this trace:
File "my_code", line 594, in some_method
response = res.result(timeout=CHILD_PROCESS_TIMEOUT)
File "/home/pavel/.pyenv/versions/3.5.1/lib/python3.5/concurrent/futures/_base.py", line 400, in result
self._condition.wait(timeout)
File "/home/pavel/.pyenv/versions/3.5.1/lib/python3.5/threading.py", line 297, in wait
gotit = waiter.acquire(True, timeout)
KeyboardInterrupt
I wonder if concurrent.futures
runs a separate thread to manage the process pool 😕
from requests-futures.
I have this same issue. Some of my requests are generating gateway timeouts, these exceptions are caught somewhere inside futures and control is not returned back to the calling application correctly. Meaning that in this pseudo code:
for a in assignments:
response = a.send_request(...)
responses.append(response)
wait(responses)
The "wait(responses)" never gets hit.
Is there some other way to setup exception handlers for requests?
from requests-futures.
Sorry @waypointsoftware whatever is happening isn't likely within requests-futures's domain to handle. It just provides a framework that uses futures. The process management etc is external to it. If you can provide a simplified failing test case I can try and take a look and help figure out what's up, but otherwise I won't be able to provide much help.
from requests-futures.
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.
from requests-futures.
Related Issues (20)
- OSError Issues when used within celery task (gevent) HOT 2
- Adding attribute to response via hook (as seen in examples) fails HOT 1
- Cannot switch to a different thread HOT 3
- Possibility to accept old SSL versions? HOT 1
- Problem with data field in header request HOT 1
- Making a future request and letting callback work with result - some are missing HOT 2
- Error When there is a large parallel Requests go. HOT 2
- Provide wheels on PyPI HOT 1
- How do we pass arguments to the hooks? HOT 3
- AttributeError: 'Future' object has no attribute 'status_code' HOT 1
- 1.0.0: missing git tag HOT 5
- Response for batched requests HOT 6
- Add type annotations HOT 4
- Not showing as updated on PyPI HOT 3
- On exception it returns xx (Caused By None), whereas the requests module returns xx (Caused By xx) HOT 4
- The elapsed time hook example does not work HOT 3
- 1.0.0: pytest is failing in two units HOT 6
- Use pytest-httpbin? HOT 8
- Attribute set via response hooks does not work with ProcessPoolExecutor HOT 4
- RFE: preserve contextvars when performing the request HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from requests-futures.