Comments (2)
Seems like there's an impedance mismatch between the 2 tools you've chosen and the problem you've set up to solve.
You want to cache individual urls which checkpoint
does, but that won't allow you to batch up the misses and take advantage of requests futures. If you want to go that route you'd need to do your own futures and have the submit
ed methods call something that's decorated with checkpoint
.
in pseudo-code, haven't seen checkpoint before so no clue whether it can work like this, it may or may not be thread safe.
class Something(object):
def __init__(self):
self.sess = Session() # not a FuturesSession
self.executor = ThreadPoolExecutor()
@checkpoint(<whatever goes here, assuming it can decorate a object method>)
def fetch_url(self, url):
resp = self.sess.get(url)
# processing here
return caption_list
def fetch_urls(self, urls):
futures = {u: self.executor.submit(self.fetch_url, u) for u in urls}
return {u: f.result() for u, f in futures.items()]
s = Something()
results = s.fetch_urls(urls)
Another option would be to manage the caching yourself and use requests futures for the misses.
class Something(object):
def __init__(self):
self.sess = FuturesSesson()
def fetch_urls(self, urls);
results = {}
misses = {}
for url in urls:
cached = self.get_from_cache(url)
if cached:
results[url] = cached
else:
misses[url] = self.sess.get(url)
results.merge({u: self.process_response_and_cache(f.result()) for u, f in misses.items()})
return results
s = Something()
results = s.fetch_urls(urls)
Anyway both of those are just sketches/starting points. The actual solutions would be more involved and likely cleaner/clearer if i were writing them up for real.
Given that this question is really about how to solve a general problem with caching and async and not how to do something specific to requests_futures or checkpoint SO is probably the right place to ask.
from requests-futures.
Thanks a lot for your suggestions!
from requests-futures.
Related Issues (20)
- OSError Issues when used within celery task (gevent) HOT 2
- Adding attribute to response via hook (as seen in examples) fails HOT 1
- Cannot switch to a different thread HOT 3
- Possibility to accept old SSL versions? HOT 1
- Problem with data field in header request HOT 1
- Making a future request and letting callback work with result - some are missing HOT 2
- Error When there is a large parallel Requests go. HOT 2
- Provide wheels on PyPI HOT 1
- How do we pass arguments to the hooks? HOT 3
- AttributeError: 'Future' object has no attribute 'status_code' HOT 1
- 1.0.0: missing git tag HOT 5
- Response for batched requests HOT 6
- Add type annotations HOT 4
- Not showing as updated on PyPI HOT 3
- On exception it returns xx (Caused By None), whereas the requests module returns xx (Caused By xx) HOT 4
- The elapsed time hook example does not work HOT 3
- 1.0.0: pytest is failing in two units HOT 6
- Use pytest-httpbin? HOT 8
- Attribute set via response hooks does not work with ProcessPoolExecutor HOT 4
- RFE: preserve contextvars when performing the request HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from requests-futures.