Comments (7)
common python-ish solutions would to either use a closure or lambda i.e.
future = session.get(url, background_callback=lambda sess, resp: fun_cb(sess, resp, id))
or
id = 'foo'
def fun_cb(sess, resp):
# id can be accessed here
pass
future = session.get(url, background_callback=fun_cb)
i haven't run across a situation with a pattern like background_callback_args in python, if you have/it's common i'd be open to the idea, just point me to some examples (unless lots of people are going to expect it i would prefer to avoid adding more ways to do things.)
best,
-rm
from requests-futures.
Lambda one works like a charm ;)
Tnxs for the python-ish response. I think it is more adaptable and let a simpler code.
from requests-futures.
cool. glad to help.
from requests-futures.
Using the lambda doesn't quite work for me -- it's not really respecting the params I'm sending down (unless I'm having the wrong expectations):
When running this test:
def bg_call(sess, resp, hostname):
# host from response.
resp_host = urlparse.urlparse(resp.url).hostname.split('www.')[-1]
log.debug('orig hostname %s, from resp %s', hostname, resp_host)
reqs_num_workers = 10
session = FuturesSession(max_workers=reqs_num_workers)
targets = ['google.com', 'foo.com']
types = ['http://', 'https://']
for hostname in targets:
log.debug('doing %s', hostname)
for schema in types:
session.get('%s%s' % (schema, hostname),
background_callback=lambda sess, resp: bg_call(sess, resp, hostname),
timeout=2)
I get this output (note how we end up with orig hostname google.com, from resp foo.com
, meaning the original hostname I sent down in the request is NOT the one corresponding to the response):
07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - doing google.com
07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - doing foo.com
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTPS connection (1): google.com
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTP connection (1): google.com
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTP connection (1): foo.com
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTPS connection (1): foo.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 301 219
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTP connection (1): www.google.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 301 220
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTPS connection (1): www.google.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 200 None
07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - orig hostname foo.com, from resp google.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 301 184
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTP connection (1): www.foo.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 200 None
07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - orig hostname foo.com, from resp google.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 200 1494
07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - orig hostname foo.com, from resp foo.com
from requests-futures.
i believe there's a common scope problem there. you're using in a variable hostname in the right side of the lambda that will have changed (many times) by the time the lambda is called.
nothing specific to requests-futures, it's just a python scope thing. this stack overflow post seems to do a good job of describing the situation and solution:
http://stackoverflow.com/questions/938429/scope-of-python-lambda-functions-and-their-parameters
from requests-futures.
meet the same question, work out it followed as @ross
http://stackoverflow.com/questions/938429/scope-of-python-lambda-functions-and-their-parameters
When a lambda is created, it doesn't make a copy of the variables in the enclosing scope that it uses. It maintains a reference to the environment so that it can look up the value of the variable later. There is just one m. It gets assigned to every time through the loop. After the loop, the variable m has value 'mi'. So when you actually run the function you created later, it will look up the value of m in the environment that created it, which will by then have value 'mi'.One common and idiomatic solution to this problem is to capture the value of m at the time that the lambda is created by using it as the default argument of an optional parameter. You usually use a parameter of the same name so you don't have to change the body of the code:
for m in ('do', 're', 'mi'): funcList.append(lambda m=m: callback(m))
so, @yulyt
session.get('%s%s' % (schema, hostname), background_callback=lambda sess, resp, hostname=hostname: bg_call(sess, resp, hostname), timeout=2)
will work.
from requests-futures.
I see, I understand the issue. Thanks for the references @ross and @crazygit :-)
from requests-futures.
Related Issues (20)
- No module named 'requests_futures.sessions' HOT 3
- OSError Issues when used within celery task (gevent) HOT 2
- Adding attribute to response via hook (as seen in examples) fails HOT 1
- Cannot switch to a different thread HOT 3
- Possibility to accept old SSL versions? HOT 1
- Problem with data field in header request HOT 1
- Making a future request and letting callback work with result - some are missing HOT 2
- Error When there is a large parallel Requests go. HOT 2
- Provide wheels on PyPI HOT 1
- How do we pass arguments to the hooks? HOT 3
- AttributeError: 'Future' object has no attribute 'status_code' HOT 1
- 1.0.0: missing git tag HOT 5
- Response for batched requests HOT 6
- Add type annotations HOT 4
- Not showing as updated on PyPI HOT 3
- On exception it returns xx (Caused By None), whereas the requests module returns xx (Caused By xx) HOT 4
- The elapsed time hook example does not work HOT 3
- 1.0.0: pytest is failing in two units HOT 6
- Use pytest-httpbin? HOT 8
- Attribute set via response hooks does not work with ProcessPoolExecutor HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from requests-futures.