response is None about grequests

[spyoungtech]

Hi @zyc130130 there may be some limits on how many open connections your client can maintain. size=50000 is probably far too large to be practical for a pool size. Generally speaking, every open socket connection requires an available file descriptor on your OS. On most Linux distributions, the default limit is 1024 or 2048 concurrent descriptors per process. This is just one of the practical limitations on how many concurrent requests you can do.

Keep in mind, the size parameter is how many concurrent greenlets will be created to handle requests. You could think of them also as "workers" if you wanted. Usually setting the size to around 200 is reasonable for most systems with average amounts of activity. You'll have to test to see what works for you. It's best to start small and work your way up to what is acceptable or stable.

So, key takeaway: lowering your size will probably solve your problem.

Does your error handler log the error? If so, could you provide the output?

Another thing that may help is utilizing a session to reuse connections for multiple requests, if that's practical for your use-case. It usually makes things much faster, too.

Without session:

In [1]: import grequests

In [2]: reqs = [grequests.get('https://httpbin.org/status/200') for _ in range(100)]

In [3]: import time

In [4]: start=time.time();grequests.map(reqs, size=10);end=time.time();print(end-start)
3.2363204956054688

With session: much faster!!

In [5]: import requests  # important! always import requests AFTER grequests

In [6]: sesh = requests.Session()

In [7]: reqs = [grequests.get('https://httpbin.org/status/200',session=sesh) for _ in range(100)]

In [8]: start=time.time();grequests.map(reqs, size=10);end=time.time();print(end-start)
1.007857322692871

😮

Also notice how I only used a pool size of 10 and was able to do about 100 req/s 😄

[zyc130130]

Hi @zyc130130 there may be some limits on how many open connections your client can maintain. size=50000 is probably far too large to be practical for a pool size. Generally speaking, every open socket connection requires an available file descriptor on your OS. On most Linux distributions, the default limit is 1024 or 2048 concurrent descriptors per process. This is just one of the practical limitations on how many concurrent requests you can do.

Keep in mind, the size parameter is how many concurrent greenlets will be created to handle requests. You could think of them also as "workers" if you wanted. Usually setting the size to around 200 is reasonable for most systems with average amounts of activity. You'll have to test to see what works for you. It's best to start small and work your way up to what is acceptable or stable.

So, key takeaway: lowering your size will probably solve your problem.

Does your error handler log the error? If so, could you provide the output?

Another thing that may help is utilizing a session to reuse connections for multiple requests, if that's practical for your use-case. It usually makes things much faster, too.

Without session:

In [1]: import grequests

In [2]: reqs = [grequests.get('https://httpbin.org/status/200') for _ in range(100)]

In [3]: import time

In [4]: start=time.time();grequests.map(reqs, size=10);end=time.time();print(end-start)
3.2363204956054688

With session: much faster!!

In [5]: import requests  # important! always import requests AFTER grequests

In [6]: sesh = requests.Session()

In [7]: reqs = [grequests.get('https://httpbin.org/status/200',session=sesh) for _ in range(100)]

In [8]: start=time.time();grequests.map(reqs, size=10);end=time.time();print(end-start)
1.007857322692871

😮

Also notice how I only used a pool size of 10 and was able to do about 100 req/s 😄

Hi,
Thanks for your replay!
At the beginning, I didn’t set the setting parameter "size". It means "size = +∞"? I have the problem, then I set "size=50000", the problem still occurs.
According to what you said，I try some test as fellow:
len(res_list )==3270
1.
req_list.append(grequests.post(sever_url, data=data_temp))
res_list = grequests.map(req_list, exception_handler=err_handler, size=1000)
when use code as 1, I got the right response which has no "None".
2.
req_list.append(grequests.post(sever_url, data=data_temp, session=sesh))
res_list = grequests.map(req_list, exception_handler=err_handler, size=1000)
when use code as 2, the same problem is occured. So i think the reason is "size".
3.
req_list.append(grequests.post(sever_url, data=data_temp, session=sesh))
res_list = grequests.map(req_list, exception_handler=err_handler, size=100)
when use code as 3, I got the right response which has no "None".

Then I compare the results of whether to use the parameter session as code 4,
4.
req_list.append(grequests.post(sever_url, data=data_temp, session=sesh))
res_list = grequests.map(req_list, exception_handler=err_handler, size=10)
when using parameter "session=sesh", program running time is 5:24s,
when not using parameter "session=sesh", program running time is 5:23s
The increase in program speed is not obvious, it seem to be not working.

My gunicorn.conf file set as fellow:
workers = 4
worker_class = 'gevent'

[AustinGilkison]

Also having the same issue.
urls only contains one URL at the moment and still doesn't work.

rs = (grequests.get(u) for u in urls)
results = grequests.map(rs, size=10)

Results is a list of None.

[spyoungtech]

Almost always this is because there is some error happening during processing of the request.

For example:

>>> import grequests
>>> rs = [grequests.get('badscheme://foobar.com')] * 5
>>> grequests.map(rs)
[None, None, None, None, None]

If you use an exception handler, you can see what is going wrong.

>>> def exception_handler(request, exception):
...    print("Request failed", exception)
>>> rs = [grequests.get('badscheme://foobar.com')] * 5
>>> grequests.map(rs, exception_handler=exception_handler)
Request failed No connection adapters were found for 'badscheme://foobar.com'
Request failed No connection adapters were found for 'badscheme://foobar.com'
Request failed No connection adapters were found for 'badscheme://foobar.com'
Request failed No connection adapters were found for 'badscheme://foobar.com'
Request failed No connection adapters were found for 'badscheme://foobar.com'
[None, None, None, None, None]