Rate limit handling about search-tweets-python

[markoarnauto]

Handling rate limits is a big issue when it comes to paid requests. I hope it will be incorporated soon.

[jeffakolb]

There appear to be different rate limits between the enterprise 30day and enterprise full archive search products. For my query, the code polls for new pages at a rate that exceeds 30 day but not FAS rate limits.

Full archive enterprise search doesn't hit rate limit:
(3.7.0) [tw-mbp-jkolb ~]$ search_tweets.py --credential-file-key enterprise_fas --filter-rule "kolb" | wc -l
5405

30-day enterprise search with max results set to limit pagination:
(3.7.0) [tw-mbp-jkolb ~]$ search_tweets.py --credential-file-key enterprise_30day --filter-rule "kolb" --max-results 1000 | wc -l
1000

30-day enterprise with no max results:
(3.7.0) [tw-mbp-jkolb ~]$ search_tweets.py --credential-file-key enterprise_30day --filter-rule "kolb" | wc -l ERROR:searchtweets.result_stream:HTTP Error code: 429: Exceeded rate limit ERROR:searchtweets.result_stream:Rule payload: {'query': 'kolb', 'next': 'eyJhdXRoZW50aWNpdHkiOiIzNDgxYWE1YmZlYzgyNDJlMjYxMjFkNDYyM2U5NjRlZTZlOWQ5Njk4MWI3YWYwZDdhYWNlYTUzZDhlZDU2ZWRkIiwiZnJvbURhdGUiOiIyMDE4MDgwODAwMDAiLCJ0b0RhdGUiOiIyMDE4MDkwNzE2MjYiLCJuZXh0IjoiMjAxODA4MjkyMTI5MTgtMTAzNDkxNTkyMzU3MDk3MDYyNi0wIn0='} Traceback (most recent call last): File "/Users/jkolb/python_installations/3.7.0/bin/search_tweets.py", line 6, in <module> exec(compile(open(__file__).read(), __file__, 'exec')) File "/Users/jkolb/repos/search-tweets-python/tools/search_tweets.py", line 196, in <module> main() File "/Users/jkolb/repos/search-tweets-python/tools/search_tweets.py", line 190, in main for tweet in stream: File "/Users/jkolb/repos/search-tweets-python/searchtweets/result_stream.py", line 216, in stream self.execute_request() File "/Users/jkolb/repos/search-tweets-python/searchtweets/result_stream.py", line 253, in execute_request rule_payload=self.rule_payload) File "/Users/jkolb/repos/search-tweets-python/searchtweets/result_stream.py", line 101, in retried_func raise requests.exceptions.HTTPError requests.exceptions.HTTPError
2100

[igorbrigadir]

A quick thing that might work is to use https://github.com/tomasbasham/ratelimit
like this: https://github.com/igorbrigadir/search-tweets-python/blob/rate-limits/searchtweets/result_stream.py#L111-L113

(but numbers of calls are different for products so they should be in a config file somewhere)

[jimmoffitt]

FWIW, these updates were recently made:

I made some updates to the HTTP error handling. Previously, there was a general mechanism to retry 3 times for every error. This retry mechanism had a backoff design, but there was not a long enough delay duration to recover from a rate limit error. It also treated all errors the same, retrying three times even if your credentials were wrong.

So, this new update does the following:

With rate limit errors, it backs off exponentially across a fifteen-minute period and should recover from such errors and continue on.

Retries with 5xx server-side errors, 30 seconds between retries.

All other errors (4xx) are treated as "one and done, no use in retrying..." errors.