Infinite loop while mass scraping data about cryptocmd

[Tomas-MP]

Thank you for the reply,

I fixed it by both making my list shorter as well as implementing a 70 seconds cooldown. Nevertheless the problem was that when extracting a lot of data from 100 coins+, there seems to be a limit around 800 requests as well. I was wondering if there is a way to solve this, anyway making the amount of requests less may do it for now.

[dc-aichara]

@tmlee Thanks for information. I have been using pycoingecko for long time. 👯‍♂

[dc-aichara]

Hello! @Boterbabbelaar
I have done similar project in past. I have calculated BVOC index of few cryptocurrencies.
Answers to your queries.

Are there any limitation to scraping historical data?

There was no limitation till April, 2019. Now, it seems that Coinmarketcap restricted per minute requests to 10 from a IP address.

is there any way I can still scrape the historical data while respecting the limitations.

Yes! There are many ways to extract historical data from coinmarketcap. See a simple example here .
Feel free to reach out to me if you have any query.

[guptarohit]

Hi, @Boterbabbelaar seems like currently, the only option we have is to reduce the rate of request, as suggested by @dc-aichara.

@dc-aichara can you please share the page quoting restricted per minute requests to 10 from an IP address or is it something you found, Thanks 👍

p.s. sorry for the late reply.

[dc-aichara]

@guptarohit

@dc-aichara can you please share the page quoting restricted per minute requests to 10 from an IP address or is it something you found, Thanks 👍

I found it. Thanks.

[tmlee]

@guptarohit @dc-aichara alternatively you can consider our CoinGecko public API at https://www.coingecko.com/en/api

Our API is not that restrictive and open thereby currently used by many services out there.

Happy to answer any questions you may have!