Comments (4)
Maybe you should also try to limit the requests / seconds / minutes you do. Since your IP is banned now, no fake useragent strings will help you with that.
If you are using scrapy framework for example, you have an option like DOWNLOAD_DELAY
:
The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same website. This can be used to throttle the crawling speed to avoid hitting servers too hard.
See also another scrapy option called CONCURRENT_REQUESTS_PER_DOMAIN
.
If however you use your own scripting without scrapy, consider adding sleeps to your crawling process.
from fake-useragent.
Also are you using Amazon AWS?
from fake-useragent.
Also are you using Amazon AWS?No. I am not.
I use the googlesearch library python, which is based on requests and beautifulsoup; And I have also used time.sleep.
Actually; I have an API that receives almost 200 Google page links per request, and I get blocked with more requests.
The IP will be blocked for a few hours, and after that you can request it again.(The duration of the blocking is not known)
I am trying to prevent IP banning by using fake useragent and proxy.
The number of your fake useragent is 260, (and I choose them randomly); while some fake useagents may be used several times, so I need more fake useragent;
I wish the number could be increased to 500.
thanks for the help.
from fake-useragent.
We want to switch to another source and also add mobile platforms.
from fake-useragent.
Related Issues (20)
- Missing isolated unit-tests HOT 4
- Tests should use unittest.mock on Python 3 HOT 1
- Broken link in cache_scraper/README.md HOT 1
- fake_useragent.errors.FakeUserAgentError: No browser user-agent strings found for browser: chrome HOT 13
- Package cache json file locally
- browsers.json not loading on Python 3.7, 3.8 & 3.9 HOT 11
- pyinstaller: ModuleNotFoundError: No module named 'fake_useragent.data' HOT 18
- HTTP Error 503: Service Unavailable HOT 6
- Heroku Free Dynos: "fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached" HOT 14
- Invalid leading whitespace HOT 2
- [Help] How do I get UserAgent list in this format ? HOT 1
- Service unavailable errors (cache too!) HOT 5
- Outdated package in `conda-forge` HOT 1
- [Bug?] user_agents in Opera contains IE user_agents HOT 2
- Error occurred during loading data. Trying to use cache server https://fake-useragent.herokuapp.com/browsers/0.1.11 HOT 4
- I used this module, but it doesn't work. Every time it runs, it's the real agent information HOT 3
- Browser version in browsers.json is too old HOT 2
- "Error occurred during getting browser: shape, but was suppressed with fallback." HOT 1
- Visual Code error: Import fake_useragent could not be resolved HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fake-useragent.