Comments (8)
I tried this using python modules mal-scraper and beautifulsoup. I am able to get all the data I want but it's taking too much time any suggestions as to how am I supposed to solve this?
from artemis-arrow.
Why do you use mal-scrapper if you are anyway using beautifulsoup? Try using requests, should be a bit faster, and you can speed it up with multiprocessing.
from artemis-arrow.
Ohh sorry I meant I was just using mal-scraper which was using beautifulsoup
from artemis-arrow.
Hmm, As I said, you can try multiprocessing, but it would be better if you use requests alone. This would make the job of dockering easier, and you can have the same functionality from requests itself
from artemis-arrow.
Even a simple request to myanimelist.net/anime/id using the package of requests takes a significant amount of time. Iterating from id =1 to 30000 would seriously take a large amount of time.
from artemis-arrow.
mal-scraper itself uses requests to make an http request to myanimelist and beautifulsoup to get a html parser. I have already written a script to use the mal-scraper which gets all the data but it has the time issue, and I suppose even dockering would wouldn't be that hard while using mal-scraper.
from artemis-arrow.
I know it would take a lot of time, pbviously, atleast around 10 hours, we can speed this up upto 10x, by multiprocessing, and anyway, this is like a one time script. So don't worry. You work on the multiprocessing part.
from artemis-arrow.
Okay I'll look into multi-processing
from artemis-arrow.
Related Issues (17)
- Create a web page HOT 3
- Scheduler for anime list generation.
- Downloading Books.
- Add animes to my animes list!
- Add Book to Goodreads bookshelf
- Create a backend for the app HOT 2
- Add Pullrequest and Bug Fixes Template in the repo
- Bugs in Youtube downloader HOT 9
- Speed up anime list scraping HOT 11
- Add Drive Facility
- Anime downloader. Rapidvideo links disabled.
- Anime Feature
- Make API credentials environmental variables
- suggestions for improvement by adding a few features HOT 1
- Piracy issues? HOT 1
- Secure your application HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from artemis-arrow.