zoranpandovski / prodirectscraper

:necktie: Web scraper for http://www.prodirectselect.com/ :shoe:

ProdirectScraper

Installation

Installing Scrapy inside a virtual environment on all platforms.

Python packages can be installed either globally (a.k.a system wide), or in user-space. We do not recommend installing scrapy system wide.

Instead, we recommend that you install scrapy within a so-called “virtual environment” (virtualenv).

Virtualenvs allow you to not conflict with already-installed Python system packages (which could break some of your system tools and scripts), and still install packages normally with pip (without sudo and the likes).

To install it globally (having it globally installed actually helps here), it should be a matter of running:

$ [sudo] pip install virtualenv

Inside virtual env install ProdirectScraper dependencies:

pip install -r requirements.txt

Config Settings

These are the basic options:

# available currency EUR,USD,GBP
currency =
# Number of pieces to display in the email
pp =

# mailer configuration options
smtp_host =
mail_from =
mail_to =
smtp_user =
smtp_pass =
smtp_port =
smtp_tls =
smtp_ssl =

After that edit the configuration specific to the category of product you would like to scrape.

For trainers:

#available sizes are from 4 to 12, e.g 4 or 4,5,10
size =

For men's clothing:

# available options:  One size, ONE-SIZE, S/M, L/XL, S, M, L, XL, XXL
size =

For women's clothing:

# available options: OSFM,One Size,8,10,12,14,16,7 - 10,4½ - 7½,ONE-SIZE,32C,6,3½,4,4½,5,5½,6½,7,7½,XXS,XS,S,M,L,XL
size =

Running the Spiders

To put our spider to work, go to the project’s top level directory and run:

scrapy crawl SCRAPER

where "SCRAPER" must be one of the following:

• trainers
• mensclothing
• womensclothing

This command runs the spider with name trainers, that will crawl http://www.prodirectselect.com/ website and send mail with lowest prices, model description and link to trainers, which size is specified in configuration.ini

prodirectscraper's Issues

Add CONTRIBUTING.md file

Subject of the issue

CONTRIBUTING.md file would encourage other maintainers to submit well formed PRs.

Add more tests

We need more tests coverage. Check https://coveralls.io/github/ZoranPandovski/ProdirectScraper?branch=master for not covered lines.

Upgrade scrappy

Upgrade Scrappy to the newest version available and test implementation.

Add sale parameter

Add option for cheap clothes e.g cheap-mens-clothing => url
https://www.prodirectselect.com/lists/cheap-mens-clothing.aspx

More spiders

Currently we got spiders that crawl trainers from http://www.prodirectselect.com/ site, we need spiders for other products too. PR's are welcome

Need more tests

Currently I got tests only for helpers class. PR's with tests for other implementations are welcome

Add more tests

We need more tests coverage. Check https://coveralls.io/github/ZoranPandovski/ProdirectScraper?branch=master for not covered lines.

Resolve vulnerabilities issues

https://snyk.io/test/github/ZoranPandovski/ProdirectScraper?targetFile=requirements.txt

Some of the requirements libraries should be updated to the latest available version.

Add more tests

We need more tests coverage. Check https://coveralls.io/github/ZoranPandovski/ProdirectScraper?branch=master for not covered lines.

Upgrade Scrappy

Upgrade Scrappy to the newest version available.

More spiders

Currently we got spiders that crawl trainers, mans and womens cloths from http://www.prodirectselect.com/ site, we need spiders that crawls other specific products too. PR's are welcome

Need more tests

We need more tests for the spiders. PR's are welcome

Generalize parse code

I noticed that all the pages that display the products use the same template so it would be possible to just use the same code for all of them with different starting urls and options.

What do you think?