Playwright Scrapy app for scraping products and their images for further finding similar products by photo. Images are stored in directory of project and product data is stored in MongoDB.
- MongoDB runs in docker container. Scrapy spiders run in regular/virtual environment.
- Scrapy items have
type
field that will be acollection name
in DB and adirectory name
for images. - User-Agent generator for each request.
- Custom ScrapeOps proxy.
- Custom auto-throttler. Standard download delay applies to the list of domains (
LIMIT_DOMAINS
insettings.py
). - Exclude products list for unnecessary items specified by
type/site/id
. - Image cleaner tool to keep image data consistent with product database. And image resizer tool for further similarity comparison.
- MongoDB 4.4 used for compatibility with servers that don't support AVX. Anyway version can be changed to the most fresh 6 or 7. Dump and restore work fine.
- Spiders can be set on CRON:
0 5 * * * sh /path_to_cron_example.sh 0 20
, so as image tools0 6 * * * python /path_to/products_crawler/tools/image_cleaner.py
and30 6 * * * python /path_to/products_crawler/tools/image_resize.py
Example of spider sh file is hereproducts_crawler/cron/example.sh
- Set Mongo connection data in
.env
- Set JD login and ScrapOps API key
- Install dependencies with
pip install -r requirements.txt
- Install Playwright environment with
playwright install
and dependencies withplaywright install-deps
- Build and start MongoDB
docker compose up -d
- Run spider