tal95shah / olx_scraper

:radio: An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

OLX_Scraper

An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

NOTE: This repository is not maintained anymore.

Screenshot

About

A Scrapy Program that scrapes recent ads about products and stores them in MONGODB Database. All the information regarding product to be searched is in args.py

Change values after return command

Usage

For proper usage first install selenium and parsel.Open Command Line and type commands given below

pip install pymongo

Configure these Settings in settings.py

ITEM_PIPELINES = {
     'olx_scraper.pipelines.MongoDBPipeline': 300,
}

MONGODB_SERVER = "localhost" (can be changed)
MONGODB_PORT = 27017(Set Whatever port mongodb is running on your system)
MONGODB_DB = "" (set this)
MONGODB_COLLECTION = "" (set this)

After all the above configurations have been successfully done.Then open command line and type:-

scrapy crawl scrape_olx

Result

Open MongoDB GUI and check database, Your result should be like Screenshot shown above.

Gotchas

1-You must have python 3.6 pre-installed to use this software.
2-Make sure mongodb is running before you run spider.

If

If any issue comes do write in issues column. Thanks!

olx_scraper's Issues

Help

Hi,
Actually, I need help regarding the OLX Scalper, Basically wanna know if there is any way I can find products in a category with that price range in models.

Ex: Category: Camera, Product name: Nikon 5600D, Price range: 25000/- -30000/-

Is there any way I could find all the things that pass the above criteria and just message them if is it available?

Thanks & Regards.