Big commerce is a scraping tool for marketplace ecommerce in latam
- python >= 3.7.4
- Pip Installed (Recommended >= 21.0.1)
- Install Python packages with pip and requirements.txt
$ pip install -r requirements.txt
Run command:
$ python main.py
- Select marketplace to scraper (Equivalent a
--marketplace
option) - select marketplace country (Equivalent a
--country
option) - select products category to scrape
whats happen?
-
Bigcommerce crawl the subcategories of the selected category and save the data in a csv file in root directory ./output/{PID}-{MARKETPLACE}-{COUNTRY}-categories.csv
What does it scrape?
Value Description id category id in marketplace name category name in marketplace href category link in marketplace hierarchy level in category tree in marketplace (Init 1 hierarchy) parent parent of category in category tree index order category in category tree subcategories number of child categories -
Big commerce crawl all the products of each category, browses through all the pages of the category and scraper the page of each product, in each iteration of the category pages, the product data is saved in a csv file in the directory ./output/{PID}-{MARKETPLACE}-{COUNTRY}-products.csv
What does it scrape?
Value Description name name of product link link product in marketplace price_simbol currency symbol price price product price_discount product discount percentage best_seller indicates if the product have flag best sellers promotional indicates if the product is sponsored category_id product category in the marketplace number_sales public number of sales in the marketplace seller product seller rating product rating -
Play time with data
Options
Optional argument | Description |
---|---|
--marketplace |
name marketplace to scrape |
--country |
Country code where you want to scrape marketplace, available: co, mx |
--recursive |
Browse product pages recursively, available: True or False |
Marketplace and available commands:
$ python main.py --help
- Fork the repo on GitHub
- Clone the project to your own machine
- Commit changes to your own branch
- Push your work back up to your fork
- Submit a Pull request so that we can review your changes