swankyalex / data-scrapper

The task is to collect data from three resources:

1. oriencoop.cl
2. som1.ru
3. naturasiberica.ru

Resulted scripts for parsing:

1. Script 1
2. Script 2
3. Script 3

Resulted data is here:

Usage

This package allows you to parse data from all links provided above

1. Clone this repository to your machine.
2. Make sure Python 3.9 and Pipenv (requirements.txt also provided) are installed on your machine.
3. Install the project dependencies (run of the following commands in a terminal, from the root of a cloned repository):

pipenv install / pipenv install --dev #to install with dev-packages (black, isort)

or if you have Make util

make venv / make venv-dev #to install with dev-packages (black, isort)

4. Run script with one of the following commands:

pipenv run python src/script1.py #or script2.py/script3.py 

make parse1 #parse2/parse3

make parse-all #to parse all resources sequentially

The data in JSON format will be saved to the folder data in root directory

5. Also CSV format supported. Just add the option -o csv:

pipenv run python src/script1.py -o csv #or script2.py -o csv/script3.py -o csv

make parse1-csv #parse2-csv/parse3-csv

make parse-all-csv #to parse all resources sequentially

The data in CSV format will be saved to the folder data in root directory

6. If you installed dev dependencies you also can automatically format code with one of the following commands:

make format #run black and isort sequentially

or

pipenv run [black/isort]