charlybutar21 / toko-scraper

This is a simple command-line tool that is used for scraping a number of products of category Mobile phones/Handphone from Tokopedia and stores it in a csv file and in a PostgreSQL database.

toko-scraper

This is a simple command-line tool that is used for scraping a number of products of category Mobile phones/Handphone from Tokopedia and stores it in a csv file and in a PostgreSQL database.

Technology

• Golang 1.21.1
• PostgresSQL
• Docker
• Go-Colly

How to Setup & Run Application

1. Clone repository

git clone https://github.com/charlybutar21/toko-scraper.git && cd toko-scraper

2. Run docker compose

docker compose up

3. Open another terminal and run command line tool (populate-products)

go run main.go populate-products

4. To see the result, open file product.csv or database console by following these steps:

• Load environment variables

source .env

• Access the PostgreSQL database console

psql "host=localhost port=5433 user=$POSTGRES_USER password=$POSTGRES_PASSWORD dbname=$POSTGRES_DB sslmode=disable"

• View list of tables

\dt 

• Switch to tabular output

\x

• Query and view the products table

select * from products;

Architecture

Command -> Service -> Repository