geekwolverine / chromedp-example

chromedp Example Go App Scraping Sneaker Price List in StockX

chromedp-example

Scraping specific sneakers' bid price list in StockX:

• How to work:
  • Go to https://stockx.com/buy/air-jordan-1-retro-high-og-defiant-couture.
  • Wait to complete loading the confirmation page.
  • Click the confirm button.
  • Wait to complete loading the size list page.
  • Retrieve the sizes and prices.

Setup

Set the project root path to GOPATH.

mkdir chromedp-example
cd chromedp-example
export GOPATH=$(pwd)
go get -d github.com/yoheimuta/chromedp-example # ignore a `no Go files` error.
cd src/github.com/yoheimuta/chromedp-example

Run your chrome headless-shell.

docker pull chromedp/headless-shell
docker run -d -p 9222:9222 --rm --name headless-shell chromedp/headless-shell

Run

There are 3 main.go which use different logic to demonstrate how to utilize the chromedp.

In fact, each code differs only repository injected in main.go.

# normal implementation.
go run cmd/stockx/main.go

# faster implementation. This uses the cookie to skip the confirmation page.
go run cmd/faststockx/main.go

# parallel support implementation. This is more performant.
## NOTE: This uses a local chrome app.
go run cmd/parallelstockx/main.go

Testing

go test -v -count 1 -timeout 240s -race ./...