kkamara / amazon-scraper

:cityscape: (Live Link) (2022) Navigates to amazon, searches for samsung phones and pulls the title and price data. I highly recommend working with Linux (including virtual machines) or MacOs.

This project is now deprecated. Please use PHP Scraper.

Amazon Scraper

(2022) Navigates to amazon, searches for samsung phones and pulls the title and price data. I highly recommend working with Linux (including virtual machines) or MacOs.

• Important note:

• Proven in a production environment

• Requirements

• Installation

• Using Docker?

• Usage

• Adding a new command

• Browser Testing

• Mail Server

• Misc

• Contributing

• License

Important note:

Before you try to scrape any website, go through its robots.txt file. You can access it via domainname/robots.txt. There, you will see a list of pages allowed and disallowed for scraping. You should not violate any terms of service of any website you scrape.

Proven in a production environment

Getting up and running on amazon ec2.

Requirements

• https://laravel.com/docs/master/installation
• https://laravel.com/docs/master/mix

Installation

cp .env.example .env
touch database/database.sqlite
composer i
make dev
# optional
# make backend-migrate
# (optional)
# npm install
# npm run dev

Using Docker?

docker build -t laravel-docker-aws .
docker run -it -p 8001:80 laravel-docker-aws

Usage

Update the command at ./app/Console/Commands/BrowseAmazon.php

php artisan browse:amazon

BrowserInvoker.php

Browser.php

Adding a new command

php artisan make:crawler crawler_test

Browser Testing

  alias sail='vendor/bin/sail'
  sail dusk

Mail Server

Mail environment credentials are at .env.

The mailhog docker image runs at http://localhost:8025.

Misc

See php scraper.

See php reactjs boilerplate.

See python amazon scraper 2.

Using Laravel dusk outside of tests.

Running ChromeDriver and Selenium in Python on an AWS EC2 Instance.

The Makefile for this project contains useful commands for a Laravel application and can be found at laravel-makefile.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

BSD

amazon-scraper's Issues

Add php artisan make:crawler