swiggy-scraper's Introduction

Swiggy Scraper

A Scraping project to scrape restaurant information from Swiggy. This scraper is able to scrape data for entire Indian restaurants.

Data

The links of all the restaurants can be find in links.csv file and scraped data can be found in scrape_data.csv file. The following variables are saved:

Restaurant Name: Name of the restaurant
Longitude: Longitude coordinate of the restaurant's location
Latitude: Latitude coordinate of the restaurant's location
Price For Two: Cost for two in rupees
Region: Region where restaurant is located
Rating: Average rating out of 5

Libraries

Following libraries are used in this project:

BeautifulSoup
requests
re
json
pandas
time

About the code

The whole project is divided into two part:

Crawler: It will crawl the entire website and fetch required links i.e url of each restaurant.
Scraper: It will take each links fetched from crawler as input and will go to each page to extract the required information.

Techniques used

Headers: Python Requests does not force you to use request headers while sending requests but there are few smart websites that does not let you to extract any data without certain headers. So it is always good to set headers.
Time Delay: A delay of 5 seconds has been given after each url hit to prevent scraper from being blocked.
Multiprocessing: As there are more than 57,000 restaurants listed on swiggy so extracing data from each url with a delay of 5 second will take very long time. So to overcome this multiprocessing is used to speed up scraping.

Recommend Projects