amaidah / listings-parser Goto Github PK

This is an enhancement, and can be within (at the end of) the current CSV/JSON or within an additional generated one.

Basically, would like to get a count of the ratings distribution, stored within the following fields:

• 1_star
• 2_star
• 3_star
• 4_star
• 5_star

This data can be viewed graphically by clicking on the "details" button next to the Yelp score.

Self-explanatory. Depending on how easy this is to do, might be worth making the script tolerant of null values for any kind of data?

Looks like we're already handling null websites just fine, but I'd imagine that SOME listings might be missing full addresses (if it's a food truck?) and maybe hours?

These bolded fields have been identified as slightly more advanced, and require a little more than simply scraping from a distinct field (the non-bolded fields would have been completed as part of basic data).

• full_name
• short_name: Leave this null
• phone_number: Stripped formatting (no parentheses, spaces, or dashes)
• website (url)
• form: Leave this null
• description: Leave this null
• seating: Default to "Indoors". If there is outdoor seating, use "Mixed"
• address_street
• address_city
• address_state
• address_zip
• address_country: Leave this null
• primary: The first cuisine/genre type listed
• secondary: The second cuisine/genre type listed
• cc_accepted: Yes/No
• budget: Count the dollar signs ("2")

The script falls down for certain restaurants, including -

• https://www.yelp.com/biz/the-hat-alhambra

The hypothesis is that it is having trouble if there is spillover for every day of the week (in contrast, https://www.yelp.com/biz/rabbit-hole-alhambra works fine).

This is where things get tricky. Yelp communicates operating hours through the webapp in reference to the start of a time period. So like:

Mon 11:00 am - 2:00 am
Tue 11:00 am - 2:00 am
Wed 11:00 am - 2:00 am

I'm looking to evaluate hours in reference to the specific day. Therefore, Tuesday in this case (expressed as military time) would be:

Tue 0000 - 0200, 1100 - 2400

The following fields for hours of operation should then be straightforward. I've accounted for up to 3 time-slots per day in this model, but I'm not sure how many places actually would need this many slots. These slots would be placed at the end of the CSV/JSON.

mon_hours_start_1
mon_hours_end_1
mon_hours_start_2
mon_hours_end_2
mon_hours_start_3
mon_hours_end_3
... and etc, for each day of the week.

"India's Oven" came out as "India’s Oven" <--- Looks like github has the same bug. Open this up in edit view.

https://www.yelp.com/biz/indias-oven-los-angeles-2

Similar issue with ampersands: https://www.yelp.com/biz/a-and-j-restaurant-arcadia

If these are difficult to fix, NBD. Manually fixing these isn't too horrible.

If the location has special hours on the day of the scrape, the script fails.

These bolded fields are the simplest, and should be the starting point for the listings parser. Basically, given a URL for a listing, the tool should gather the following information and output it (in the given order) as a CSV or JSON.

The non-bolded fields are more advanced and will be done in a separate issue.

• full_name
• short_name: Leave this null
• phone_number: Stripped formatting (no parentheses, spaces, or dashes)
• website (url)
• form: Leave this null
• description: Leave this null
• seating: Default to "Indoors". If there is outdoor seating, use "Mixed"
• address_one
• address_two
• address_city
• address_state
• address_zip
• address_country: Leave this null
• primary: The first cuisine/genre type listed
• secondary: The second cuisine/genre type listed
• cc_accepted: Yes/No
• budget: Count the dollar signs ("2")

Ex:
https://www.yelp.com/biz/din-tai-fung-arcadia-3?page_src=related_bizes

Should work in the parser and automatically work out to:
https://www.yelp.com/biz/din-tai-fung-arcadia-3

https://www.yelp.com/biz/cut-beverly-hills-3

Actually, never mind. The way Yelp stores this is pretty crappy and not reliable.