eduardoharo13 / web-scraping-scrolling-website-using-r

This R script shows how I web scraping scrolling website, a kind of website just like twitter and the Fortune 500 2018 list (http://fortune.com/fortune500/list/), when you scroll down the webpage, the webpage will extend and show more information there, even though the url remains the same. Hence, when you use the conventional way of web scraping by using “rvest” alone, it will only give you the first twenty companies, the ones you can see without scrolling. Here’s how I address this issue.

Web-Scraping-Scrolling-Website-Using-R

This R script shows how I web scraping scrolling website, a kind of website just like twitter and the Fortune 500 2018 list (http://fortune.com/fortune500/list/), when you scroll down the webpage, the webpage will extend and show more information there, even though the url remains the same. Hence, when you use the conventional way of web scraping by using “rvest” alone, it will only give you the first twenty companies, the ones you can see without scrolling. Here’s how I address this issue. Moreover, I also compare the 2018 list with 2016 list and found out those who are new companies in 2018 list.