octoparse / scrape-twitter-data

Scrape Twitter data for data analytics, data mining, sentiment analytics

Scrape Twitter Data for Data Analytics

Just download the .otd file, import into Octoparse - a free automatic web scraping tool.

You can get the twitter data you want by simply clicking.

---

Data Extracted in the Scraper

• Name
• Publish time
• Content
• Image URL
• Tweet URL
• Numbers of comments, retweets, and likes

---

Change the scraping URL of the crawler

Search on Twitter in your browser 👉 copy the URL of the result page 👉 paste the URL into the Page URL of the first step: Go To Web Page, click "OK". Now you could run the scraper to get the data.

scrape-twitter-data's Issues

Infinite scrolling limitations

I'm assisting a University research team extracting tweets from a hashtag which has quite a lot of tweets (circa 5,000-10,000). However, I'm finding some difficulties with the infinite scrolling

I tried making something more dynamic, where everytime I extract the data of one item in the loop, I do some scrolling, this way, I don't need to predict in advance the number of scrolls. However, I've been unsuccessful here.

I think the key could be in a nested loop, where the parent loop goes through all items, and the child loop extracts the data of each item and performs a few scrolls. However, I found a few issues here: parent loop is not dynamically finding new items, and I cannot make the child loop to only process the relevant item.

Could you provide some guidance here?

Thank you so much!