cimentadaj / dataharvesting

Material for the course 'Data Harvesting' for the masters in computational social science - UC3M

dataharvesting

Material for the course 'Data Harvesting' for the masters in computational social science - UC3M.

The book for this course can be found in book/.

dataharvesting's Issues

Add HIMYY case study

https://cimentadaj.github.io/blog/2017-10-16-scraping-and-visualizing-how-i-met-your-mother/scraping-and-visualizing-how-i-met-your-mother/

Extend it and make heavy use of xpath and regex.

Redo all images of website to be bigger -- too small at the moment

If you cron doesn't work update PATH on cron

You need to add PATH=/usr/local/bin:/usr/bin:/bin to your cron if it doesn't run, because it's running on a different environment. Add this to the chapter and the slides on automation.

Move Xpath chapter before regex

You noticed you use a heavy xpath in the regex chapter. Check if it makes sense to move xpath before regex. Are there any regex in the xpath chapter? If not, move order.

Rewrite automation chapters for Windows

Nearly all students are windows users. You should have two sections in the chapter: one for unix (including mac) and another for windows. Explain that servers will always use unix so learning the unix way is also a good idea.

Do extra chapter on selector gadget and link it to the XPath chapter as an extension but easier

Fix view source/edit page on website so students can do PRs

Fix images from history of france.

Add selenium script as something to work on on class 6

Class 6 can be very short and nearly all students use selenium in their final project. In the slides URL of the website, add a link to a script containing all the work from the chapter on selenium. That way for the first 1:30 of the class you can go over the selenium chapter and on the second part you can talk about the amazon API

Add general sections for chapters in menu of website. See tidymodels website and R4DS

Add cron schedule for GH actions

https://blog--simonpcouch.netlify.app/blog/r-github-actions-commit/

Make internal API from examples

One example: https://gist.github.com/slopp/721e84ba595da5c3dc074a1957aa7929

Gelman example is only landing page -- can you download the website and make it small?

Add user-agent and sys.sleep through all webscraping examples and refer to ethics chapter

Harmonize all images inside each chapter folder with further subdirectories (only for the automatic ones)

Make Selenium into 1 or 2 more chapters, exploring several examples

Redo all pictures to have better zoom.

I think it should eliminate

Make El Pais website example better

The website has too many incomplete parts and some sections are disfigured. Probably can be fixed by leaving some deleted parts of the extra files.