simsjarrod / web-scraping

Notes on how to get started scraping content from the web

620-mod6-web-scraping

Module Objectives

• Retrieve web page programmatically. (CO3)
• Parse HTML to search for desired elements. (CO3)
• Apply NLP to web text. (CO3, CO5)
• Visualize results from processed web text. (CO6)

See BeautifulSoup Quickstart Guide

Choose a BeautifulSoup parser:

• 'html.parser' (default, you get this with BeautifulSoup)
• 'html5lib' (install separately if desired)

Getting Started

Fork (copy into your GitHub account) and clone down (to your machine) this repo to get started with web scraping.

Sometimes the data we need isn't available through an API, but we can still access and view it through a web browser. One possible option for dealing with this kind of situation would be to manually copy the text of the web page, but as data analysts and programmers we can do better. In this module we will use previously seen tools to get the HTML of a web page and introduce the capability to automatically parse and search through the HTML to get the content we want to work with.

Read the following early in the week to help you complete your assignments.

Read the Quick Start Guide for Beautiful SoupLinks to an external site. and make sure you have BeautifulSoup4 installed in your active virtual environment. You do not need to read the entire page of documentation, just the Quick Start Guide and installation instructions (if necessary) Read and listen to the online lecture material on the following topics.

Download and make sure you can read and run the code in Web Scraping Notebook (updated F22).

Be curious. There are common terms and libraries available for processing text.

Reflections

• What is a token?
• What is a lemma?
• What is BeautifulSoup?
• What is spaCy?
• What is a spaCy pipeline?
• What are stop words and how are they used to improve results?
• What is whitespace?

If any of these are unfamiliar, do a quick search or ask your favorite AI assistant to fill you in. Use your resources to grow your vocabulary and exposure - there's way too much to memorize, but understanding key terms can make us much more productive.