peterhuistyping / url_finder

Finding the URLs from a given web page in both Python and Go

URL_Finder

Requirement

Implement a very simple web crawler. It should accept a single starting URL, such as https://news.ycombinator.com, as its input. It should download the web page available at the input URL and extract the URLs of other pages linked to from the HTML source code. Although there are several types of link in HTML, just looking at the href attribute of tags will be sufficient for this task. It should then attempt to download each of those URLs in turn to find even more URLs, and then download those, and so on. The program should stop after it has discovered 100 unique URLs and print them (one URL per line) as its output.

Brief Introduction

Solving the problem in Python and Go

Python

Dependencies

import requests
# pip install requests
from bs4 import BeautifulSoup
# Beautiful Soup(bs4) is a Python library for pulling data out of HTML and XML files. 

Go

Using goquery

Dependencies

Type it in the terminal:

go get github.com/PuerkitoBio/goquery

import

"github.com/PuerkitoBio/goquery"