abhijeet-0019 / webcrawler-http

project for the learn http course, and make world a better place

HTTP Web Crawler in JavaScript

This is a simple HTTP web crawler built in JavaScript, which crawls the web and fetches the URLs of the web pages. The crawler also generates a report, which can be used to gain insights into the web pages that have been crawled.

How it works

The web crawler is designed to crawl the web in a recursive manner. Starting from a specified URL, the crawler fetches the contents of the page and extracts all the links from it. These links are then added to the list of URLs that need to be crawled.

The crawler continues to crawl the URLs in the list until there are no more URLs left to crawl. During the crawling process, the crawler generates a report, which includes information about each web page that has been crawled.

🛠 Technologies Used

• JavaScript

Authors

• @abhijeet-0019

Acknowledgements

• This app was built as part of a personal project to learn the https basics.

• Thanks to Lane Wagner - of @boot.dev to guide me to build this amazing project.