evilscott / gocrawler

Web crawler in Go

Go Crawler

A web crawler in Go.

gocrawler's Issues

Quiet/verbose modes

Add command line flags for quiet/verbose modes

configurable buffer size

shrink the todos buffer size to 100 by default and expose the size to configuration options for flexibility

documentation and examples

Need examples via go, and documentation in industry standard go format (markdown as well?)

obey meta

Don't crawl nofollow and obey robots.txt

handle redirects

Follow redirects (default) but don't crawl if landing outside of domain. Option to not follow redirects or only follow a few levels.

fix race condition

results are non-deterministic. possible race condition.

Add user-agent

Add a custom user-agent (and allow custom setting via CLI)

Code refactor

Code is getting ugly; last step before MVP should be refactoring code for readability and maintainability.

JSON output

Output should be JSON, with the option to output to file instead of stdout.

collect additional data for each link

Collect data on links like internal/external, anchor, query string, etc

handle non 2xx/3xxs gracefully

non 4xx/5xxs need to be surfaced, especially if the initial URL comes back with one.

Allow for config file instead of command line args

File configuration should be available (either checking a default path or passing in via command line) since the number of configuration options is growing.

Integration test

Create an integration test by spinning up a test server and hitting it with the main crawler.

use tabs

go standard is tabs. use them and then keep with gofmt

Allow for crawl throttling

Via command line flag. This will need to be in place to support robots.txt Crawl-delay directive anyway

avoid potential deadlock from waitgroup

https://github.com/EvilScott/gocrawler/blob/6a1512e7ed46dc41ec7ca1efa16046b84777b551/crawl/crawl.go#L60

Need to add wg.Done() to continue calls to avoid deadlock on error during crawl