aaasen / mycelium Goto Github PK

Mycelium currently relies on Redis for task queuing and storing results.

Investigate ways to deploy Redis. Amazon EC2? Google Cloud Compute?
A similar alternative?

I'm currently "translating" some old projects to golang.

Mycelium does not currently respect robots.txt, which means that it shouldn't be deployed in the wild, or really used at all.

At the moment, Mycelium will recursively crawl forever.
A depth control should be implemented for tasks like crawling individual websites.

Currently, the only TaskQueue implementation uses Redis. This is a pain to set up, and only really makes sense in distributed settings.

Currently only absolute urls are followed which can create problems.

Tasks are currently ordered by system time, with the oldest tasks being resolved first.

The replacement solution should address these problems:

• directs many crawlers to the same domain at one time
• potentially wastes time crawling many irrelevant pages from one domain

A SQLite (or other SQL) datastore would suit large scale crawls better than Redis.

The current logging system doesn't give much insight into how the crawler is operating.

A solution to this issue should show the following statistics:

• number of backlogged tasks in each buffered channel
• pages per sec
• number of task requests per sec