h4kor / koios

A web crawler written in Elixir

koios's Issues

Rework crawlers for more predictable behavior

Currently crawlers can terminate "without warning" and have a lot of internal messaging. This makes it difficult to reason about their behavior. This also makes it difficult to test them properly.

• Rework crawlers to allow better testing
• Rework for more predictable behavior

Create example showing usage of library

Write a simple example showing how to extract some information from a website.

Implement statistics for crawler

Feature

As a user I want to observe the crawlers, to estimate speed and progress of them.

Implement one function returning a map including all stats. If some stat is expensive to retrieve. it may be excluded and implemented in a separate function.

Stats to include:

• number of visited pages
• queued urls
• currently running tasks
• maximum tasks
• start url
• start time
• request in the last minute
• discarded urls due to constraints