arthur3486 / born2crawl Goto Github PK

Currently crawling logic relies heavily on the Depth-First Search (DFS) algorithm for traversal, which is effective in most cases but may not be optimal for all scenarios. Introducing support for the Breadth-First Search (BFS) algorithm would enhance the crawler's versatility and flexibility, enabling clients to configure it to meet their specific requirements.

At the moment of writing this issue, the crawler supports only two crawling events, namely CrawlingFinished and CrawlingFailed. These existing events are sufficient for most but not all of the potential use cases, as occasionally clients want to know when a particular crawling session gets started. Therefore it makes sense to introduce support for a new crawling session event - CrawlingStarted, which will ensure that crawling session lifecycle is fully covered.

It appears that some of the external dependencies are missing from the library artifacts which causes build issues for the library clients. This concerns only those dependencies that are exposed via public interfaces. To mitigate the issues, all such dependencies should be made transitive & be included in the final library artifacts.