arthur3486 / born2crawl Goto Github PK
View Code? Open in Web Editor NEWA highly performant and versatile crawling engine, designed with scalability and extensibility in mind.
License: Apache License 2.0
A highly performant and versatile crawling engine, designed with scalability and extensibility in mind.
License: Apache License 2.0
Currently crawling logic relies heavily on the Depth-First Search (DFS) algorithm for traversal, which is effective in most cases but may not be optimal for all scenarios. Introducing support for the Breadth-First Search (BFS) algorithm would enhance the crawler's versatility and flexibility, enabling clients to configure it to meet their specific requirements.
At the moment of writing this issue, the crawler supports only two crawling events, namely CrawlingFinished
and CrawlingFailed
. These existing events are sufficient for most but not all of the potential use cases, as occasionally clients want to know when a particular crawling session gets started. Therefore it makes sense to introduce support for a new crawling session event - CrawlingStarted
, which will ensure that crawling session lifecycle is fully covered.
It appears that some of the external dependencies are missing from the library artifacts which causes build issues for the library clients. This concerns only those dependencies that are exposed via public interfaces. To mitigate the issues, all such dependencies should be made transitive & be included in the final library artifacts.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.