scascketta / robotstxt Goto Github PK
View Code? Open in Web Editor NEWThis project forked from folyd/robotstxt
A native Rust port of Google's robots.txt parser and matcher C++ library.
Home Page: https://crates.io/crates/robotstxt
License: Apache License 2.0