alexander-irbis / robots_txt Goto Github PK

Using directives jointly

The Allow and Disallow directives from the corresponding User-agent block are sorted according to URL prefix length (from shortest to longest) and applied in order. If several directives match a particular site page, the robot selects the last one in the sorted list. This way the order of directives in the robots.txt file doesn't affect how they are used by the robot.

url had a major bump to 2.x, would be nice to see an upgrade.

Yandex Clean-param

Clean-param: p0[&p1&p2&..&pn] [path]

While comparing url paths with a couple of robots.txt I stuck on a panic caused by an ä (German umlaut).
Is this something that can be fixed or do I have to normalize the path ?

Error Stack:

thread '' panicked at 'byte index 9 is not a char boundary; it is inside 'ä' (bytes 8..10) of /infos/häufig-gestellte-fragen.html', src/libcore/str/mod.rs:2034:5
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
stack backtrace:
0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
1: std::sys_common::backtrace::_print
at src/libstd/sys_common/backtrace.rs:71
2: std::panicking::default_hook::{{closure}}
at src/libstd/sys_common/backtrace.rs:59
at src/libstd/panicking.rs:197
3: std::panicking::default_hook
at src/libstd/panicking.rs:211
4: <std::panicking::begin_panic::PanicPayload as core::panic::BoxMeUp>::get
at src/libstd/panicking.rs:474
5: std::panicking::continue_panic_fmt
at src/libstd/panicking.rs:381
6: std::panicking::try::do_call
at src/libstd/panicking.rs:308
7: ::type_id
at src/libcore/panicking.rs:85
8: core::str::traits::<impl core::slice::SliceIndex for core::ops::range::Range>::index::{{closure}}
at src/libcore/str/mod.rs:2034
9: core::ptr::real_drop_in_place
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/str/mod.rs:1821
10: core::str::traits::<impl core::slice::SliceIndex for core::ops::range::RangeFrom>::index
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/option.rs:388
11: core::str::traits::<impl core::slice::SliceIndex for core::ops::range::RangeTo>::index
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/str/mod.rs:1821
12: core::str::traits::::eq
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/str/mod.rs:1623
13: <core::str::LinesAnyMap as core::ops::function::FnMut<(&str,)>>::call_mut
at /Users/thorstenclaus/.cargo/registry/src/github.com-1ecc6299db9ec823/robots_txt-0.6.0/src/matcher.rs:51

Take into account * and $ in Allow/Disallow

Disallow: /*sid=
Disallow: /private*html
Disallow: /private$

Robots.txt Specifications.