4jx / reqwest-impersonate Goto Github PK

In lycheeverse/lychee#733 we've been discussing whether reqwest-impersonate could be a solution to getting blocked by various websites. After some testing we found that indeed we would be able to circumvent some of the issues thanks to this project.
The plan was to integrate reqwest-impersonate into lychee as an optional feedback and a fallback to reqwest for hard cases where the server does any browser detection.
However to achive this, we would need a publicly available crate. I saw that this project applied some patches to upstream crates and I wonder if you could publish these dependencies as well as reqwest-impersonate on crates.io.
It's totally understandable if you rather would not want to do this for maintainability reasons. In such case we'd have to find an alternative solution for lychee.

Thanks for this crate in any case.

Hello,

Thank you very much for publishing this, as it may be invaluable for surveying the web without tripping up Cloudflare and other WAFs. I am trying to use it (my test repo), but it seems to be failing where reqwest would not:

test-reqwest-impersonate$ ./target/debug/test-reqwest-impersonate https://letsencrypt.org/
Error: error sending request for url (https://letsencrypt.org/): error trying to connect: TLS handshake failed: cert verification failed - certificate has expired [CERTIFICATE_VERIFY_FAILED]

Caused by:
    0: error trying to connect: TLS handshake failed: cert verification failed - certificate has expired [CERTIFICATE_VERIFY_FAILED]
    1: TLS handshake failed: cert verification failed - certificate has expired [CERTIFICATE_VERIFY_FAILED]
    2: [CERTIFICATE_VERIFY_FAILED]
    3: [CERTIFICATE_VERIFY_FAILED]

(Debian 11.4 with ca-certificates installed, rustc 1.63.0)

When used on NixOS 22.05 which has a different /etc/ssl, it seems to fail on all https:// sites with a different error:

test-reqwest-impersonate# ./target/debug/test-reqwest-impersonate https://www.google.com/
Error: error sending request for url (https://www.google.com/): error trying to connect: TLS handshake failed: cert verification failed - unable to get local issuer certificate [CERTIFICATE_VERIFY_FAILED]

Caused by:
    0: error trying to connect: TLS handshake failed: cert verification failed - unable to get local issuer certificate [CERTIFICATE_VERIFY_FAILED]
    1: TLS handshake failed: cert verification failed - unable to get local issuer certificate [CERTIFICATE_VERIFY_FAILED]
    2: [CERTIFICATE_VERIFY_FAILED]
    3: [CERTIFICATE_VERIFY_FAILED]

# ls -lR /etc/ssl
/etc/ssl:
total 4,096
drwxr-xr-x 1 root root 64 2022-09-17 10:19 certs/
lrwxrwxrwx 1 root root 28 2022-09-17 10:19 trust-source -> /etc/static/ssl/trust-source/

/etc/ssl/certs:
total 8,192
lrwxrwxrwx 1 root root 35 2022-09-17 10:19 ca-bundle.crt -> /etc/static/ssl/certs/ca-bundle.crt*
lrwxrwxrwx 1 root root 41 2022-09-17 10:19 ca-certificates.crt -> /etc/static/ssl/certs/ca-certificates.crt*

# ls -l /etc/static/ssl/certs
total 8,192
lrwxrwxrwx 1 root root 87 1970-01-01 00:00 ca-bundle.crt -> /nix/store/sg1d11cjmc63iz6krh8mray50fkk0gqk-nss-cacert-3.80/etc/ssl/certs/ca-bundle.crt*
lrwxrwxrwx 1 root root 87 1970-01-01 00:00 ca-certificates.crt -> /nix/store/sg1d11cjmc63iz6krh8mray50fkk0gqk-nss-cacert-3.80/etc/ssl/certs/ca-bundle.crt*

# ls -l /nix/store/sg1d11cjmc63iz6krh8mray50fkk0gqk-nss-cacert-3.80/etc/ssl/certs
total 495,616
-r-xr-xr-x 3 root root 494,993 1970-01-01 00:00 ca-bundle.crt*

Is there something I should be doing to give reqwest-impersonate/BoringSSL a Chrome-like set of trusted certificate authorities? It would be very helpful if this were documented, or if it could trust all the certificates in webpki-roots or similar.

thread 'main' panicked at '"enum_(unnamed_at_deps/boringssl/src/include/openssl/err_h_291_1)" is not a valid Ident

System

• Windows 11

Dependencies

[dependencies]
reqwest-impersonate = { git = "https://github.com/4JX/reqwest-impersonate.git", default-features = false, features = [
    "chrome",
    "blocking",
] }

[patch.crates-io]
hyper = { git = "https://github.com/4JX/hyper.git", branch = "v0.14.18-patched" }
h2 = { git = "https://github.com/4JX/h2.git", branch = "imp" }

main.rs

use reqwest_impersonate::browser::ChromeVersion;

fn main() {
    // Build a client to mimic Chrome 104
    let client = reqwest_impersonate::blocking::Client::builder()
        .chrome_builder(ChromeVersion::V104)
        .build()
        .unwrap();

    // Use the API you're already familiar with
    match client.get("https://yoururl.com").send() {
        Ok(res) => {
            println!("{:?}", res.text().unwrap());
        }
        Err(err) => {
            dbg!(err);
        }
    };
}

It's possible to add support to wasm?