[Feature request] Proxy about website-stalker

[b-reich]

It states they are enabled by default and might already work out of the box with the website-stalker.

They were referring to system-wide proxy which I can't use.

i think you could use export https_proxy= for a specific proxy in your shell session

[rEnr3n]

Does not work:
$ http_proxy=socks5://127.0.0.1:1080 ~/Downloads/website-stalker run --all ❌

I have sockd listen on port 1080 and tinyproxy on port 1081.

Confirmed my proxies are working:

$ curl -x socks5://127.0.0.1:1080 icanhazip.com
CORRECT_IP

$ curl -x http://127.0.0.1:1081 icanhazip.com
CORRECT_IP

[EdJoPaTo]

I'm not sure how proxy handling is done. My current understanding is the proxy gets used system-wide and processes simply use the system network which includes the proxy? In that case it would already work but it seems there is something I don't know yet?

[rEnr3n]

I don't want to use the proxy system-wide. That would interfere with my workflow. I only want to use it for specific commands.

[EdJoPaTo]

I'm not used to using proxies and only know them in theory. I'm not sure how this might be added best.

From looking on reqwest, the HTTP library being used, there is a section on proxies. It states they are enabled by default and might already work out of the box with the website-stalker.
Can you give it a try? If that already works I guess its a nice thing to be added to the readme.

[b-reich]

@rEnr3n you could use a PAC as a hacky workaround until this feature is implemented

[rEnr3n]

It states they are enabled by default and might already work out of the box with the website-stalker.

They were referring to system-wide proxy which I can't use.

[rEnr3n]

Right 😝. I tested it quickly but it won't work. Running curl icanhazip.com on the same shell gives me the ip address of the proxy though.

[rEnr3n]

This is what I did:

$ export http_proxy="socks5://127.0.0.1:1080"
$ curl icanhazip.com
<PROXY IP ADDRESS>
$ website-stalker run --all --commit
<TIMEOUT ERROR>

$ export https_proxy="socks5://127.0.0.1:1080"
$ curl icanhazip.com
<REAL IP ADDRESS>
$ website-stalker run --all --commit
<TIMEOUT ERROR>

These are two separate shells btw.

[EdJoPaTo]

Looks like export https_proxy= is behind a feature flag of reqwest. I added that in 3ba2ddb.
If you want to test it out get the latest build from the build artifacts. I'm interested if that already works.

[rEnr3n]

I have socks proxy on port 1080 and http proxy on 1081.

Works:
$ https_proxy=socks5://127.0.0.1:1080 ~/Downloads/website-stalker run --all ✔️

Does not work:
$ http_proxy=http://127.0.0.1:1081 ~/Downloads/website-stalker run --all ❌

[EdJoPaTo]

Your second example used http instead of socks5. Does it work with socks5 and the port? Or is the port fixed with socks5?

But it seems like it’s working with the feature flag.

[EdJoPaTo]

As this seems to be working in general (environment variable is picked up and used) I will close this for now. If someone has ideas how to improve things here feel free to comment.

Improvements to the proxy support might also be suggested for reqwest which will end up in this tool.