Getting 'unknown error: net::ERR_CONNECTION_CLOSED' Error about arsenic

[dimaqq]

Doesn't ChromeDriver, by default, only allow connections from localhost?
The log shows that it's being accessed over an ipv4 address instead, and the helpful URL that chromedriver adds to the log suggests using --allowed-ips to allowlist your arsenic host.

[pythonic-shk]

Both the Chromedriver and Chrome browser are locally installed in the Docker Image. I am able to scrape for almost 50 percent of the webpages. Some where in the middle I get the ERR_CONNECTION_CLOSED error and because of it I get asyncio.futures.TimeoutError: and the container exits. What is your recommendation in this case

[pythonic-shk]

@dimaqq I am getting various errors when trying to obtain session
ERR_CONNECTION_CLOSED
ERR_PROXY_CONNECTION_FAILED
unknown error: cannot kill Chrome
DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser
unknown error: Chrome crashed

One of the errors above always occur.

Also when stop_session is initiated at exit. It is sometimes not able to terminate the process (Subprocess) and I get a warning

Although I handle all these errors, The script becomes very slow after a point.

I am testing it with around 2000 urls (All urls belong to same domain) inside docker with RHEL 7 Image. I am trying to run 10 webdriver sessions at a time.

So, Do I need to be aware of any limitations of using this package and minimum resources required to run this without any issues.

[dimaqq]

This is an open source project. You’re welcome to make it better.