HTTP/2 Support and shared clients about thirtyfour

[TilBlechschmidt]

Just to give some spoilers (although this is getting more and more off-topic):

We were able to run ~700 browsers behind a single WebGrid instance across 41 K8s nodes. All with screen recordings running, obviously 😜

[TilBlechschmidt]

Also check out https://github.com/stevepryde/xenon - it works as a lightweight selenium server (but just as fast) and may be possible to add support for HTTP/2. If that works then it might be an option to test against as well.

Oh look another project that noticed the immense resource consumption of Selenium Grid and used Rust for a more efficient implementation 😆

Jokes aside, thanks for the link!

[TilBlechschmidt]

If you have other questions @lucaswxp, we should probably move the discussion over to the appropriate repository though to prevent this issue from going off-topic.

Either the GitHub Discussions or the Discord server would be a good place 🙂

[TilBlechschmidt]

Sharing the Client struct should be relatively easy since it internally uses an Arc to share the connection already so we just have to clone it when creating further WebDriver instances.

[stevepryde]

It probably makes sense to wait until this is supported by Selenium so that it can actually be tested 🤔.

Meanwhile if we want to share the same client across multiple WebDrivers then all we really need is to clone the sender channel used to send webdriver requests. All requests will still be done in series though, not parallel, due to the way the channel is consumed only via a single thread. If this isn't sufficient then a more elaborate solution would be needed.

[stevepryde]

Also check out https://github.com/stevepryde/xenon - it works as a lightweight selenium server (but just as fast) and may be possible to add support for HTTP/2. If that works then it might be an option to test against as well.

[TilBlechschmidt]

You can test it by pointing it at a deployment of this* 🤷‍♂️

It'd obviously increase test complexity by requiring yet another grid software deployed but would work.

_{*The HTTP/2 feature is not yet in a release but that is coming soon — I just need to patch up a few things that broke by migrating to h2.}

[stevepryde]

You can test it by pointing it at a deployment of this* man_shrugging

Ha! This looks awesome, although I need to read more of the docs to see what it can do. And yes I routinely find selenium hub consuming more than 1 core on my PC for what should be effectively a thin proxy. Not to mention consuming upwards of 300MB of memory constantly. Good ol' JVM. Of course browsers are another story :(
If you're able to test against your project then that works too :)

[lucaswxp]

@TilBlechschmidt Just out of curiosity, what kind of machines were those node? Memory/CPU/Datacenter etc.

[TilBlechschmidt]

@TilBlechschmidt Just out of curiosity, what kind of machines were those node? Memory/CPU/Datacenter etc.

I can't remember exactly but they were running in the Google Compute Cloud (GKE) and must've been around 8-16 cores each with probably 32GB of RAM. However, it really depends on your type of workload how much power and memory you need. Browsers typically have a large startup overhead but once they are running, the requirements are pretty modest. In the future, I plan to do some more detailed analysis based on real-world workloads.

At some point, I'll have to write a blog post about all this but I'm afraid it'll be some time until that happens 😄

On-premise we've been running 50-100 browsers on a 40 core, 256GB machine which was not anywhere near its capacity limits -- again all depends on workload as this was while running a stress test designed for WebGrid and not the browsers. Then again, it ended up being more of a stress test for our K8s Control plane ...