stevepryde / thirtyfour

Selenium WebDriver client for Rust, for automated testing of websites

thirtyfour

Thirtyfour is a Selenium / WebDriver library for Rust, for automated website UI testing.

It supports the W3C WebDriver v1 spec. Tested with Chrome and Firefox although any W3C-compatible WebDriver should work.

Why is it called "thirtyfour" ?

Thirty-four (34) is the atomic number for the Selenium chemical element (Se) ⚛️.

Getting Started

Check out The Book 📚!

Features

• All W3C WebDriver and WebElement methods supported
• Create new browser session directly via WebDriver (e.g. chromedriver)
• Create new browser session via Selenium Standalone or Grid
• Find elements (via all common selectors e.g. Id, Class, CSS, Tag, XPath)
• Send keys to elements, including key-combinations
• Execute Javascript
• Action Chains
• Get and set cookies
• Switch to frame/window/element/alert
• Shadow DOM support
• Alert support
• Capture / Save screenshot of browser or individual element as PNG
• Chrome DevTools Protocol (CDP) support (limited)
• Advanced query interface including explicit waits and various predicates
• Component Wrappers (similar to Page Object Model)

Feature Flags

• rustls-tls: (Default) Use rustls to provide TLS support (via reqwest).
• native-tls: Use native TLS (via reqwest).
• component: (Default) Enable the Component derive macro (via thirtyfour_macros).

Examples

The examples assume you have chromedriver running on your system.

You can use Selenium (see instructions below) or you can use chromedriver directly by downloading the chromedriver that matches your Chrome version, from here: https://chromedriver.chromium.org/downloads

Then run it like this:

chromedriver

Example (async):

To run this example:

cargo run --example tokio_async

use thirtyfour::prelude::*;

#[tokio::main]
async fn main() -> WebDriverResult<()> {
     let caps = DesiredCapabilities::chrome();
     let driver = WebDriver::new("http://localhost:9515", caps).await?;

     // Navigate to https://wikipedia.org.
     driver.goto("https://wikipedia.org").await?;
     let elem_form = driver.find(By::Id("search-form")).await?;

     // Find element from element.
     let elem_text = elem_form.find(By::Id("searchInput")).await?;

     // Type in the search terms.
     elem_text.send_keys("selenium").await?;

     // Click the search button.
     let elem_button = elem_form.find(By::Css("button[type='submit']")).await?;
     elem_button.click().await?;

     // Look for header to implicitly wait for the page to load.
     driver.find(By::ClassName("firstHeading")).await?;
     assert_eq!(driver.title().await?, "Selenium - Wikipedia");
    
     // Always explicitly close the browser.
     driver.quit().await?;

     Ok(())
}

Minimum Supported Rust Version

The MSRV for thirtyfour is currently 1.75 and will be updated as needed by dependencies.

LICENSE

This work is dual-licensed under MIT or Apache 2.0. You can choose either license if you use this work.

See the NOTICE file for more details.

SPDX-License-Identifier: MIT OR Apache-2.0

thirtyfour's Issues

Convenience method for performing actions in a new tab

Most of the time when I perform an operation in a new tab, I simply need to open a new tab, do some stuff, then close it and return to the original. A convenience method or trait on WebDriver to do this would go a long way to decluttering some of my code. I quickly wrote up the simple function version of what I'm talking about.

use std::future::Future;
use thirtyfour::prelude::*;
use tokio;

pub async fn in_new_tab<F, Fut, T>(driver: &WebDriver, f: F) -> WebDriverResult<T>
where
    F: FnOnce() -> Fut,
    Fut: Future<Output = WebDriverResult<T>>,
{
    // Opening Tab ----------------------------------------
    let handle = driver.current_window_handle().await?;
    driver
        .execute_script(r#"window.open("about:blank", target="_blank");"#)
        .await?;
    let handles = driver.window_handles().await?;
    driver.switch_to().window(&handles[1]).await?;
    // ----------------------------------------------------

    let result = f().await;

    // Closing Tab --------------------------------------------
    driver.execute_script(r#"window.close();"#).await?;
    driver.switch_to().window(&handle).await?;
    // --------------------------------------------------------

    result
}

#[tokio::main]
async fn main() -> WebDriverResult<()> {
    let caps = DesiredCapabilities::firefox();
    let driver = WebDriver::new(&format!("http://localhost:{}/wd/hub", "4444"), &caps).await?;

    driver.get("https://google.com").await?;

    in_new_tab(&driver, || {
        async {
            driver.get("https://wikipedia.com").await?;
            Ok(())
        }
    }).await?;

    driver.get("https://youtube.com").await?;
    driver.quit().await?;

    Ok(())
}

I started to write an async_trait to implement on WebDriver but I ran into issues with the closure not being Send. What do you think about something similar to this becoming part of the WebDriverCommands trait? So the we could have something like

driver.in_new_tab(|| { async { Ok(()) } }).await?;

or for the nightly folks with async closures

driver.in_new_tab(async || { Ok(()) }).await?;

feature: innerHtml && outerHtml

Errors: browserVersion cannot be empty and no match capabilities found.

I am trying to build a small web scaper while learning rust. However, I am unable to connect to chromedriver.

Arch Linux
rustc 1.44.0-nightly
ChromeBrowser Version: 83.0.4103.61
Chromedriver Version: 83.0.4103.39

Here is my original code:

let caps = DesiredCapabilities::chrome();
let driver = WebDriver::new("http://localhost:4444", &caps)
    .await
    .expect("could not connect to webdriver");

This results in a invalid argument: cannot parse capability error the Rust DEBUG logs:

thread 'main' panicked at 'could not connect to webdriver: InvalidArgument(WebDriverErrorInfo { status: 400, error: "", value: WebDriverErrorValue { message: "invalid argument: cannot parse capability: browserVersion\nfrom invalid argument: cannot be empty", error: Some("invalid argument"), stacktrace: Some("#0 0x5646c0b6c579 <unknown>\n"), data: None } })', src/main.rs

And the chromedriver logs:

[INFO]: RESPONSE InitSession ERROR invalid argument: cannot parse capability:
 browserVersion from invalid argument: cannot be empty
[DEBUG]: Log type 'driver' lost 1 entries on destruction
[INFO]: COMMAND InitSession {
   "capabilities": {
      "alwaysMatch": {
         "browserName": "chrome",
         "browserVersion": "",
         "platformName": "ANY"
      },
      "firstMatch": [ {

      } ]
   },
   "desiredCapabilities": {
      "browserName": "chrome",
      "platform": "ANY",
      "version": ""
   }
}
[INFO]: RESPONSE InitSession ERROR invalid argument: cannot parse capability: 
browserVersion from invalid argument: cannot be empty
[DEBUG]: Log type 'driver' lost 1 entries on destruction

So I tried adding the browserVersion manually:

let mut caps = DesiredCapabilities::chrome();
caps.set_version("83.0.4103.61")
    .expect("failed to set browserVersion for webdriver");
let driver = WebDriver::new("http://localhost:4444", &caps)
    .await
    .expect("could not connect to webdriver");

But I get a invalid argument: cannot be empty error in Rust DEBUG logs:

thread 'main' panicked at 'could not connect to webdriver: InvalidArgument(WebDriverErrorInfo { status: 400, error: "", value: WebDriverErrorValue { message: "invalid argument: cannot parse capability: browserVersion\nfrom invalid argument: cannot be empty", error: Some("invalid argument"), stacktrace: Some("#0 0x5646c0b6c579 <unknown>\n"), data: None } })', src/main.rs

And a "No matching capabilities found" error in the chromedriver logs:

[INFO]: RESPONSE InitSession ERROR session not created: No matching capabilities found
[DEBUG]: Log type 'driver' lost 1 entries on destruction
[INFO]: COMMAND InitSession {
   "capabilities": {
      "alwaysMatch": {
         "browserName": "chrome",
         "browserVersion": "83.0.4103.61",
         "platformName": "ANY"
      },
      "firstMatch": [ {

      } ]
   },
   "desiredCapabilities": {
      "browserName": "chrome",
      "platform": "ANY",
      "version": "83.0.4103.61"
   }
}
[INFO]: RESPONSE InitSession ERROR session not created: No matching capabilities found
[DEBUG]: Log type 'driver' lost 1 entries on destruction

What am I doing wrong here?

Check if element exists

Hi,

is there a way to check if a element on a page exists. Currently I'm using:

async fn html_contains_any_element_webdriver(driver: &WebDriver, url: &str, elements: Vec<&str>) -> Result<bool> {
    driver.get(url).await?;

    for element in elements.into_iter() {
        let filtered_elements = driver.find_element(By::XPath(element)).await;

        if filtered_elements.is_ok() {
            return Ok(true);
        }
    }

    Ok(false)
}

But this just returns after 30 seconds.
Why does the crate needs 30 seconds to check if the current html contains an given element? (30 seconds for each element)

T

Build Error

Thirtyfour does not build with "async-std-runtime" feature enabled.

/thirtyfour-0.15.1/src/http_async/surf_async.rs:40:31
   |
40 |             request = request.body_json(&x)?;
   |                               ^^^^^^^^^ method not found in `surf::RequestBuilder`

Idiomatically await elements of a Vec

Hey there! This is really about Async/Await in general but as I'm encountering the issue in the context of Selenium I thought you might have a good answer. I'm processing hrefs from a list of links scraped from the page, and I want to await every call to get the href attribute, but I can't do async in a map.
My code:

// Pull the urls from the href attribute of each link and filter out urls that have already been processed
            let new_order_urls = new_order_links.iter()
            .map(|link| { link.get_attribute("href").await })
            .map(|href_result_option| {
                if let Ok(Some(url)) = href_result_option {
                    return url;
                }

                // Gets filtered out of the list, can't crash on error or no href, will need to try to get the url on the next go round
                "error_url".to_string()
            })
            .filter(|url| !enforced_orders.contains(url)).collect();

This of course gives the error

error[E0728]: `await` is only allowed inside `async` functions and blocks
  --> src\unassigned_email_queue.rs:68:27
   |
68 |             .map(|link| { link.get_attribute("href").await })
   |                  ------   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ only allowed inside `async` functions and blocks
   |                  |
   |                  this is not `async`

What is the best way to call and await get_attribute for each link in the list?

Error with [ElementNotInteractable] when test with React app input element

I was testing with async API of this crate with my React frontend at www.steadylener.com

What I want is very simple

1. I want to grab input part
2. Type Rust
3. Then, find blog titles with get_elements API
4. Write Rust code to make the result file with .md format

The code I used is similar to this.

use thirtyfour::error::WebDriverResult;
use thirtyfour::{
    By, 
    DesiredCapabilities, 
    WebDriver,
    Keys,
    TypingData,
};

use tokio;

#[tokio::main]
async fn main() -> WebDriverResult<()> {
     let caps = DesiredCapabilities::chrome();
     let driver = WebDriver::new("http://localhost:4444/wd/hub", &caps).await?;

     let target = "https://www.steadylearner.com";
     println!("Wait before you play with {}", &target);

     // Navigate to https://www.steadylearner.com.
     driver.get(target).await?;
     println!("{:#?}", &driver.title().await?); // Should be "Steadylearner Home"
     let search_input = driver.find_element(By::Id("searchInput")).await?;
     println!("{}", &search_input.get_attribute("value").await?); // ""

     // The problem happens with the code below
     search_input.send_keys("Rust").await?;
     search_input.send_keys(Keys::Enter).await?;
     println!("{:#?}", &driver.title().await?); // "Posts for Rust"
    
     Ok(())
}

But, after a long wait, it shows this error.

Error: ElementNotInteractable(WebDriverErrorInfo { status: 400, error: "", value: WebDriverErrorValue { message: "element not interactable\n  (Session info: chrome=79.0.3945.117)", error: Some("element not interactable"), stacktrace: Some("#0 0x5566c2005479 <unknown>\n"), data: None } })

I don't use Chrome browser, but it is installed in my machine and used it to follow the example easily.

My doubts are

1. Is this problem of the configuration of my machine?
2. Is the crate or selnium have a problem with the React app(Steadylearner)?
3. Or there are a problem in my code?

I want to test Selenium with React or other single page apps. But, I haven't used it for a long time. If I missed something, please let me know.

Element query proposal

When it comes to querying elements, the built-in WebDriver element selectors are good but often we need more advanced functionality, for example polling or filtering based on some predicate.

In my experience I have found it better to turn off implicit waits and implement polling separately outside of selenium. This provides far greater control over element queries, including powerful any/all filtering or polling multiple selectors in parallel and returning whichever element is found first (useful when CSS has changed between old/new versions of a website).

This also allows for different retry criteria. You might wish to retry a set number of times regardless of how long it takes, or you might only care about retrying within a set duration, or you might want some combination of the two (to avoid the scenario where a slow network means you never actually retried due to the first query taking too long).

One possibility is a functional approach, which should allow for a composable queries. Something like this (none of this has been tested):

fn poll_query<F>(timeout: Duration, interval: Duration, f: F) -> F
where 
    F: Fn() -> WebDriverResult<Vec<WebElement>>
{
    return || {
        let now = Instant::now();
        while (now.elapsed() < timeout) {
            match f() {
                Ok(x) => return Ok(x),
                Err(WebDriverError::NoSuchElement(_)) => break,
                Err(e) => return Err(e)
            }
            sleep(interval);
        }
        // Always do one last find.
        f()
    }
}

let found = poll_query(Duration::new(10, 0), Duration::new(0, 200), || e.find_elements(By::Tag("sometag")))?;

Another possibility is a builder approach, which would allow something like this:

let found = ElementQuery(driver).poll(Duration::new(10, 0), Duration::new(0, 200)).find(By::Tag("sometag"))?;

In both cases you could reduce the code required by having default timeout/interval that can be configured/overridden as needed.

So that would shorten to something like:
Functional version:

let found = poll_query(|| e.find_elements(By::Tag("sometag")))?;

Builder version:

let found = ElementQuery(driver).poll().find(By::Tag("sometag"))?;

Both would allow further expansion using functions such as:
find_any(Vec<By>) - return the first one that is non-empty - if nested below poll() this will execute both queries once per poll iteration.
with_retry(num_tries: u32) - retry the query a specified number of times if not found, rather than using a fixed timeout.
poll_missing() - poll until the query returns an empty vec.

Also since these return a Vec you can easily iterate and filter as needed as well. The filtering could even be applied at any stage of the pipeline (easier in the functional version).

Personally I prefer the functional version since it allows for easier composition and it's easier to add your own functions. There are some pros and cons to each pattern - and I'm not opposed to supporting both (the builder pattern would simply wrap the functional pattern).

All of this will go in a separate crate at least until it is deemed stable. That allows for more experimentation and rapid version bumps without affecting this crate, which by now should be looking at stabilization as much as possible.

Is it possible to add an chrome extension?

I have a nodejs chrome driver script that I would like to port to rust.

In project I have a packed CRX file that I would like to include into chrome before running chromedriver. Is that possible using thrityfour?

Using predefined session

Hello, is there any way to not open a new session but reuse an existing one ? I haven't seen a way to do that in the doc.
Thanks for the great project, best regards.

More cookie getters

Hello!

I'm creating a web scraper. I have a website that's hard to log into using plain POST, so I create a temporary Selenium instance, log in and copy the session cookies to be later reused by reqwest::Client.

However, I ran into a problem - I have to access the name field of the Cookie struct, but it's private. I temporarily resorted to creating a MyCookie struct and invoking unsafe { std::mem::transmute<Cookie, MyCookie>(instance) }. Unfortunately this is risky, since Cookie is not a #[repr(C)] struct.

Please consider adding more getters to the struct, that would be immensely helpful.

DesiredCapabilities::default panic

From my perspective DesiredCapabilities::default() isn't able to be any useful. It might be a good candidate for being deleted?

'tests::test_ignore_url' panicked at 'called `Option::unwrap()` on a `None` value', /home/zhiburt/.cargo/registry/src/github.com-1ecc6299db9ec823/thirtyfour-0.15.1/src/common/capabilities/desiredcapabilities.rs:34:19

click() has no effect on <a> elements

Hi Guys,

I'm trying to trigger a click on a tag.
click() on form elements works.
But on elements it doesn't work. Is there a different method for such a purpose?

T

Can I listen CDP events in thirtyfour?

We've able to execute cdp command in thirtyfour, but I've not found a way to listen CDP event.

Sometimes, I hope to use fetch.enable to catch all network request in a page. When page send a network request, a requestPaused event is issued, and I should capture this event and send fulfillRequest or failRequest.

Like this: https://stackoverflow.com/questions/59420493/how-to-use-chrome-devtools-protocol-in-selenium-using-python-for-capturing-htt

Firefox fails when calling window_handles()

Hi I am using this with geckodriver which is connected to my already running instance of firefox.

When I tried to run the window_handles() function it failed with the following error and crashed firefox (but not geckodriver):

Error while executing action: Unknown error:
    Status: 500
    Additional info:
        TypeError: browser.browsingContext is null
        Error: unknown error
        Stacktrace:
            GeckoDriver.prototype.getIdForBrowser@chrome://marionette/content/driver.js:1398:15
            get@chrome://marionette/content/driver.js:239:28
            GeckoDriver.prototype.getWindowHandles@chrome://marionette/content/driver.js:1443:3
            despatch@chrome://marionette/content/server.js:297:40
            execute@chrome://marionette/content/server.js:267:16
            onPacket/<@chrome://marionette/content/server.js:240:20
            onPacket@chrome://marionette/content/server.js:241:9
            _onJSONObjectReady/<@chrome://marionette/content/transport.js:504:2

discussion: support for `Element::is_*` functions family

What do you think about adding this list of functions?

• is_visible
• is_present
• is_enabled
• is_clickable
• etc.

It useful to have these with wait_for_* type functions. To not have an explicit sleep. Which speed up general testing scripts. And make them more reliable.

Honestly I've spend a few days playing with this in order to implement a month ago. And it seems like not an trivial task at all 😞

I've also gone through the seleniums libraries code base to check how they implemented it. I expected to see a simple js script which does a check or some check by driver functions itself.

As far as I got lucky to understand they reuse the same javascript files in all libraries (They call these files atoms).

To sum up I post this issues just to hear from you what do you think about such functionality. And if you think it appropriate the best way to have it done.

An example of display property which is an equivalent of is_displayed

https://github.com/SeleniumHQ/selenium/blob/941dc9c6b2e2aa4f701c1b72be8de03d4b7e996a/dotnet/src/webdriver/Remote/RemoteWebElement.cs#L186

Timed out receiving message from renderer: -0.019

After
driver.set_page_load_timeout( Duration::new(5_u64, 0)).await?;

Sometimes:

[1619342394.318][SEVERE]: Timed out receiving message from renderer: 5.000
[1619342394.337][SEVERE]: Timed out receiving message from renderer: -0.019
[1619342394.359][SEVERE]: Timed out receiving message from renderer: -0.019

Is it a normal value -0.019 ?
Timeout with a negative value ?

Thank you for perfect crates!

Capabilities & ChromeOptions example

hi all, who can give code example use Capabilities & ChromeOptions with thirtyfour same here:
https://chromedriver.chromium.org/capabilities
ex:
Ruby
caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {"args" => [ "--disable-web-security" ]})
driver = Selenium::WebDriver.for :remote, url: 'http://localhost:4444/wd/hub' desired_capabilities: caps
thank you so much.

What are the default durations for the WebDriver::Query timeout and polling / What is a reasonable polling time

What are the default durations for the WebDriver::Query timeout and polling? I'm working with a website with slow load times and I'd like to increase the explicit wait timeout. I know I can set this with .wait() on the query but I'm not sure what a reasonable default for the poll time is, as I've never had a need to manually set it before.

Authorization header in async-std-runtime

It looks like the async-std-runtime is not working properly because the header is not being configured as in the tokio-runtime version.

Usage of invalid driver causes an endless execution?

Hey @stevepryde,
so I encounter if you're running in your environment chromedriver but provide thirtyfour with firefox Capabilities Webdriver::new function never returns.

You can run this test to repeat it.

https://github.com/Plato-solutions/doonop/blob/d5ace574ac64477a9dea3e07b4db66a63d2d01ab/tests/crawling.rs#L27

Firefox is set but using chrome

Please read this: #Firefox is set but using chrome

README example not working, getting `Error: SessionNotCreated, Unable to create new service: ChromeDriverService`

Steps to reproduce:

(1) Run docker run --rm -d --network host --name selenium-server -v /dev/shm:/dev/shm selenium/standalone-chrome

(2) Run the example (async or sync)

Output:

Error: SessionNotCreated(WebDriverErrorInfo { status: 500, error: "", value: WebDriverErrorValue { message: "Unable to create new service: ChromeDriverService\nBuild info: version: \'3.141.59\', revision: \'e82be7d358\', time: \'2018-11-14T08:25:53\'\nSystem info: host: \'dsternlicht\', ip: \'172.16.10.252\', os.name: \'Linux\', os.arch: \'amd64\', os.version: \'5.4.15-arch1-1\', java.version: \'1.8.0_232\'\nDriver info: driver.version: unknown", error: Some("session not created"), stacktrace: Some("org.openqa.selenium.SessionNotCreatedException: Unable to create new service: ChromeDriverService\nBuild info: version: \'3.141.59\', revision: \'e82be7d358\', time: \'2018-11-14T08:25:53\'\nSystem info: host: \'dsternlicht\', ip: \'172.16.10.252\', os.name: \'Linux\', os.arch: \'amd64\', os.version: \'5.4.15-arch1-1\', java.version: \'1.8.0_232\'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.grid.session.remote.ServicedSession$Factory.lambda$get$0(ServicedSession.java:135)\n\tat org.openqa.selenium.grid.session.remote.ServicedSession$Factory.apply(ServicedSession.java:152)\n\tat org.openqa.selenium.remote.server.ActiveSessionFactory.lambda$apply$12(ActiveSessionFactory.java:180)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:440)\n\tat java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)\n\tat java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)\n\tat java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)\n\tat java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)\n\tat java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)\n\tat java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)\n\tat java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)\n\tat java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531)\n\tat org.openqa.selenium.remote.server.ActiveSessionFactory.apply(ActiveSessionFactory.java:183)\n\tat org.openqa.selenium.remote.server.NewSessionPipeline.lambda$null$2(NewSessionPipeline.java:66)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)\n\tat java.util.Collections$2.tryAdvance(Collections.java:4719)\n\tat java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)\n\tat java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)\n\tat java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)\n\tat java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)\n\tat java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)\n\tat java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531)\n\tat org.openqa.selenium.remote.server.NewSessionPipeline.lambda$createNewSession$3(NewSessionPipeline.java:69)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.DistinctOps$1$2.accept(DistinctOps.java:175)\n\tat java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)\n\tat java.util.stream.Streams$StreamBuilderImpl.tryAdvance(Streams.java:405)\n\tat java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:728)\n\tat java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)\n\tat java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)\n\tat java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)\n\tat java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)\n\tat java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)\n\tat java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531)\n\tat org.openqa.selenium.remote.server.NewSessionPipeline.createNewSession(NewSessionPipeline.java:72)\n\tat org.openqa.selenium.remote.server.commandhandler.BeginSession.execute(BeginSession.java:65)\n\tat org.openqa.selenium.remote.server.WebDriverServlet.lambda$handle$0(WebDriverServlet.java:235)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.reflect.InvocationTargetException\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.openqa.selenium.grid.session.remote.ServicedSession$Factory.lambda$get$0(ServicedSession.java:131)\n\t... 50 more\nCaused by: java.lang.RuntimeException: Unable to find a free port\n\tat org.openqa.selenium.net.PortProber.findFreePort(PortProber.java:67)\n\tat org.openqa.selenium.remote.service.DriverService$Builder.build(DriverService.java:351)\n\tat org.openqa.selenium.chrome.ChromeDriverService.createDefaultService(ChromeDriverService.java:94)\n\t... 55 more\n"), data: None } })

Error, the webdriver won't quit correctly.

I was testing the async api with a runtime support.
And a very strange behaviour appeared, the code bellow won't move out at the end of the block_on closure.

The issue has been tested with 0.8 and 0.9 version of thirtyfour, tokio 0.2.19 and using the selenium docker command given in the readme.

use thirtyfour::prelude::*;
use tokio;

fn main() -> WebDriverResult<()> {

    let mut runtime = tokio::runtime::Builder::new()
        .basic_scheduler()
        .enable_all()
        .build().unwrap();

    let result : WebDriverResult<()> = runtime.block_on(async {

        let caps = DesiredCapabilities::chrome();
        let driver = WebDriver::new("http://localhost:4444/wd/hub", &caps).await?;

        println!("{:?}", driver);
        Ok(())
    });

    println!("End");

    result
}

After some investigation, the problem seems to come from the function drop (webdriver.rs:159).

fn drop(&mut self) {
        if self.quit_on_drop && !(*self.session_id).is_empty() {
            if let Err(e) = block_on(self.quit()) {
                error!("Failed to close session: {:?}", e);
            }
        }
    }

For some unknown reason, the block_on(self.quit()) loops without returning.
I tried to explicitly call driver.quit() before the end of the closure, but it didn't solve the problem.

Thanks

Tokio panic with Unordered

Hi Guys,

this lines causes an panic:

 let mut filtered_needed_elements = FuturesUnordered::new();
    for element in needed_elements.into_iter() {
        filtered_needed_elements.push(driver.find_element(By::XPath(element)));
    }
    while let Some(filtered_needed_element) = filtered_needed_elements.next().await {
        if filtered_needed_element.is_ok() {
            return Ok(false);
        }
    }

This panic occurs somewhen after executing this lines.

Here the stacktrace:

thread 'tokio-runtime-worker' panicked at 'Failed to send response: Err(NoSuchElement(WebDriverErrorInfo { status: 404, error: "", value: WebDriverErrorValue { message: "Unable to locate element: //*[@id=\"buy-now-button\"]", error: Some("no such element"), stacktrace: Some("WebDriverError@chrome://marionette/content/error.js:181:5\nNoSuchElementError@chrome://marionette/content/error.js:393:5\nelement.find/</<@chrome://marionette/content/element.js:306:16\n"), data: None } }))', C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\thirtyfour-0.23.0\src\session.rs:47:30
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\std\src\panicking.rs:493
   1: core::panicking::panic_fmt
             at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\core\src\panicking.rs:92
   2: core::option::expect_none_failed
             at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\core\src\option.rs:1268
   3: core::result::Result<tuple<>, core::result::Result<serde_json::value::Value, thirtyfour::error::WebDriverError>>::expect<tuple<>,core::result::Result<serde_json::value::Value, thirtyfour::error::WebDriverError>>        
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\result.rs:933
   4: thirtyfour::session::session_runner::{{closure}}
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\thirtyfour-0.23.0\src\session.rs:47
   5: core::future::from_generator::{{impl}}::poll<generator-0>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\future\mod.rs:80
   6: tokio::runtime::task::core::{{impl}}::poll::{{closure}}<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\core.rs:173
   7: tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<core::future::from_generator::GenFuture<generator-0>>>::with_mut<tokio::runtime::task::core::Stage<core::future::from_generator::GenFuture<generator-0>>,core::task::poll::Poll<tup
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\loom\std\unsafe_cell.rs:14
   8: tokio::runtime::task::core::Core<core::future::from_generator::GenFuture<generator-0>, alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\core.rs:158
   9: tokio::runtime::task::harness::{{impl}}::poll::{{closure}}<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\harness.rs:107
  10: core::ops::function::FnOnce::call_once<closure-0,tuple<>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ops\function.rs:227
  11: std::panic::{{impl}}::call_once<core::task::poll::Poll<core::result::Result<tuple<>, tokio::runtime::task::error::JoinError>>,closure-0>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panic.rs:322
  12: std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<tuple<>, tokio::runtime::task::error::JoinError>>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panicking.rs:379
  13: thirtyfour::webdrivercommands::start_session::{{closure}}::_::{{impl}}::deserialize::{{impl}}::expecting
  14: std::panicking::try<core::task::poll::Poll<core::result::Result<tuple<>, tokio::runtime::task::error::JoinError>>,std::panic::AssertUnwindSafe<closure-0>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panicking.rs:343
  15: std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<tuple<>, tokio::runtime::task::error::JoinError>>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panic.rs:396
  16: tokio::runtime::task::harness::Harness<core::future::from_generator::GenFuture<generator-0>, alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\harness.rs:89
  17: tokio::runtime::task::raw::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\raw.rs:104
  18: tokio::runtime::task::raw::RawTask::poll
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\raw.rs:66
  19: tokio::runtime::task::Notified<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::run<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\mod.rs:171
  20: tokio::runtime::thread_pool::worker::{{impl}}::run_task::{{closure}}
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\thread_pool\worker.rs:370
  21: tokio::coop::with_budget::{{closure}}<core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core, alloc::alloc::Global>, tuple<>>,closure-0>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\coop.rs:121
  22: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::try_with<core::cell::Cell<tokio::coop::Budget>,closure-0,core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core, alloc::alloc::Global>, tuple<>>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\thread\local.rs:272
  23: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::with<core::cell::Cell<tokio::coop::Budget>,closure-0,core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core, alloc::alloc::Global>, tuple<>>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\thread\local.rs:248
  24: tokio::coop::with_budget
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\coop.rs:114
  25: tokio::coop::budget
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\coop.rs:98
  26: tokio::runtime::thread_pool::worker::Context::run_task
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\thread_pool\worker.rs:348
  27: tokio::runtime::thread_pool::worker::Context::run
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\thread_pool\worker.rs:318
  28: tokio::runtime::thread_pool::worker::run::{{closure}}
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\thread_pool\worker.rs:303
  29: tokio::macros::scoped_tls::ScopedKey<tokio::runtime::thread_pool::worker::Context>::set<tokio::runtime::thread_pool::worker::Context,closure-0,tuple<>>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\macros\scoped_tls.rs:61
  30: tokio::runtime::thread_pool::worker::run
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\thread_pool\worker.rs:300
  31: tokio::runtime::thread_pool::worker::{{impl}}::launch::{{closure}}
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\thread_pool\worker.rs:279
  32: tokio::runtime::blocking::task::{{impl}}::poll<closure-0,tuple<>>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\blocking\task.rs:42
  33: tokio::runtime::task::core::{{impl}}::poll::{{closure}}<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\core.rs:173
  34: tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<closure-0>>>::with_mut<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<closure-0>>,core::task::poll::Po
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\loom\std\unsafe_cell.rs:14
  35: tokio::runtime::task::core::Core<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\core.rs:158
  36: tokio::runtime::task::harness::{{impl}}::poll::{{closure}}<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\harness.rs:107
  37: core::ops::function::FnOnce::call_once<closure-0,tuple<>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ops\function.rs:227
  38: std::panic::{{impl}}::call_once<core::task::poll::Poll<core::result::Result<tuple<>, tokio::runtime::task::error::JoinError>>,closure-0>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panic.rs:322
  39: std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<tuple<>, tokio::runtime::task::error::JoinError>>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panicking.rs:379
  40: tokio::time::driver::wheel::level::slot_for
  41: std::panicking::try<core::task::poll::Poll<core::result::Result<tuple<>, tokio::runtime::task::error::JoinError>>,std::panic::AssertUnwindSafe<closure-0>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panicking.rs:343
  42: std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<tuple<>, tokio::runtime::task::error::JoinError>>>
             at C:\Users\TiTan\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panic.rs:396
  43: tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedul
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\harness.rs:89
  44: tokio::runtime::task::raw::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\raw.rs:104
  45: tokio::runtime::task::raw::RawTask::poll
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\raw.rs:66
  46: tokio::runtime::task::Notified<tokio::runtime::blocking::schedule::NoopSchedule>::run<tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\task\mod.rs:171
  47: tokio::runtime::blocking::pool::Inner::run
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\blocking\pool.rs:277
  48: tokio::runtime::blocking::pool::{{impl}}::spawn_thread::{{closure}}
             at C:\Users\TiTan\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.0.1\src\runtime\blocking\pool.rs:257
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

T

Is there any "Selector"?

Hi, I want to select a drop-down item in the selector.
In Selenium with Python, I can do it by :

from selenium.webdriver.support.ui import Select

s1 = Select(driver.find_element_by_id('s1Id'))

s1.select_by_index(1)
s1.select_by_value("o2") 
s1.select_by_visible_text("o3") 

Is there any equivalent to do these things in ThirtyFour?

Connection refused (os error 111) with Firefox

Hello,

this error: error sending request for url (http://localhost:6565/session): error trying to connect: tcp connect error: Connection refused (os error 111) is returned by the WebClient new function when I call it.

How to reproduce:

1. run geckodriver using ./path/to/geckodriver --port=6565
2. execute an application with the following code: WebDriver::new("http://localhost:6565/", my_capabilities).await?;

Implicitly waiting before calling ::new() doesn't work either.

I can't figure out how to use Feature Flag to make thirtyfour work correctly.

Hi, I am back to test your project again and I think that the API have changed a lot especially using a feature flag to use async-std, tokio, sync etc.

I keep having the project work at runtime because I am not sure how I can configure features flags to use tokio and async feature in Cargo.toml

[package]
name = "login"
version = "0.1.0"
authors = ["www.steadylearner.com/blog/search/Rust"]
edition = "2018"

[dependencies]
tokio = "0.2.21"

# https://doc.rust-lang.org/cargo/reference/features.html#the-features-section
# https://github.com/stevepryde/thirtyfour#feature-flags
# https://rust-random.github.io/book/features.html

[dependencies.thirtyfour]
version = "0.12.0"
default-features = false
features = ["tokio-runtime"]

It is currently working with $cago c with the main.rs but when I use $cargo run it shows the error below and I think that it is from Cargo.toml file.

Error: ReqwestError(reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) })
[Finished running. Exit status: 1]

Can you share the fully working example? I just need to visit that I can visit the reddit website with your project.

p.s It is difficult to what I need to do to make feature flags work correctly. Would you include them at separate folders or README.md at your project?

It will be better for beginners like myself to use your project better.

I want to commit the Reddit login example to your project so please help me to get through this.

[Feature request] find_elements() that doesn't wait.

I wanted to migrate my fantoccini project to thirtyfour since it had find_element timeouts. But when was trying to migrate I realized that I need a method that doesn't wait for the element and returns the current element even if it doesn't exist/hasn't loaded. I know I can set the timeout but it became annoying when i had to set the timeout to 0 and then reset it back. And bacause i didn't always have a handle to WebDriver.

What I want:

• A version of find_element() and find_elements() that doesn't wait and returns the current state (AKA error that no element was found).
• If possible I would like to get a reference to the element if not a simple boolean would suffice.

Why:

• I need to know if a element exists and/or is loaded.

How I would implement it:

• The optimal thing would be to have a wait_find_element() and find_element() but that would break code compatibility,
• Have an argument defining the timeout
• Have a element_exist() method, or with a different name like has_element(), is_present(), that returns a bool
• Have a find_optional_element() that returns a Option
• Have afind_present_element() that returns a Result.

NOTE OF course all of them would be wrapped with a WebdriverResult.

Please if you could find a solution because this is the only problem stopping me from migration.

Impl driver.execute_cdp_cmd API

This is the api of selenium, hope this library can achieve this function: execute_cdp_cmd
This function is very useful and can be used to limit the operation of the webdriver, For example, the following example is:
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source": """
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined
})
"""
})

Thanks !

"Content-Type header does not indicate utf-8 encoded json: application/json" error with Selenium running on Docker

Hi,

I'm facing the following error while trying to start a new WebDriver session (WebDriver::new(selenium_server_location, &caps).await?;) with a local standalone Chrome selenium docker container (docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:85.0):

Error: UnknownError(WebDriverErrorInfo { status: 500, error: "", value: WebDriverErrorValue { message: "Content-Type header does not indicate utf-8 encoded json: application/json", error: Some("unknown error"), stacktrace: Some(""), data: None } })

The selenium_server_location is http://localhost:4444/wd/hub.

The issue is not present if I run the chromedriver locally with chromedriver --port=4444.

I tried to supply RUST_BACKTRACE=1 but I didn't get any additional info about the error.

P.S: do not hesitate to ask for any kind of additional info 🙇

WebElement::find_element lifetime is fully correct?

Hey @stevepryde are these lifetime discrepancies are intentional?

WebElement

    pub async fn find_element(&self, by: By<'a>) -> WebDriverResult<WebElement<'a>>;

WebDriver

    async fn find_element<'a>(&'a self, by: By<'_>) -> WebDriverResult<WebElement<'a>>;

What I am trying to say is that why we bound to 'a lifetime in a WebElement?

*Kind of bump on the issue with this 😥

Multiple WebDrivers One Window

I'm working on a project where I manage between 3 and 7 WebDrivers at any given time. They are really cluttering up my desktop while they're running. Is the a way to run multiple unique WebDrivers in the same window just on different tabs?

Tokio Time vs Core Time for query.wait()

The wait method on query is documented as taking core::time::Duration. Can I / should I use the tokio::time::Duration here, and what are the effects in terms of yielding a thread vs. blocking (I would like to avoid blocking). Sorry I'm still wrapping my head around async w/Tokio.

Downloading a file

Hi,

I have a form that I want to submit, which then prompts a file download with Content-Disposition: attachment

Is it possible to download this file using thirtyfour/chromedriver, to a specific directory, or somehow interact with the download request?

Add custom parameter to FirefoxProfile?

Hello @stevepryde I was trying to set a custom User-Agent for geckodriver as I understand to set it we should do the following in python

profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")

But as I see there's no such option currently

thirtyfour/src/common/capabilities/firefox.rs

Lines 126 to 139 in dd8830d

	pub struct FirefoxProfile {
	#[serde(rename = "webdriver_accept_untrusted_certs", skip_serializing_if = "Option::is_none")]
	pub accept_untrusted_certs: Option<bool>,
	#[serde(rename = "webdriver_assume_untrusted_issuer", skip_serializing_if = "Option::is_none")]
	pub assume_untrusted_issuer: Option<bool>,
	#[serde(rename = "webdriver.log.driver", skip_serializing_if = "Option::is_none")]
	pub log_driver: Option<FirefoxProfileLogDriver>,
	#[serde(rename = "webdriver.log.file", skip_serializing_if = "Option::is_none")]
	pub log_file: Option<String>,
	#[serde(rename = "webdriver.load.strategy", skip_serializing_if = "Option::is_none")]
	pub load_strategy: Option<String>,
	#[serde(rename = "webdriver_firefox_port", skip_serializing_if = "Option::is_none")]
	pub webdriver_port: Option<u16>,
	}

I could imagine that there's a bigger list of such settings so might a custom parameter not a bad idea?

PS: I didn't test setting of preference

• https://stackoverflow.com/questions/29916054/change-user-agent-for-selenium-web-driver

async webriver drop causes endless blocking

Issue

In GenericWebDriver::drop implementation futures::executor::block_on never returns if it run inside tokio runtime.
Specifically if driver is dropped inside a tokio::task.

Not sure if it's caused only by a usage of a different executor.

To reproduce

        use thirtyfour::prelude::*;

        let driver = WebDriver::new("http://localhost:4444", &DesiredCapabilities::firefox()).await?;

        tokio::spawn(async move {
            println!("START");
            driver.close().await.unwrap();
            println!("FINISH");
        }).await;

You will see START and FINISH in output but it will never end the spawned task

I spend a good number of hours to locate the issue 😃

How do I use thirtyfour in a actix-web integration test?

I'm trying to setup some integration tests with actix-web and thirtyfour but I can't get it to work. I've tried both the async and sync version.

Sync:

#[cfg(test)]
mod tests {
    use thirtyfour::sync::prelude::*;

    use crate::routes::routes;
    use actix_web::{test, App};

    #[actix_rt::test]
    async fn test_index_get() -> WebDriverResult<()> {
        let mut app = test::init_service(App::new().configure(routes)).await;
        let req = test::TestRequest::get().uri("/").to_request();
        let resp = test::call_service(&mut app, req).await;
        assert!(resp.status().is_success());

        let caps = DesiredCapabilities::chrome();
        let driver = WebDriver::new("http://localhost:4444", &caps)?;
        Ok(())
    }
}

thread 'tests::e2e::tests::test_index_get' panicked at 'Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.', /home/oskar/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/shutdown.rs:49:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Panic in Arbiter thread.

Async

#[cfg(test)]
mod tests {
    use thirtyfour::prelude::*;

    use crate::routes::routes;
    use actix_web::{test, App};

    #[actix_rt::test]
    async fn test_index_get() -> WebDriverResult<()> {
        let mut app = test::init_service(App::new().wrap(NormalizeSlashes).configure(routes)).await;
        let req = test::TestRequest::get().uri("/").to_request();
        let resp = test::call_service(&mut app, req).await;
        assert!(resp.status().is_success());

        let caps = DesiredCapabilities::chrome();
        let driver = WebDriver::new("http://localhost:4444", &caps).await?;
        Ok(())
    }
}

Error: JsonError(Error("invalid type: null, expected struct ConnectionResp", line: 0, column: 0))
thread 'tests::e2e::tests::test_index_get' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /home/oskar/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/macros.rs:16:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

WebDriverSession type is not public

WebDriverSession is not public, but it's a part of WebDriverCommands interface. This prohibits for from implementing their own WebDrivers.

Why do I need this in the first place? The GenericWebDriver doesn't accept existing client connection but always creates a new one. This with combination of HTTP/1 restrictions on chromedriver prohibits users from scaling for more than 5 parallel sessions.

So there are to solutions to this:

• Make WebDriverSession public where users can implement their own webdrivers
• Allow GenericWebdriver to reuse existing connections

In both cases I would be happy to help

Expose default timeouts to public

Hi @stevepryde what do you thing to make a public constant or exposed method for a default TimeoutConfiguration?

It would be useful to be able to obtain a default one to be able to switch back after a usage of thirtyfour_query and set_implicit_wait_timeout

thirtyfour/src/webdrivercommands.rs

Lines 78 to 83 in 83c91ce

	// Set default timeouts.
	let timeout_config = TimeoutConfiguration::new(
	Some(Duration::new(60, 0)),
	Some(Duration::new(60, 0)),
	Some(Duration::new(30, 0)),
	);

Reqwest feature "rustls-tls"

First, thanks for a great project.

So I have an Actix Web project project that uses rust-tls https://github.com/contor-systems/contor

I'm trying to write the integration tests with thirtyfour. https://github.com/contor-systems/contor/tree/master/tests

We're both using Reqwest accept I use the feature "rust-tls". I add thirtyfour as a dev-dependency.

Cargo doesn't have a way to disable dev dependencies during a build in my CI/CD. So I get an openssl error.

My first reaction was to look for a rust-tls feature on thirtyfour.

I can try some work arounds, but just thought I'd let you know.

geckodriver fails

Hi Guys,

I found several scenerios where the geckdodriver becomes a state, where I can't create a new connection.
These cases I found:

• If I exit the programm with cmd + c while a find_element action is active.
• If you create a driver with new_with_timeout() and you got the timeout while find_element(s) .

In the cases after the application restarts you get these exception:

2021-04-23 18:31:30,333 INFO  [app] Connect to WebDriver
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: SessionNotCreated(WebDriverErrorInfo { status: 500, error: "", value: WebDriverErrorValue { message: "Session is already started", error: Some("session not created"), stacktrace: Some(""), data: None } })', src\main.rs:20:80
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\item_tracker.exe ./` (exit code: 101)

T

openssl-sys is pulled down by reqwest even when using rustls

First of all thanks for your work on this project, it's been a pleasure to use it. The async version is really handy as I need to find elements in batch and doing them concurrently has been a huge win.

I was hoping I could do a standalone musl build out of it, but that didn't work. The build is looking for openssl-sys. I assumed using the following feature flag would disable native-tls.

thirtyfour = { version = "0.23.0", features = [ "tokio-runtime", "reqwest-rustls-tls" ] }

However when I checked the dep tree, it shows the following

│   ├── reqwest v0.11.3
│   │   ├── base64 v0.13.0
│   │   ├── bytes v1.0.1
│   │   ├── encoding_rs v0.8.28 (*)
│   │   ├── futures-core v0.3.14
│   │   ├── futures-util v0.3.14 (*)
│   │   ├── http v0.2.4 (*)
│   │   ├── http-body v0.4.1 (*)
│   │   ├── hyper v0.14.5 (*)
│   │   ├── hyper-rustls v0.22.1
│   │   │   ├── futures-util v0.3.14 (*)
│   │   │   ├── hyper v0.14.5 (*)
│   │   │   ├── log v0.4.14 (*)
│   │   │   ├── rustls v0.19.1 (*)
│   │   │   ├── tokio v1.5.0 (*)
│   │   │   ├── tokio-rustls v0.22.0
│   │   │   │   ├── rustls v0.19.1 (*)
│   │   │   │   ├── tokio v1.5.0 (*)
│   │   │   │   └── webpki v0.21.4 (*)
│   │   │   └── webpki v0.21.4 (*)
│   │   ├── hyper-tls v0.5.0
│   │   │   ├── bytes v1.0.1
│   │   │   ├── hyper v0.14.5 (*)
│   │   │   ├── native-tls v0.2.7
│   │   │   │   ├── log v0.4.14 (*)
│   │   │   │   ├── openssl v0.10.33
│   │   │   │   │   ├── bitflags v1.2.1
│   │   │   │   │   ├── cfg-if v1.0.0
│   │   │   │   │   ├── foreign-types v0.3.2
│   │   │   │   │   │   └── foreign-types-shared v0.1.1
│   │   │   │   │   ├── libc v0.2.93
│   │   │   │   │   ├── once_cell v1.7.2
│   │   │   │   │   └── openssl-sys v0.9.61
│   │   │   │   │       └── libc v0.2.93
│   │   │   │   │       [build-dependencies]
│   │   │   │   │       ├── autocfg v1.0.1
│   │   │   │   │       ├── cc v1.0.67
│   │   │   │   │       └── pkg-config v0.3.19
│   │   │   │   ├── openssl-probe v0.1.2
│   │   │   │   └── openssl-sys v0.9.61 (*)
│   │   │   ├── tokio v1.5.0 (*)
│   │   │   └── tokio-native-tls v0.3.0
│   │   │       ├── native-tls v0.2.7 (*)
│   │   │       └── tokio v1.5.0 (*)

Anyway to turn it off completely please? Thanks

Combobox interaction?

Hi and thanks for this wonderful library.

Could you please comment on whether or not it's possible to interact with a dropdown combobox? And if so, how?

Thanks.

WASM support

I haven't tried compiling this to WASM but I will assume it doesn't 😅
I might sound weird wanting to run a browser automation tool from a browser environment but I have a use case of a plug-in system where plugins are JS or Rust compiled to WASM running in a WebWorker, this plug-ins can run in the browser and the server with the help of Deno. The browser APIs limit me to use fetch or WebSocket based libraries so I was looking for Webdriver rust libraries that would use reqwest as HTTP client which supports compiling to browser WASM, I used fantoccini a bit and was planning to work on a fork that used reqwest but I'm glad to see Thirtyfour exists and the API might even be nicer.

socksProxy requires socksVersion

Issue

Setting such option will cause a error

    cops.set_proxy(thirtyfour::Proxy::Manual {
        ftp_proxy: None,
        http_proxy: None,
        ssl_proxy: None,
        socks_proxy: Some(proxy.clone()),
        socks_password: None,
        socks_username: None,
        no_proxy: None,
    })?;

Unable to create WebDriver session:
    Status: 500
    Additional info:
        InvalidArgumentError: Expected "socksVersion" to be a positive integer, got [object Undefined] undefined
        Error: session not created
        Stacktrace:
            WebDriverError@chrome://marionette/content/error.js:175:5
            InvalidArgumentError@chrome://marionette/content/error.js:304:5
            assert.that/<@chrome://marionette/content/assert.js:479:13
            assert.integer@chrome://marionette/content/assert.js:325:48
            assert.positiveInteger@chrome://marionette/content/assert.js:343:10
            fromJSON@chrome://marionette/content/capabilities.js:333:35
            match_@chrome://marionette/content/capabilities.js:539:21
            fromJSON@chrome://marionette/content/capabilities.js:518:25
            GeckoDriver.prototype.newSession@chrome://marionette/content/driver.js:800:38
            despatch@chrome://marionette/content/server.js:305:40
            execute@chrome://marionette/content/server.js:275:16
            onPacket/<@chrome://marionette/content/server.js:248:20
            onPacket@chrome://marionette/content/server.js:249:9
            _onJSONObjectReady/<@chrome://marionette/content/transport.js:501:20

As I understand currently socksVersion field is missed

links

https://www.w3.org/TR/webdriver1/#proxy

Driver

Geckodriver

[Design] .get_attribute() should return a Option if null

Problem

Currently .get_attribute() returns a "null" if the attribute doesn't exist.

Intended hehavior

I think it should return a None insted of "null"

Reason

If someone had a <div data-some="null">, how would that someone detect the attribute data-some is defined.

Ways to implement

• Change the .get_attribute() function to return a Option<String> (BREAKING CHANGE),
• Add a method called .get_attribute_opt() that returns a Option<String>,

HTTP/2 Support and shared clients

Even though the official Selenium Grid does not support HTTP/2 just yet, there is a distinct possibility that it will eventually do so. Additionally, there may be other grid solutions out there that do support it already (shameless plug 😉).

Especially when running multiple tests in parallel across multiple browser sessions, using a single HTTP/2 connection to the Grid with request multiplexing can help reduce network load and latency. Thus I'd really like to see this feature.

Regarding the implementation, it should be somewhat straightforward. We need a way to share the Reqwest Client structure between multiple WebDrivers and call the http2_prior_knowledge function on the Client to force HTTP/2 (it does not appear to negotiate it on its own).

How to use ElementPredicate!

Hey!

Let me begin by saying this is a wonderful library and I'm having a great time developing with it.

I have just one question, could you please post an example using with_filter from the ElementQuery interface?

I'm having a hard time getting my head around it ...

Cheers!

Unusable error message when anything but 200 OK is returned

When the Selenium Endpoint does not return a valid JSON object but instead throws some kind of error (e.g. 502 Bad Gateway) that error code and the response body is not part of the thrown error. Instead, it contains null:

The WebDriver server returned an unrecognised response: Server returned unknown response: null

Here is the corresponding HTTP response captured with Wireshark:

HTTP/1.1 502 Bad Gateway
Content-Length: 148
Date: Fri, 07 May 2021 07:42:20 GMT
Content-Type: text/plain; charset=utf-8

Unable to forward request: error trying to connect: tcp connect error: Operation timed out (os error 110)

It would be neat if that error message and code would be propagated instead of discarded 🙂