codebndr / seleniumtests

Selenium tests for the codebender website

Codebender Selenium Tests

This repo contains Selenium tests for the codebender website. The tests are written in Python 2, and utilize pytest as a testing framework and Selenium for browser automation.

Running Tests

Tests are run by invoking tox. Run tox --help to see all the Codebender-specific arguments that can be passed to py.test.

In addition to these arguments, there are certain environment variables that should be set when running tests:

• CODEBENDER_SELENIUM_HUB_URL: the URL of the Selenium Hub. If you are using SauceLabs, the URL has the following format: http://{USERNAME}:{ACCESS_KEY}@ondemand.saucelabs.com:80/wd/hub. You can also use a docker-selenium hub. In that case, it is necessary to link the docker-selenium instance to the Docker instance from which tests are running.
• CODEBENDER_TEST_USER: username that the webdriver will use to log into the site in order to perform tests.
• CODEBENDER_TEST_PASS: password for CODEBENDER_TEST_USER.

Rather than invoking tox directly, the easiest way to run tests is with Docker. If you are not familiar with Docker, please consult the documentation for an introduction.

First, build the image with $ docker build . -t codebender/selenium.

Then invoke tox via docker run. Here is a sample command to run all tests, where the Codebender server is running at http://192.168.1.2:8080:

$ docker run -e CODEBENDER_SELENIUM_HUB_URL=http://johndoe:[email protected]:80/wd/hub \
             -e CODEBENDER_TEST_USER=tester \
             -e CODEBENDER_TEST_PASS=1234 \
             -it codebender/selenium \
             tox -- --url http://192.168.1.2:8080 --source bachelor

Running Tests Manually

The recommended way of running tests is with Docker. If you would like to manually provision your machine to be able to run tests, you can use the Dockerfile as a step-by-step guide for provisioning. Then invoke tox to run tests.

Specifying a URL for Tests

Tests can either be run for the bachelor version of the site, running locally, or for the live site. The version of the site that is running is inferred from the --url parameter. You can run $ tox --url http://localhost to run the tests for a locally running bachelor site (this is the default url), or $ tox --url http://codebender.cc to run the tests for the live site.

Certain tests are specially written for one site or the other. This is implemented with a custom pytest marker. Tests that require a certain --url are decorated with @pytest.mark.requires_url(<url>).

Changing Test Configuration

Various global configuration parameters are specified in codebender_testing/config.py. Such parameters include URLs and site endpoints which are subject to change. This is also where the webdrivers (Firefox and Chrome) are specified.

Compilation Logs

Certain tests exist to iterate through groups of sketches and compile them one-by-one. Since these tests take a long time, they are not run in full by default. You can run them by specifying the --full option; for example: $ tox tests/cb_compile_tester --full.

The following test cases are compile tests that generate such logs:

• tests/libraries/test_libraries.py::TestLibraryExamples
• tests/compile_tester/test_compile_tester_projects.py::TestCompileTester

The generated logs are placed in the logs directory. They give detailed output in JSON format containing the codebender site URL that was used to run the tests, along with the URLs of the individual sketches that were compiled, and whether they succeeded or failed to compile.

Framework Overview

The following outlines the structure of the repository as well as important framework components.

Directory Structure

tests/

The tests/ directory contains all of the actual unit tests for the codebender site. That is, all of the tests discovered by py.test should come from this directory.

tests/conftest.py contains the global configuration for pytest, including specifying the webdriver fixtures as well as the available command line arguments.

codebender_testing/

This is where all major components of the testing framework live. All of the unit tests rely on the files in this directory.

codebender_testing/config.py specifies global configuration parameters for testing (see "Changing Test Configuration" above).

codebender_testing/utils.py defines codebender-specific utilities used to test the site. These mostly consist of abstractions to the Selenium framework. The most important class is SeleniumTestCase, which all of the unit test cases inherit from. This grants them access (via self) to a number of methods and attributes that are useful for performing codebender-specific actions.

codebender_testing/capabilities_{firefox, chrome}.yaml defines a list of capabilities to be passed as arguments when instantiating remote webdrivers. In particular, it specifies the web browsers that we would like to use. Consult this file for more information.

batch/

The batch/ directory contains any executable scripts not directly used to perform tests. For example, it contains a script fetch_projects.py which can be used to download all of the public projects of a particular codebender user.

extensions/

The extensions/ directory contains the codebender browser extensions to be used by the Selenium webdrivers.

test_data/

The test_data/ directory contains any data used for testing. For example, it contains example projects that we should successfully be able to upload and compile.

logs/

The logs/ directory contains the results of running certain tests, e.g. whether certain sets of sketches have compiled successfully (see "Compilation Logs").

seleniumtests's Issues

Independent tests for cb_compile_tester's projects

Assigned to @bduffany & @hciwork

For this one, you'll have to compile the projects of a user called cb_compile_tester (https://codebender.cc/user/cb_compile_tester). It's a fake user whose projects are weird cases we've come across during the past months. We'd like to be able to test all of cb_compile_tester's projects at once and get a structured output containing which of them compiled successfully and which didn't.

Update 1
The /libraries testing task was removed and placed into a separate issue.

Update 2
Following the same pattern as on #6, we could add the ability to only run the tests for cb_compile_tester's projects, like python run_tests.py --compile-tester and we could also choose which codebender instance we'd like to test, so the above command would become python run_tests.py --compile-tester --url http://localhost in order to test cb_compile_tester's sketches your bachelor instance.
Remeber that on bachelor you will have to use verify() JavaScript function instead of compilerflasher.verify().

Update 3 (assuming we have accomplished testing upload functionality)
Having added the ability to choose which codebender instance we want to test, we can stick to the bachelor (remember it's our open source version). You might have noticed that there is no cb_compile_tester on the bachelor version you have installed on your VMs 😧
What we thought you could do, is add all cb_compile_tester's to the tests repo in zip files, and then login with tester user, upload each one of the zip files, compile the project and report the result.
After the compilation of each sketch, you could have the tests delete it. This way we can test the whole "upload a project, use it and then delete it" process.

Independent tests for codebender libraries

Initial /libraries task
For this one, we need you to iterate over all the examples found in https://codebender.cc/libraries page and check whether the code compiles or not. For example, under the Ethernet library, you have the UDPSendReceiveString example. So you open example (https://codebender.cc/example/Ethernet/UDPSendReceiveString), select Arduino UNO board (for now at least) and then compile the code. You might notice that the only option you have on a library example page, is to Run on Arduino, which means compile the code and then upload the code on the board. So if we just want the compilation part, we can call the corresponding Javascript. If you try calling compilerflasher.verify() then you will get the compilation result only without any attempt to load the code on the board. And this will output either a success message or an error message.
So we kind of want to automate this for all the examples found on the /libraries page of the live codebender website.
After the test execution, we should be able to get the output with the successful and unsuccessful examples in JSON format.

Update 1
It would also be good to have the ability to run the tests only for the libraries,
for example (supposing that the command python run_tests.py runs your tests)
something like python run_tests.py --libraries could run the tests for the /libraries page.

Update 2
Expanding one library's examples and then selecting an example and compiling it is pretty much the same in the bachelor as well as the live website. So we thought we could also use your tests for the bachelor too (reminder: bachelor is open source and your tests are supposed to test an open source project). In this case we could have the tests get the url of the version of the site that we want to test, for example (assuming same things as Update 1) python run_tests.py --libraries --url http://localhost in order to test your bachelor's library examples.
Note that on the bachelor, you will have to use verify() JavaScript function instead of compilerflasher.verify() (maybe you could first check if compilerflasher is defined, and if not, then just use verify())

First part of selenium tests for Bachelor

Assigned to @bduffany & @hciwork

Click checkboxes after each test is done

LandingPage/Login

• Login div appears
• Login with invalid username
• Login with invalid password
• Login takes you to user's home (the one with the project-list and menu)

User's homepage

• Upload project (single file)
• Create new project with no project name
• Create new project with valid project name
• Create new project with invalid name (e.g. backslashes in project name)
• Upload project (need to test zip functionality, is that possible?)

Sketch page

• Boards dropdown (does it exist?)
• Board selection from dropdown (must be on the placeholder after selection)
• Ports dropdown (does it exist?)
• Run On Arduino with empty ports dropdown (none selected)
• Speeds dropdown (does it exist?)
• Open Serial Monitor with empty ports dropdown
• Clone project (does it take you to a "projectName copy" page?)

Option to run minimal set of tests

We should have an option like --minimal (or its complement, --full) to run the absolute basic tests. Right now when we run tox it goes through and uploads all of cb_compile_tester's projects, and also compiles every single one of the libraries. Instead we should have the option to only compile one from each category, just to verify basic functionality, and to make sure all tests pass.

Second part of selenium tests for Bachelor

Assigned to @bduffany & @hciwork

Some more test cases for the Bachelor.

Click checkboxes after each test is done

Sketch Page

• Add new file to project with new filename
• Add new file to project via upload new file
• Add new file to project via upload new file with error (unsupported file)
• Delete a file - check the modal that pops-up
• Delete a file - refresh and verify that it's deleted
• How about the progress bar? I guess this only applies to the verify action

Boards (supported boards) page
Note: You can use one of the board definitions found here https://github.com/arduino/Arduino/blob/ide-1.0.x/hardware/arduino/boards.txt

• File successful upload
• File upload with error
• Delete board button
  Ignore the Edit board button, it's not working on bachelor codebender version