18f / circle-wdio-federalist

Circle CI + WebDriverIO + Sauce Labs + Federalist = Nirvana?

Automated cross-browser testing with Circle CI, WebDriverIO, Sauce Labs, and Federalist

This is a standalone, working example that integrates Circle CI, WebDriverIO, Sauce Labs, and Federalist into a (mostly) sane cross- browser automated testing setup.

How It Works

Here are the moving parts:

• Federalist is a web publishing platform that listens for commit hooks from GitHub and can either directly publish static files or build Jekyll sites and publish the resulting sites to S3. It also has a nifty branch preview feature that publishes unique, predictable URLs for each git branch, and can be configured to make these URLs public.

• Circle CI, like many other popular continuous integration services, also listens for commit hooks from GitHub and can be configured via a circle.yml file to automatically run tests of any kind on each git branch.

• WebDriverIO is one of many browser testing frameworks that uses Selenium under the hood to script interactions with real and virtual web browsers. This allows us to write tests that can load web pages, click buttons and links, or detect whether certain elements are visible or hidden.

• Sauce Labs is a web service that lets you run Selenium tests on hundreds of different web browsers on different platforms (Windows, OS X, Linux, iOS, Android), collect information about which tests succeed or fail, and even watch the scripted interactions after the fact.

The core piece of infrastructure here is a single shell script, test/ci.sh, which infers the Federalist URL from Circle CI's environment variables and fetches a commit.txt file until its contents match the git commit ID (SHA1) in Circle's $CIRCLE_SHA. You can see this at work by viewing this repo's published commit.txt and the passing build on Circle CI.

Setup

To start, you can either fork this repo or manually copy the relevant files into your own. You'll need:

• Jekyll, which Federalist uses to generate your static site.
• circle.yml, which tells Circle CI to use Node.js 4.x and run our custom test script.
• commit.txt, which will contain the latest git commit ID when Federalist builds your site.
• package.json to manage Node.js dependencies
• test/ci.sh, which is our Circle CI-specific test runner that ensures tests are being run against the same git commit on Circle and Federalist.
• test/wdio.*.js configuration files for WebDriverIO.
• one or more browser tests in the test/spec/ directory.

Dependencies

1. Run npm install to install the Node.js dependencies.

2. Install Selenium Server. There are lots of different ways to do this, but the best that I've found is:

   # install webdriver-manager globally
   npm install -g webdriver-manager
   # install the Chrome driver
   webdriver-manager --chrome

Federalist Setup

Next, add your repository to Federalist, and configure its settings to make preview URLs public.

Test Setup

1. Write some WebDriverIO tests in the test/spec/ directory.
2. Configure your local environment variables:

• Copy the .env template and rename it to .env.
• Modify your .env file to include your GitHub username, repo identifier, and Sauce Labs credentials.
• You can then either source .env manually or use autoenv to have the .env file sourced automatically whenever you enter your project directory.

1. Run your tests locally:

• Run Selenium Server in a new shell:

  webdriver-manager start

• Start up your Jekyll server, also in a new shell, with:

  jekyll serve

• Run the WebDriverIO tests locally with Chrome (the default):

  npm test

Circle CI Setup

Add your repository to Circle CI, and configure it with your Sauce Labs username and access key in the SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables.

Publishing

To run the tests, just push a commit to GitHub, and watch your build status on Circle CI.

Happy browser testing!

circle-wdio-federalist's Issues

Commit SHA removed from GovCloud Federalist builds

Recently the new GovCloud Federalist has stopped adding commit details into the site.commit.commit Jekyll variable which means Circle doesn't know when Federalist is done building.

Commit details we're removed with this commit: cloud-gov/pages-build-container/pull/36

Issue is being tracked over at cloud-gov/pages-core/issues/884

Jekyll Git_metadata plugin doesn't use long_sha for merge commits

After a merge commit, I noticed that the Jekyll Git_metadata plugin's long_sha was still pointed at the previous "regular" commit. This appears to be by design of the plugin. This has the effect that the sha check won't match as CIRCLE_SHA1 uses the merge commit sha.

I wrote a simpler version of this gem as a local jekyll plugin if merge commits are of interest to this project.

How to handle fork PRs?

Federalist only builds branches on the main repo, so forked PRs won't work with this approach because there won't be a preview URL for them until somebody manually merges the fork's commits into a new branch on the main repo.

The nice thing to do in this case would be to skip the test suite altogether and output instructions to copy the commits into a new branch.