automattic / newspack-e2e-tests

E2E tests suite for the Newspack project

Newspack e2e

Requirements

node, npm, docker, docker-compose installed.

Running the tests locally

1. Create a scripts/.secrets file, based on secrets/.secrets-sample
2. Add newspack-e2e.com to hosts file (add a line with 127.0.0.1 newspack-e2e.com)
3. Start docker-compose: $ docker-compose up --build
4. In a new terminal session, run setup script in the container: $ npm run wp:setup
5. Start Cypress:

• no UI, run headless - $ npm run test:ci
• with Cypress UI - $ npm run test

Resetting the data

Run $ npm run wp:reset to reset the WP DB and re-run setup script.

For a real hard reset, stop the docker-compose process, remove containers (docker-compose down --volumes), and repeat the initial setup form above.

Running tests against a local plugin version

1. Remove the ./wordpress/wp-content/plugins/newspack-plugin
2. Sync the local content to the Docker container volume: $ rsync -a --exclude-from='.distignore' . /path/to/newspack-e2e-tests/wordpress/wp-content/plugins/newspack-plugin

Pro tip: use chokidar to sync local repository to the Docker machine, by running this in the plugin folder:

$ chokidar "." -c "rsync -a --exclude-from='.distignore' . /path/to/newspack-e2e-tests/wordpress/wp-content/plugins/newspack-plugin"

Testing channels

There are three "testing channels" available, meaning three sources for the Newspack plugins:

1. Stable releases channel – versions currently distributed on Github
2. master branches channel – versions built from master branches
3. alpha branches channel – versions currently distributed on Github as pre-releases

The CI for this project should be configured to run tests periodically – after release days for the stable & alpha channels, and daily for master branches channel.

The stable channel is the default. To use a different channel while developing tests locally, add TEST_CHANNEL variable in scripts/.env.

Visual regression testing

Not ready yet for running on CI. Due to font differences the results fail.

After adding a new visual regression test, run $ npm run test:visual:setup to create the base images.

To check for visual regressions, run $ npm run test:visual:check.

The images are stored in cypress/snapshots/base. After a test run, the image diffs will be stored in cypress/snapshots/diff - in case of tests failing on visual check, inspect those for the regressions.

The visual regression testing is set up to run in Electron browser in order to achieve same results locally and on CI (where Electron is the only available option).

Note that Cypress UI will disappear momentarily when taking a screenshot.

Debugging

• After each test run, a video of the test will be stored in artifacts/video
• Enter the WordPress docker container with docker exec -it wordpress_local /bin/bash - the WP CLI will be installed.
  • all wp commands have to be ran with --allow-root option
• WP is available locally at https://newspack-e2e.com

Slack notifications

If SLACK_CHANNEL_ID and SLACK_AUTH_TOKEN environment variables are set on the CI environment, the selected Slack channel will be updated whenever a test suite fails.

newspack-e2e-tests's Issues

Setup wizard - Jetpack & Site Kit (publicly accessible domain)

Issue #3 outlines the need to test the setup wizard in full – with all plugins. Two of these – Jetpack & Site Kit – require a publicly accessible domain to be set up properly.
This can be done with ngrok, and some of the groundwork for that has already been laid. Proxying via nrok makes the tests much slower, though editing the hosts file might help here.

Setup wizard - with plugins

This setup step turns out to be very brittle in e2e environment - plugins (and themes) are frequently not being installed.

E.g. one error that surfaced was this (during jetpack plugin installation):

[17-Mar-2020 16:44:28 UTC] PHP Warning:  Invalid argument supplied for foreach() in /var/www/html/wp-content/plugins/newspack-plugin-ZbCMw1/includes/wizards/class-setup-wizard.php on line 221
[17-Mar-2020 16:46:28 UTC] PHP Warning:  dir(/var/www/html/wp-content/upgrade/jetpack.8.3-JMCGbg/jetpack/_inc/build/photon/): failed to open dir: No such file or directory in /var/www/html/wp-admin/includes/class-wp-filesystem-direct.php on line 571
[17-Mar-2020 16:46:29 UTC] PHP Warning:  dir(/var/www/html/wp-content/upgrade/jetpack.8.3-JMCGbg/jetpack/modules/custom-css/custom-css/css/rtl/): failed to open dir: No such file or directory in /var/www/html/wp-admin/includes/class-wp-filesystem-direct.php on line 571

Another time, the AMP plugin did seem to install, but possibly not completely, and threw this error (on the front-end):

Warning: include(/var/www/html/wp-content/plugins/amp/vendor/composer/../sabberworm/php-css-parser/lib/Sabberworm/CSS/Parsing/ParserState.php): failed to open stream: No such file or directory in /var/www/html/wp-content/plugins/amp/vendor/composer/ClassLoader.php on line 444 Warning: include(): Failed opening '/var/www/html/wp-content/plugins/amp/vendor/composer/../sabberworm/php-css-parser/lib/Sabberworm/CSS/Parsing/ParserState.php' for inclusion (include_path='.:/usr/local/lib/php') in /var/www/html/wp-content/plugins/amp/vendor/composer/ClassLoader.php on line 444 Fatal error: Uncaught Error: Class 'Sabberworm\CSS\Parsing\ParserState' not found in /var/www/html/wp-content/plugins/amp/vendor/sabberworm/php-css-parser/lib/Sabberworm/CSS/Parser.php:26 Stack trace: #0 /var/www/html/wp-content/plugins/amp/includes/sanitizers/class-amp-style-sanitizer.php(1599): Sabberworm\CSS\Parser->__construct('img.wp-smiley,\n...', Object(Sabberworm\CSS\Settings)) #1 /var/www/html/wp-content/plugins/amp/includes/sanitizers/class-amp-style-sanitizer.php(1685): AMP_Style_Sanitizer->parse_stylesheet('img.wp-smiley,\n...', Array) #2 /var/www/html/wp-content/plugins/amp/includes/sanitizers/class-amp-style-sanitizer.php(1435): AMP_Style_Sanitizer->prepare_stylesheet('img.wp-smiley,\n...', Array) #3 /var/www/html/wp-content/plugins/amp/includes/sanitizers/class-amp-style-sanitizer.php(1193): AMP_Style_Sanitizer->process_stylesheet('img.wp-smiley,\n...', Array) #4 /var/www/html/wp-content/plugins/amp/includes/sanitizers/class-amp-style-sanitizer.php(853): AMP_Style_Sanitizer->process_style_element(Object(DOM in /var/www/html/wp-content/plugins/amp/vendor/sabberworm/php-css-parser/lib/Sabberworm/CSS/Parser.php on line 26

In one case, the theme did not fully download (or rather extract) - header.php was missing in the theme folder.

Visual regression on CI

Visual regression testing should be ran in CI.
Right now, this part is skipped, because of a font mismatch problem: CI run a Ubuntu system, which has a slightly different default sans-serif font than macOS. For this reason, all visual regression tests will fail because of that.

Possible solution: generate the base images on CI and commit from there (or download as artifacts and commit locally). This makes for a pretty cumbersome workflow, but maybe it's better in the long run.