10up / wpacceptance

ARCHIVED: A team scalable solution for reliable WordPress acceptance testing.

WP Acceptance

WP Acceptance is a toolkit that empowers developers and CI pipelines to test codebases using version controlled acceptance tests and sharable environments.

Caution

As of 12 April 2024, this project is archived and no longer being actively maintained.

Requirements

• PHP 7.2+
• mysqli PHP extension
• Docker (WP Local Docker is highly recommended as the local development environment but not required.)
• Node >= 8 (WP Acceptance uses Puppeteer behind the scenes.)

Note: WP Acceptance should be run on your HOST machine and not within Docker.

How It Works

There are many acceptance tests frameworks out there. They all have one major flaw - everyone executing the acceptance tests must be running the exact same code on the exact same database and environment to guarantee the same results. Ensuring a team of developers (and a CI pipeline) are all using the same database in the same environment has been nearly impossible until now. WP Acceptance is unique in that it allows you to run your acceptance tests against a codebase in defined and shareable environments. Read more in our announcement blog post.

Install

Installation instructions are on the docs site.

Usage

Learn how to use WP Acceptance on the docs site.

Support Level

Archived: This project is no longer maintained by 10up. We are no longer responding to Issues or Pull Requests unless they relate to security concerns. We encourage interested developers to fork this project and make it their own!

Like what you see?

wpacceptance's Issues

TestCase::last_modifying_query can be empty during tearDown()

After having it working fine for a while, all of a sudden it started displaying the following error:

Trying to access array offset on value of type null
vendor/10up/wpacceptance/src/classes/PHPUnit/TestCase.php:67

Steps to Reproduce

I'm not 100% sure about how to reproduce it but it happened to me with a very simple test (I was just activating a plugin). I got it happening locally and on a GitHub Action.

Environment information

• OS: Linux Mint 19.1 (local) and ubuntu-latest (GitHub Action)

Suggested fix
There is a big chance I'm ignoring something here but it seems we should change this if (WPAcceptance\PHPUnit\TestCase::tearDown()) from

if ( ! empty( $new_last_modifying_query ) && $new_last_modifying_query['event_time'] !== $this->last_modifying_query['event_time'] ) {

to

if ( ! empty( $this->last_modifying_query ) && ! empty( $new_last_modifying_query ) && $new_last_modifying_query['event_time'] !== $this->last_modifying_query['event_time'] ) {

Although $this->last_modifying_query is initially set as an empty array, it is set as
$this->last_modifying_query = $this->getLastModifyingQuery();
during setUp(). If any query run, this will be null, triggering the error during tearDown().

Please let me know if this all makes sense and I can open a PR if it helps. Thanks in advance!

"WP Setup Instruction File" support

WP Snapshots is a barrier to entry for many people since it relies on AWS. For many use cases, people don't need a custom snapshot, rather a simple environment with WP version X and a few plugins. WP Acceptance should support a snapshot OR a set of instructions for spinning up an environment e.g.:

INSTALL wordpress 5.1
INSTALL distributor:1.2

I'm thinking of this kind of like a Dockerfile.

Calling wpacceptance init on windows fails to write wpacceptance.json

Describe the bug

Calling wpacceptance init on windows fails to write file due to bug in path resolution for wpacceptance.json. I'll manually make my wpacceptance.json for now but figured I'd post this here as I would really like to contribute to this project however I can.

Steps to Reproduce

$ ./vendor/bin/wpacceptance init

Project Slug (letters, numbers, _, and - only): noknok-woo-marketing-platform
Tests location (defaults to ./tests/*.php): 
Do you want to require a fresh database for each test? This will make tests slower but is needed if you intend on modifying the database during tests. (yes or no) no
Do you have an existing snapshot ID you would like to test against? Default is none: 

Warning: file_put_contents(C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform/C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform/wpacceptance.json): failed to open stream: No such file or directory in C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\10up\wpacceptance\src\classes\Config.php on line 105

Call Stack:
    0.4048     415152   1. {main}() C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\10up\wpacceptance\bin\wpacceptance:0
    0.4436    1798712   2. require_once('C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\10up\wpacceptance\src\bootstrap.php') C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\10up\wpacceptance\bin\wpacceptance:13        
    1.0891    7719184   3. Symfony\Component\Console\Application->run() C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\10up\wpacceptance\src\bootstrap.php:36
    1.1271    8063456   4. Symfony\Component\Console\Application->doRun() C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\symfony\console\Application.php:149
    1.1293    8066832   5. Symfony\Component\Console\Application->doRunCommand() C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\symfony\console\Application.php:273
    1.1293    8066832   6. WPAcceptance\Command\Init->run() C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\symfony\console\Application.php:1009
    1.1307    8068664   7. WPAcceptance\Command\Init->execute() C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\symfony\console\Command\Command.php:255
   15.4983    8141352   8. WPAcceptance\Config->write() C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\10up\wpacceptance\src\classes\Command\Init.php:71
   15.4985    8143120   9. file_put_contents() C:\Users\Cody Rees\Documents\_noknok-studios\docker-projects\docker.woocommercesandbox.local\sites\localhost\html\wp-content\plugins\noknok-woo-marketing-platform\vendor\10up\wpacceptance\src\classes\Config.php:105

Expected behavior
wpacceptance.json would be saved to the filesystem but instead the file_put_contents throws an error

Environment information

• OS: Microsoft Windows 10 (Error exists in both Git Bash and standard Command Prompt)

Support later versions of Jane PHP package to allow latest major version of symfony/options-resolver

Is your enhancement related to a problem? Please describe.

jane-php/open-api-runtime was pinned to a specific 4.2.0 to avoid a bug in a later version. This version uses symfony/options-resolver ^3.1|^4.0, but that package is now on v5.

Describe the solution you'd like

jane-php/open-api-runtime 5.2.2 supports using symfony/options-resolver ^5.0, so can wpacceptance update the version of jane-php/open-api-runtime that it supports to include this? (^5.2.2 or 4.2.0 | ^5.2.2 or similar).

The plugin I want to use wpacceptance in already has symfony/options-resolver 5 installed, and while I will remove the vendor directory and do a full re-install and end up with options-resolver 4, I'm not seeing a good reason to have wpacceptance be stuck on an old version of the jane-php dependency anyway :-)

Error: Call to a member function getPage() on null

I get this error when I don't ge an actor ($this->openBrowserPage()).

Looks like we need to check if $this->last_action is not null here: https://github.com/10up/wpacceptance/blob/master/src/classes/PHPUnit/TestCase.php#L81

Example
 ✘ Example
   │
   │ Error: Call to a member function getPage() on null
   │ 
   │ /home/nicholas_io/wp-local-docker-sites/XXX/wordpress/wp-content/vendor/10up/wpacceptance/src/classes/PHPUnit/TestCase.php:81
   │ /home/nicholas_io/wp-local-docker-sites/XXX/wordpress/wp-content/vendor/10up/wpacceptance/src/classes/Command/Run.php:300
   │ /home/nicholas_io/wp-local-docker-sites/xxxx/wordpress/wp-content/vendor/symfony/console/Command/Command.php:255
   │ /home/nicholas_io/wp-local-docker-sites/xxxx/wordpress/wp-content/vendor/symfony/console/Application.php:953
   │ /home/nicholas_io/wp-local-docker-sites/xxxxx/wordpress/wp-content/vendor/symfony/console/Application.php:248
   │ /home/nicholas_io/wp-local-docker-sites/xxxxx/wordpress/wp-content/vendor/symfony/console/Application.php:148
   │ /home/nicholas_io/wp-local-docker-sites/xxxxx/wordpress/wp-content/vendor/10up/wpacceptance/src/bootstrap.php:34
   │ 

Enable support for multiple snapshots

Summary

WP Acceptance runs tests against a snapshot which is configured with the environment we want to test again. For many plugins, we'll actually want to test against multiple environments. For example, on the Distributor plugin where we are using WP Acceptance, we would like to test against WordPress 4.9 and 5.0, with or without the Gutenberg plugin enabled, with or without the Classic Editor plugin enabled. By testing against every environment, we can ensure changes made for one configuration don't break other configurations.

Desired Behavior

• wpacceptance.json should accept an array of snapshot ids (ideally a keyed array named for easy reference, eg { wp49: [SNAPSHOT_ID], WP50: [SNAPSHOT_ID] }
• wpacceptance run command should run all tests against all snapshots
• wpacceptance run command should include an option to run against a single snapshot (by name or id)
• the wpacceptance run --save flag should be snapshot specific, requiring passing a snapshot id or name when the config file contains more than one snapshot
• possible follow up feature - enable support in test files to exclude certain tests from snapshots

Current Behavior

WP Acceptance only accepts a single snapshot id, making it difficult to to test against multiple configurations.

readme updates

• add support level section & badge
• add license badge
• add clarifying links where helpful (e.g., mysqli)

Support testing different versions of PHP.

Title says it all. This is an important feature to add.

Conflict with S3 Drop-in Plugin

If using WPAssure with the S3 plugin it looks like there's an edge-case issue where it redefines the S3 classes and causes a fatal error in some hosting environments. For example, on our 10up Managed Hosting the S3 plugin is installed as a drop-in plugin, which is also installed via composer as a dependency for WP Assure. The two conflict since neither is checking for the existence of the classes/functions before using and error out.

Improve Documentation of Project/System/Platform Requirements

Is your enhancement related to a problem? Please describe.

Yes, the documentation for installing WP Acceptance does not clearly define the project, platform and/or system requirements. This causes issue when trying to use Composer without the specific project requirements of WP Acceptance.

For example, while working on improving Composer support for the Restricted Site Access plugin, I ran into an issue when running composer install which stated that my system did not contain the necessary requirements:

➜  restricted-site-access git:(feature/composer-support) ✗ composer install
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - league/uri 5.3.0 requires ext-intl * -> the requested PHP extension intl is missing from your system.
    - league/uri 4.2.3 requires ext-intl * -> the requested PHP extension intl is missing from your system.
    - league/uri 4.2.2 requires ext-intl * -> the requested PHP extension intl is missing from your system.
    - league/uri 4.2.1 requires ext-intl * -> the requested PHP extension intl is missing from your system.
    - league/uri 4.2.0 requires ext-intl * -> the requested PHP extension intl is missing from your system.
    - jane-php/open-api-runtime v4.2.0 requires jane-php/json-schema-runtime ^4.0 -> satisfiable by jane-php/json-schema-runtime[4.x-dev, v4.0.0, v4.0.1, v4.0.2, v4.0.4, v4.1.0, v4.2.0, v4.3.0, v4.4.0, v4.5.0, v4.5.1, v4.5.2, v4.5.3, v4.5.4].
    - 10up/wpacceptance dev-master requires jane-php/open-api-runtime 4.2.0 -> satisfiable by jane-php/open-api-runtime[v4.2.0].
    - jane-php/json-schema-runtime 4.x-dev requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.0.0 requires league/uri ^4.2 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3].
    - jane-php/json-schema-runtime v4.0.1 requires league/uri ^4.2 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3].
    - jane-php/json-schema-runtime v4.0.2 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.0.4 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.1.0 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.2.0 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.3.0 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.4.0 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.5.0 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.5.1 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.5.2 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.5.3 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - jane-php/json-schema-runtime v4.5.4 requires league/uri ^4.2|^5.3 -> satisfiable by league/uri[4.2.0, 4.2.1, 4.2.2, 4.2.3, 5.3.0].
    - Installation request for 10up/wpacceptance dev-master -> satisfiable by 10up/wpacceptance[dev-master].

This is expected, as these are required to run WP Acceptance. I personally use Lando for local development, so I went into the shell to run the install, but still received errors for missing system requirements. This was unexpected, as my local Lando configuration met the requirements documented on wpacceptance.readthedocs.io:

PHP 7.2+
mysqli PHP extension
Docker
Node >= 8 (WP Acceptance uses Puppeteer behind the scenes)

As such, I installed wp-local-docker-v2 in order to test installing WP Acceptance on a new WordPress install, in a blank plugin. In this scenario, I started a new site with 10updocker and configured a Single WordPress installation, with default content deleted. From there I created a blank test plugin, configured Composer with minimum-stability of dev and ran the installer command for WP Acceptance (within shell):

composer require 10up/wpacceptance:dev-master --dev

While the other requirements above were met (expected), to my surprise there was a different install error:

[www-data@4fae65ff5e05 html]$ composer require 10up/wpacceptance:dev-master --dev
./composer.json has been created
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for 10up/wpacceptance dev-master -> satisfiable by 10up/wpacceptance[dev-master].
    - 10up/wpacceptance dev-master requires 10up/wpsnapshots dev-master -> satisfiable by 10up/wpsnapshots[dev-master] but these conflict with your requirements or minimum-stability.

Installation failed, deleting ./composer.json.

Here is the composer.json for that blank repo:

{
    "name": "kopepasah/wp-test-plugin",
    "authors": [
        {
            "name": "Justin Kopepasah",
            "email": "[email protected]"
        }
    ],
    "require": {},
    "minimum-stability": "dev"
}

Describe the solution you'd like

As a developer that could find WP Acceptance useful, it would be beneficial to use this as a Composer package that clearly stated all requirements in order to use the package. As it stands now, there seem to be some required "behind the scenes" configurations that are not yet documented, including the requirement to run Composer commands within the Docker containers.

My expectation is that when the environment (local, stage, prod) is correctly configured (according to the documentation) running composer require 10up/wpacceptance:dev-master --dev would not throw any errors.

In those cases where system/platform requirements are not met, better notifications to the developer would be useful in debugging and understanding the requirements of this package.