codeception / phpunit-wrapper

PHPUnit bridge for Codeception

PHPUnit Wrapper

Builds:

• 9.0
• 8.0
• 7.1
• 6.5
• 6.0

Codeception heavily relies on PHPUnit for running and managing tests. Not all PHPUnit classes fit the needs of Codeception, that's why they were extended or redefined.

Releases follow major PHPUnit versions.

phpunit-wrapper's Issues

Update phpunit dependency to 7.4

A few months ago, the phpunit dependency was changed to be more strict, but now phpunit 7.4 is out. For projects that have phpunit listed as a dependency without the same strict rule, it causes codeception/phpunit-wrapper to revert back to version 7.1. Not sure if this is a better option.

If anything, would be great if this package could be updated to support phpunit 7.4 so that it doesn't revert back to 7.1 in cases like the one I just described. 7.4 doesn't have any breaking changes.

HTML Report: Successful Tests Can Be Incorrectly Coloured Red

In the HTML report, when using a Cest with a dataprovider, if one of the tests fails then all subsequent tests using that dataprovider are coloured in red even if they pass.

Example

Cest

<?php
namespace { # global namespace

    class FailsCest
    {

       /**
        * @dataprovider pageProvider
        */
        public function withProvider(AcceptanceTester $I, \Codeception\Example $example)
        {
            $I->amOnPage($example['url']);
            $I->seeInTitle($example['title']);
        }

        public function failedWithoutProvider(AcceptanceTester $I)
        {
            $I->amOnPage("http://www.bbc.co.uk");
            $I->see('rumplestiltskin', 'h1');
        }

        public function passedWithoutProvider(AcceptanceTester $I)
        {
            $I->amOnPage("http://www.bbc.co.uk");
            $I->see('BBC Homepage', 'h1');
        }

        public function failedAssert(AcceptanceTester $I)
        {
            $I->assertEquals(true, false);
        }

        public function passedAssert(AcceptanceTester $I)
        {
            $I->assertEquals(true, true);
        }

        /**
         * @return array
         */
        protected function pageProvider()
        {
            return [
                [
                    'url'   => "http://www.bbc.co.uk",
                    'title' => "BBC - Home"
                ],
                [
                    'url'   => "http://codeception.com",
                    'title' => "Zebra"
                ],
                [
                    'url'   => "http://www.php.net",
                    'title' => "PHP: Hypertext Preprocessor"
                ],
                [
                    'url'   => "http://www.google.com",
                    'title' => "Google"
                ],

            ];
        }
    }
}

CLI

$ codecept run acceptance --html
Codeception PHP Testing Framework v2.4.1
Powered by PHPUnit 7.0.2 by Sebastian Bergmann and contributors.

Acceptance Tests (8) ---------------------------------------------------------------------------------------------------------------
✔ FailsCest: With provider | "http://www.bbc.co.uk","BBC - Home" (0.56s)
✖ FailsCest: With provider | "http://codeception.com","Zebra" (1.25s)
✔ FailsCest: With provider | "http://www.php.net","PHP: Hypertext Preprocessor" (0.15s)
✔ FailsCest: With provider | "http://www.google.com","Google" (0.14s)
✖ FailsCest: Failed without provider (0.45s)
✔ FailsCest: Passed without provider (0.55s)
✖ FailsCest: Failed assert (0.00s)
✔ FailsCest: Passed assert (0.00s)
------------------------------------------------------------------------------------------------------------------------------------


Time: 3.24 seconds, Memory: 12.00MB

There were 3 failures:

---------
1) FailsCest: With provider | "http://codeception.com","Zebra"
 Test  tests/acceptance/FailsCest.php:withProvider
 Step  See in title "Zebra"
 Fail  page title contains Zebra
Failed asserting that '\n
        \n
        Codeception\n
        \n
    ' contains "Zebra".

Scenario Steps:

 2. $I->seeInTitle("Zebra") at tests/acceptance/FailsCest.php:13
 1. $I->amOnPage("http://codeception.com") at tests/acceptance/FailsCest.php:12


---------
2) FailsCest: Failed without provider
 Test  tests/acceptance/FailsCest.php:failedWithoutProvider
 Step  See "rumplestiltskin","h1"
 Fail  Failed asserting that any element by 'h1' on page / 
+ <h1 class="hp-main-heading" id="hp-bbc-homepage-content">BBC Homepage</h1>
contains text 'rumplestiltskin'

Scenario Steps:

 2. $I->see("rumplestiltskin","h1") at tests/acceptance/FailsCest.php:19
 1. $I->amOnPage("http://www.bbc.co.uk") at tests/acceptance/FailsCest.php:18


---------
3) FailsCest: Failed assert
 Test  tests/acceptance/FailsCest.php:failedAssert
 Step  Assert equals true,false
 Fail  Failed asserting that false matches expected true.

Scenario Steps:

 1. $I->assertEquals(true,false) at tests/acceptance/FailsCest.php:30


FAILURES!
Tests: 8, Assertions: 8, Failures: 3.
- HTML report generated in file:///Users/Tenz/Repositories/demo/tests/_output/report.html

HTML

Expected Results

"With provider" results for http://www.php.net and http://www.google.com should be green.

JsonContains mismatches nested sequential array

Hi,
I'm not sure if it's a bug or expected behaviour.

This example matches correctly:

<?php
$actualJson = [
    'foo' => [
        'bar' => ['cat', 'dog']
    ]
];

$expectedJson = ['cat', 'dog'];

(new JsonContains($expectedJson))->evaluate(json_encode($actualJson));
// success

While this example fails - it can not match a sequential array when it's inside of another sequential array:

<?php
$actualJson = [
    'foo' => [
        ['cat', 'dog']
    ]
];

$expectedJson = ['cat', 'dog'];

(new JsonContains($expectedJson))->evaluate(json_encode($actualJson));
// failed

There is a workaround to wrap the expected value in an array one more time, but still looks weird that it does not work in the example above.

<?php
$actualJson = [
    'foo' => [
        ['some noise'],
        ['cat', 'dog'],
        ['some noise'],
    ]
];

$expectedJson = [['cat', 'dog']];

(new JsonContains($expectedJson))->evaluate(json_encode($actualJson));
// success

Method matches of JsonContains must return bool

phpunit-wrapper/src/Constraint/JsonContains.php

Line 45 in a499012

$comparator->assertEquals($this->expected, $jsonResponseArray->toArray());

Method returns nothing if assertEquals didn't throw an exception, but boolean is expected. I think this part should be:

try {
    $comparator->assertEquals($this->expected, $jsonResponseArray->toArray());
} catch (ComparisonFailure $failure) {
    throw new \PHPUnit\Framework\ExpectationFailedException(
        "Response JSON does not contain the provided JSON\n",
        $failure
    );
}

return true;

bug?get Errors

Undefined property: Codeception\PHPUnit\Constraint\Crawler::$string

using a private property $string defined in Codeception\PHPUnit\Constraint\Page
and use it in the child class Codeception\PHPUnit\Constraint\Crawler and Codeception\PHPUnit\Constraint\CrawlerNot.

JSON format in report.json not valid with --json option

See #5023

The v7 version of phpunit-wrapper is incompatible with phpunit 7

The reason lies in this commit -> sebastianbergmann/phpunit@7e9e3aa

Please update, and make it compatible with phpunit ~7, or rollback the dependency to phpunit ~6.

Codeception PHP Testing Framework v2.4.0
Powered by PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
PHP Fatal error:  Declaration of Codeception\PHPUnit\ResultPrinter\Report::endTest(PHPUnit\Framework\Test $test, $time) must be compatible with PHPUnit\Util\TestDox\ResultPrinter::endTest(PHPUnit\Framework\Test $test, float $time): void in ../vendor/codeception/phpunit-wrapper/src/ResultPrinter/Report.php on line 9

Symfony DeprecationHandler produces useless info for deprecations

When you have the symfony/phpunit-bridge installed, codeception will register the test listener to enhance the test suite and deprecation output. But because the tests run through codeception the information in the output about where a certain deprecation is triggered is quite useless.

Normally the will show where a deprecation is triggered but it now only shows Application::run as the trigger location. In a "normal" phpunit setup it shows the correct class where the deprecation is triggered.

Phpunit output:

2x: The "%provider_keys%" is deprecated. Use "toolbar_firewall_names" instead
    2x in ConfigurationTest::testConfigGeneratesAsExpected from Kunstmaan\AdminBundle\Tests\DependencyInjection

Codeception output:

2x: The "%provider_keys%" is deprecated. Use "toolbar_firewall_names" instead
    2x in Application::run from Codeception

This is checked in this part of the deprecation handler: https://github.com/symfony/phpunit-bridge/blob/a00e342346a34c87bd11c3d4f2f93cee1585fb2c/DeprecationErrorHandler.php#L116

The deprecation handler will get the full trace and will wall back over it until it finds a "usefull" class that triggered the deprection. So it will skip phpunit classes, but in the codeception setup it will find codeception classes first so it thinks these classes have triggered the deprecation.

Example phpunit trace

* actual class triggering the deprecation
* phpunit runner
* phpunit
* phpunit

Example codeception trace

* actual class triggering the deprecation
* phpunit runner
* phpunit
* phpunit
* Symfony console
* codeception internal classes
* codeception application

Improve compatibility logic for the EventDispatcher

See failed job in Codeception/module-laravel#12 . Caused by:

phpunit-wrapper/src/DispatcherWrapper.php

Lines 20 to 26 in f5d311f

	if (!interface_exists(TraceableEventDispatcherInterface::class)) {
	//Symfony 5
	$dispatcher->dispatch($eventObject, $eventType);
	} else {
	//Symfony 2,3 or 4
	$dispatcher->dispatch($eventType, $eventObject);
	}

Suggested change (based on https://github.com/lexik/LexikJWTAuthenticationBundle/pull/669/files):

- use Symfony\Component\EventDispatcher\Debug\TraceableEventDispatcherInterface;
+ use Symfony\Contracts\EventDispatcher\EventDispatcherInterface as ContractsEventDispatcherInterface;

//...

-   if (!interface_exists(TraceableEventDispatcherInterface::class)) {
+   if ($dispatcher instanceof ContractsEventDispatcherInterface) {
        $dispatcher->dispatch($eventObject, $eventType);
    } else {
        $dispatcher->dispatch($eventType, $eventObject);
    }

Related to: Codeception/Codeception#6065

Support PHPUnit 10

SelfDescribing interface is incompatible with phpunit 7/8

The interface SelfDescribing in the file phpunit7-interfaces.php is incompatible with phpunit 7.5 and 8.2

https://github.com/sebastianbergmann/phpunit/blob/7.5/src/Framework/SelfDescribing.php
https://github.com/sebastianbergmann/phpunit/blob/8.2/src/Framework/SelfDescribing.php

The method toString has not return type

    /**
     * Returns a string representation of the object.
     */
    public function toString(): string;

I think this is related to this issue Codeception/Codeception#5031

Bug with Constraint\Crawler

https://github.com/Codeception/phpunit-wrapper/blob/7.0/src/Constraint/Crawler.php#L17
cause in Page class string property are private - all blowing up after update.
maybe set it to protected
https://github.com/Codeception/phpunit-wrapper/blob/7.0/src/Constraint/Page.php#L9

HTML Report: undefined method PHPUnit\Framework\AssertionFailedError::getComparisonFailure()

When using the fail() method in the Asserts module and generating an HTML report you get a fatal error:

Fatal error: Uncaught Error: Call to undefined method PHPUnit\Framework\AssertionFailedError::getComparisonFailure()

Example

actor: AcceptanceTester
modules:
  enabled:
    - Asserts

<?php
namespace { # global namespace

    class FailsCest
    {
        public function failedAssert(AcceptanceTester $I)
        {
            $I->assertEquals(true, false);
        }

        public function passedAssert(AcceptanceTester $I)
        {
            $I->assertEquals(true, true);
        }

        public function failHard(AcceptanceTester $I)
        {
            $I->fail('Hard fail');
        }
    }
}

$ codecept run acceptance --html
Codeception PHP Testing Framework v2.4.1
Powered by PHPUnit 7.0.2 by Sebastian Bergmann and contributors.

Acceptance Tests (3) ---------------------------------------------------------------------------------------------------------------
✖ FailsCest: Failed assert (0.00s)
✔ FailsCest: Passed assert (0.00s)
✖ FailsCest: Fail hard (0.00s)
Fatal error: Uncaught Error: Call to undefined method PHPUnit\Framework\AssertionFailedError::getComparisonFailure() in /Users/Tenz/Repositories/Codeception/vendor/codeception/phpunit-wrapper/src/ResultPrinter/HTML.php:300
Stack trace:
#0 /Users/Tenz/Repositories/Codeception/vendor/codeception/phpunit-wrapper/src/ResultPrinter/HTML.php(252): Codeception\PHPUnit\ResultPrinter\HTML->cleanMessage(Object(PHPUnit\Framework\AssertionFailedError))
#1 /Users/Tenz/Repositories/Codeception/vendor/phpunit/phpunit/src/Framework/TestResult.php(348): Codeception\PHPUnit\ResultPrinter\HTML->addFailure(Object(Codeception\Test\Cest), Object(PHPUnit\Framework\AssertionFailedError), 6.5803527832031E-5)
#2 /Users/Tenz/Repositories/Codeception/src/Codeception/Test/Feature/ErrorLogger.php(23): PHPUnit\Framework\TestResult->addFailure(Object(Codeception\Test\Cest), Object(PHPUnit\Framework\AssertionFailedError), 6.5803527832031E-5)
#3 /Users/Tenz/Repositories/Codeception/src/Codeception/Test/Test.php(107): Codeception\Test\Test->errorLoggerEnd( in /Users/Tenz/Repositories/Codeception/vendor/codeception/phpunit-wrapper/src/ResultPrinter/HTML.php on line 300



FATAL ERROR. TESTS NOT FINISHED.
Uncaught Error: Call to undefined method PHPUnit\Framework\AssertionFailedError::getComparisonFailure() in /Users/Tenz/Repositories/Codeception/vendor/codeception/phpunit-wrapper/src/ResultPrinter/HTML.php:300
Stack trace:
#0 /Users/Tenz/Repositories/Codeception/vendor/codeception/phpunit-wrapper/src/ResultPrinter/HTML.php(252): Codeception\PHPUnit\ResultPrinter\HTML->cleanMessage(Object(PHPUnit\Framework\AssertionFailedError))
#1 /Users/Tenz/Repositories/Codeception/vendor/phpunit/phpunit/src/Framework/TestResult.php(348): Codeception\PHPUnit\ResultPrinter\HTML->addFailure(Object(Codeception\Test\Cest), Object(PHPUnit\Framework\AssertionFailedError), 6.5803527832031E-5)
#2 /Users/Tenz/Repositories/Codeception/src/Codeception/Test/Feature/ErrorLogger.php(23): PHPUnit\Framework\TestResult->addFailure(Object(Codeception\Test\Cest), Object(PHPUnit\Framework\AssertionFailedError), 6.5803527832031E-5)
#3 /Users/Tenz/Repositories/Codeception/src/Codeception/Test/Test.php(107): Codeception\Test\Test->errorLoggerEnd( 
in /Users/Tenz/Repositories/Codeception/vendor/codeception/phpunit-wrapper/src/ResultPrinter/HTML.php:300

Cause

getComparisonFailure is a method of ExpectationFailedException, not AssertionFailedError

https://github.com/sebastianbergmann/phpunit/blob/489bd4bee689ef05308d6ae6ec7c7b3682b93c6c/src/Framework/ExpectationFailedException.php#L37-L44

and fail() does not throw an ExpectationFailedException.

Invalid JsonType reference

I've been trying to use JsonType, but it appears that since codeception v4 this is no longer possible. There's no documentation reference to what happened to it, only the release diff shows it was removed. I found a class in this repo with the same name, but it features a reference to that removed class from main codeception repo:

use Codeception\Util\JsonType as JsonTypeUtil;

Do you guys know what happened to that class?

Support phpunit 8

The phpunit ">=7.1 <7.6" limitation does not allow to migrate to phpunit 8.

Error in Yii2 before test

Warning: Declaration of Codeception\PHPUnit\ResultPrinter\UI::printDefect(PHPUnit\Framework\TestFailure $defect, $count): void should be compatible with PHPUnit\TextUI\ResultPrinter::printDefect(PHPUnit\Framework\TestFailure $defect, int $count): void

Need to change in UI.php
protected function printDefect(\PHPUnit\Framework\TestFailure $defect, $count)
to
protected function printDefect(\PHPUnit\Framework\TestFailure $defect, int $count)