excessive failed tests with make test? about suhosin

[ck-on]

Yup there is definitely something up with the way skipifcli.inc and other includes are being processed during make test

Either the include is not happening or it is not being evaluated for some reason.

The answer is the problem is the path on the include.

The includes are being looked for in the same directory as run-tests.php

The includes then move up a directory because include "../skipifcli.inc";

Which is then outside the path of the suhosin build entirely.

I can make a symbolic link but that is not a proper fix of course.

Then I get these results with php 5.5 and suhosin trunk

Number of tests :  173               143
Tests skipped   :   30 ( 17.3%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :   95 ( 54.9%) ( 66.4%)
Expected fail   :    0 (  0.0%) (  0.0%)
Tests passed    :   48 ( 27.7%) ( 33.6%)

[stefanesser]

On a current system not a single test case should fail. I suspect something is going wrong here with how you PHP is compiled/configured.

Do I understand you correctly that you compile --without-cgi ?

If you can provide us with some more info we will have to look into this.
However all Suhosin developers will not have time to fix any bugs in the coming week.

[ck-on]

If you mean PHP, yes I build it with --disable-cgi and --enable-fpm for php-fpm use.

What is strange is I can see make test still run the PHPSESSID tests which would be impossible to test in CLI mode and should be excluded by skipifcli.inc and then it includes them in the FAIL list.

Are you saying the tests require php to have cgi mode available to run the session and get tests and that is why all of them fail? That makes sense and would explain a great deal.

Let me try building php differently and I will report back here.

Okay I removed disable-cgi from the php build and I saw it create -o sapi/cgi/php-cgi and did a make install.

redid phpize and configure/make suhosin and I get the same failures

I will explore further how some of the failed tests run and see if I can trace why they are running.

[stefanesser]

We have to check if all the tests are including the right .inc.

Anyway it seems strange that when you have no CGI available the tests are not correctly skipped.

Can you tell me on what OS you are trying this? Linux, BSD, Darwin?

[ck-on]

I am using it on CentOS 7.0, just tried it both with php 5.5 and then php 5.6, same behavior.

I can also test it on CentOS 6.5 which I am about to go do.

I also noticed it copies tmp-php.ini from the existing php.ini, so just in case I tried copying the stock php.ini from the php install but that did not change anything.

As I noted before, I removed the disable-cgi and re-built it but no change.

Will report back with centos 6.5 status.

It occurs to me that skipping cookie, session, get and post tests is not something I really want to do, so if I can force cgi mode temporarily instead of using the cli, that would be a much better solution. Just not sure offhand how to do that. Looking at run-tests.php it seems I might be able to set TEST_PHP_EXECUTABLE to override behavior.

[ck-on]

since run-tests.php is from the php building package, it looks for the cgi in wrong location $php_cgi = $cwd . '/sapi/cgi/php-cgi'; by default.

I can override that with TEST_PHP_CGI_EXECUTABLE=/usr/local/bin/php-cgi and then run-tests.php shows two summaries at the start, one for cli, the other for cgi, so it finds them both.

However it is not switching to the cgi and sticking with cli mode.

I am not even sure how sessions could run in cli mode

  FAIL PHPSESSID session id not too long [tests/session/PHPSESSID_max_id_length_ok.phpt] 

By the way, note your includes are inconsistent.
It looks like the newer mysqli tests have the proper include('skipif.inc'); but then older tests have the two dots prefix include "../skipifcli.inc";

[ck-on]

Making a little progress. Discovered the session failures were due to a bad path in php.ini

With php 5.6 I now get this result

Number of tests :  173               141
Tests skipped   :   32 ( 18.5%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :   61 ( 35.3%) ( 43.3%)
Expected fail   :    0 (  0.0%) (  0.0%)
Tests passed    :   80 ( 46.2%) ( 56.7%)

I will keep trying to hunt down causes. The bulk of the failures seem to be related to suhosin input filter so will explore if I have a bad configuration for that somewhere.

Seems like you have a busy July with ios stuff so will be patient for updates maybe in August.

Update: found the bulk of the problem - I had some legacy suhosin settings in php.ini where I had just disabled the module after 5.3 and left the settings in place.

Your tests do not ignore existing suhosin settings in php.ini as make test will just hunt down the currently active php.ini and copy it to the build directory and use it as is. Then tests will fail because existing settings might be too restrictive for them to pass.

Maybe you can just unset all prior suhosin settings during runtime before applying the tests?

This looks much better, the only fails left are mysqli related and that is probably because I only allow unix socket connections, investigating. Oh also suhosin.upload.disallow_binary=On [tests/filter/suhosin_upload_disallow_binary_on.phpt] fails. But that is it.

Number of tests :  173               141
Tests skipped   :   32 ( 18.5%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :   16 (  9.2%) ( 11.3%)
Expected fail   :    0 (  0.0%) (  0.0%)
Tests passed    :  125 ( 72.3%) ( 88.7%)

I cannot tell you how thrilled I am to see suhosin working with php 5.6 again, it is wonderful.

[ck-on]

Could the mysqli failures be because I am using mysqlnd instead of legacy mysqli library in php ?

Most Mysqli connect tests pass
but
All Mysqli query tests fail.

I guess I will build php with mysqli instead of mysqlnd and see what happens.

update: nope, mysqlnd vs old libraries makes no difference

It looks like Ben did the work on the mysqli? If so maybe he will have some insight.

[stefanesser]

I was away from GitHub for a week, but a colleague looked into this and the reason for the excessive failing tests might be a stdout/stderr buffering problem.

Whenever something is violated Suhosin will print out an error message inside the tests. The tests will then evaluate these errors. Right now it seems on some platforms (like yours) the error message comes out in a different line of the output (buffering problem?). This does not match what the test expected and therefore it fails.

We will look into this and try to fix it before the next real release, but it most probably means that the only thing failing is the position of the error message in the output. So the actual Suhosin features should all work as expected.

[bef]

First of all, there seems to be some kind of stdout caching problem for some test cases running on Linux. This will be triggered by excessive error output, which in some cases includes the current filename of the test. So, as a quick fix, you may shorten your path, e.g. checkout suhosin to /tmp/s or similar short path and run make test again. As far as I can see, all tested features are working perfectly fine, even if some test cases related to this issue fail. This will be resolved before the next official release.

Then there is mysqli. These tests should be skipped altogether unless mysqli is actually available to php during testing. For this to work, there must be a current Mysql installed and available as well. Environment variables control host, user, password etc. - see connect.inc:

    $host      = getenv("MYSQL_TEST_HOST")     ? getenv("MYSQL_TEST_HOST") : "localhost";
    $port      = getenv("MYSQL_TEST_PORT")     ? getenv("MYSQL_TEST_PORT") : 3306;
    $user      = getenv("MYSQL_TEST_USER")     ? getenv("MYSQL_TEST_USER") : "root";
    $passwd    = getenv("MYSQL_TEST_PASSWD")   ? getenv("MYSQL_TEST_PASSWD") : "";
    $db        = getenv("MYSQL_TEST_DB")       ? getenv("MYSQL_TEST_DB") : "test";
    $socket    = getenv("MYSQL_TEST_SOCKET")   ? getenv("MYSQL_TEST_SOCKET") : null;

So, you may try a command similar to

make test NO_INTERACTION=1 MYSQL_TEST_HOST=dbvm MYSQL_TEST_USER=user MYSQL_TEST_PASSWD=password TESTS=tests/sql/

This behaviour was copied from the mysqli test suite shipped with PHP.