cekit / behave-test-steps

CEKit

About

Container image creation tool.

CEKit helps to build container images from image definition files with strong focus on modularity and code reuse.

Features

• Building container images from YAML image definitions
• Integration/unit testing of images

Documentation

Documentation is available here.

behave-test-steps's Issues

xml_steps doesn't correctly compare the expected element count to the actual count

I just realized I probably should have opened an issue for this, since you appear to be using issue :)

PR here:
#7

Document available steps

It should be possible to specify the timeout for particluar test

The default timeout should be used only when a timeout value was not specified in scenario.

Encoding exception when executing tests

Describe the bug
Encoding exception at the end of the test although test succeeds

To reproduce
Use image in https://github.com/jfdenise/wildfly-s2i/tree/wf-17/wildfly-builder-image
Add the feature:

Feature: WildFly basic features

  @wildfly/wildfly-centos7
  Scenario: Check basic build
    Given s2i build https://github.com/jfdenise/wildfly-s2i from test/test-app using wf-17

Expected behavior
No error

Logs

    Given s2i build https://github.com/jfdenise/wildfly-s2i from
test/test-app using wf-17 # steps/s2i_steps.py:47 21.106s
2019-05-28 16:42:29,367 - dock.middleware.container - INFO - Removing
container
'0cb1c0586261ab3893df637a65bf9876da6ba6679edb4bc1ccc7f61d5d7dccf3', 1 try...
2019-05-28 16:42:29,382 - dock.middleware.container - INFO - Container
'0cb1c0586261ab3893df637a65bf9876da6ba6679edb4bc1ccc7f61d5d7dccf3' removed
Exception UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in
position 2080: ordinal not in range(128)
2019-05-28 16:42:29,386 cekit        ERROR    An error occurred while
executing tests
2019-05-28 16:42:29,386 - cekit - ERROR - An error occurred while
executing tests

Environment information:

• RHEL 7
• CEKit version: 3.0.1
• docker: Docker version 18.09.6, build 481bc77156

Drop CI env variable

Instead we should use MAVEN_MIRROR_URL, if provided.

"When container is started with args" waits for a java process even when not calling java

We use the step "When container is started with args" in quite a few tests, e.g. all of these ones which run a variety of binaries: ls,rpm,bash, but notably not java.

That step is defined here
https://github.com/cekit/behave-test-steps/blob/v1/steps/container_steps.py#L86
Note that function has a parameter pname defaulting to java, but nothing alters that value: it's not parameterized in the @given or @when variants that select it.

start_container_with_args calls wait_for_process(context, pname) (https://github.com/cekit/behave-test-steps/blob/v1/steps/container_steps.py#L167 , always with pname=java) which does a busy-loop poll of the process table (via invoking ps) with a hard-coded 10 second timeout.

Thus, any test using the step "When container is started with args" will have an up-to 10 second delay added to its runtime looking for a java process before completing.

(I've discovered this while trying to figure out why our GHA tests runs all time out and fail. The last thing they log as running is one of the wait_for_process polling loops, although that might be a coincidence. The tests stick in that state for 5-6h before GHA kills them.)

Rename and document environment variables

Timeout not configurable for steps that check file content

The timeout is hardcoded to 10secs and can't be configured.

Fix behave escaping

I am seeing

======================================================== warnings summary =========================================================
tests/test_validate.py::test_simple_image_test
  steps/container_steps.py:210: DeprecationWarning: invalid escape sequence \(

tests/test_validate.py::test_image_test_with_override
  steps/container_steps.py:210: DeprecationWarning: invalid escape sequence \(

tests/test_validate.py::test_image_test_with_multiple_overrides
  steps/container_steps.py:210: DeprecationWarning: invalid escape sequence \(

tests/test_validate.py::test_image_test_with_override_on_cmd
  steps/container_steps.py:210: DeprecationWarning: invalid escape sequence \(

loader references python2 dependencies

This is referring to loader.py. There are two problems here:

1. EPEL 7 differs from EPEL 8 in terms of Python 3 packages (but within CEKit they are only recognised as both being under the CentOS platform) e.g.

        # Fedora / EPEL 8 : python3-requests
        # EPEL 7 : python36-requests
        deps['requests'] = {
            'library': 'requests',
            'package': 'python-requests',
            'fedora': {
                'package': 'python3-requests',
            }
        }

This could be partially resolved within CEKit if the package determination in tools.py also allowed <platform><release as well as <platform> e.g. centos7 so we could then have:

            'package': 'python3-requests',
            'centos7': {
                'package': 'python36-requests',
            }

2. EPEL 7 and 8 don't have a Python3 package for behave built

        deps['behave'] = {
            'library': 'behave',
            'package': 'python2-behave',
            'fedora': {
                'package': 'python3-behave',
            }
        }

This is https://bugzilla.redhat.com/show_bug.cgi?id=2106695 and https://bugzilla.redhat.com/show_bug.cgi?id=1885570

cc @jmtd

s2i steps default to git ref "master", but not all git repositories have a master branch

In the absence of an explicit ref in the step "Given s2i build...", i.e., Given s2i build https://...somerepo.../ with no using branchname suffix, the S2I steps will default to try to check out a branch named master, which may not exist in the repository.

Instead I think it should use the repository's default branch name.

We hit this (side issue) by introducing a new repository for cekit/behave tests, https://github.com/jboss-container-images/openjdk-test-applications, which has the default branch name main. This is the new default, I think, for both Git and Github. (At the time of writing this, I'm about to push a master branch to that repository as a temporary mitigation).

The culprits are

behave-test-steps/steps/s2i_steps.py

Line 14 in e92a239

def s2i_inner(context, application, path='.', env="", incremental=False, tag="master", runtime_image=""):

and

behave-test-steps/steps/s2i_steps.py

Line 59 in e92a239

def s2i_build(context, application, path='.', env="", incremental=False, tag="master", run=True, runtime_image=""):

and possibly a few other places.

Make S2i steps parameters configurable by env variables.

We are in a case where tests scenarii could be shared by multiple images if the s2i build step was configurable (eg: build src repo, tag, context-dir, ...).
A user could write something like:
Given s2i build $BEHAVE_TEST_REPO from $BEHAVE_TEST_APP1 using $BEHAVE_TEST_REPO_TAG

The various env variables would have to be set in the context of the test execution.

We could consider that, if a parameter starts with '$', it is an env variable to be resolved. If no such env variable exists, then the parameter is passed as is to the s2i command.

warnings about comparisons with literals

These seem new! In my terminal output when running a "behave test":

2021-05-24 10:08:25,965 collector.py:41         INFO  Fetching common steps from 'https://github.com/cekit/behave-test-steps.git'.
2021-05-24 10:08:26,655 collector.py:73         INFO  Collecting finished!
steps/container.py:170: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if retcode is not 0:
steps/container_steps.py:255: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if len(output) is 0:
/home/jon/rh/git/images/openjdk/target/test/steps/steps.py:66: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if retcode is not 0:
/home/jon/rh/git/images/openjdk/target/test/steps/container.py:170: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if retcode is not 0:
steps/steps.py:66: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if retcode is not 0:

Better handling of non-running containers

When killing the container after the test it may happen that it's already not running. In such case we should print some message to suggest to take a look at the container logs because most probably the application did not start correctly, crashed and the container was already removed.

Step to check that log contains a message n times

We have testcases that check for a certain number of occurrences of a message.