pelioniot / client-e2e-python-test-library

Python Test library for Izuma DM Client end-2-end tests

Client E2E Python test library

Izuma Client E2E tests verify that a target platform can perform essential Device Management Client operations. The library is designed to be used with the pytest test framework.

Prerequisites

Python 3.5 or later.

Installation

$ git clone https://github.com/PelionIoT/client-e2e-python-test-library.git
$ pip install wheel
$ python3 setup.py bdist_wheel
$ cd dist/
$ pip install -I client_test_lib*.whl

Basic usage

• Build the Device Management Client example application for your board and flash it.

• Set the CLOUD_API_KEY environment variable with your access key.

  • Linux: export CLOUD_API_KEY=<access_key_here>
  • Windows: set CLOUD_API_KEY=<access_key_here>

• Default API address is https://api.us-east-1.mbedcloud.com. You can change this by defining CLOUD_API_GW environment variable in similar way as CLOUD_API_KEY is done above.

• Test run will create temporary API key for the WebSocket callback channel by default. If you want to prevent that and use only the exported API key, add --use_one_apikey startup argument.

• Tests use Mbed LS to select the board from the serial port.

  • If you have only one board connected to the serial port, you don't need to select the device for the tests.
  • If there are multiple boards connected to the serial port, run mbedls to check the target board's ID, and use it in the test run's argument --target_id=[id].

  $ mbedls
  +---------------+----------------------+-------------+--------------+--------------------------------------------------+-----------------+
  | platform_name | platform_name_unique | mount_point | serial_port  | target_id                                        | daplink_version |
  +---------------+----------------------+-------------+--------------+--------------------------------------------------+-----------------+
  | K64F          | K64F[0]              | D:          | /dev/ttyACM0 | 0240000032044e4500257009997b00386781000097969900 | 0244            |
  +---------------+----------------------+-------------+--------------+--------------------------------------------------+-----------------+
  

• You can also run tests with the Linux build of the Device Management Client by passing the compiled binary in the --local_binary=./mbedCloudClientExample.elf argument.

Running a test set

To run a test set for Device Management Client use the command:

pytest tests/dev-client-tests.py

Read later instructions how to setup the update test to go with the test set.

Running a single test

To run a single test from the set, use the -k argument to set the test name as a keyword:

pytest tests/dev-client-tests.py -k get_resource

Running the update test

Before running the update test, make sure you create update-related configuration and initialize the developer environment properly, as describe the Device Management Client example tutorial.

The update test uses the manifest-dev-tool utility to create the actual manifest for update campaigns.

To run the update test, provide the mandatory update image path and optional manifest-dev-tool init path arguments at startup:

• --update_bin=/home/user/mbed-cloud-client-example_update.bin absolute path for the update image
• --manifest_tool=/home/user/mbed-os-example-pelion absolute path where manifest-dev-tool init is executed - defaults to the current working directory.

To leave the firmware image, manifest and campaign in your account after the test, add the --no_cleanup startup argument.

Manifest tool 2.0.0 supports two manifest schema versions: v1 and v3. By default, the update test creates v3 manifests, but you can create v1 manifests by passing the --manifest_version=v1 startup argument.

Results output

Add the startup arguments to adjust the generated output:

• --log-cli-level=DEBUG adds more details
• --html=results.html generates an HTML report
• --junitxml=junit.xml provides output for CI systems, for example Jenkins

The library also writes a separate client.log file from the Device Management Client output.

Customized test runs

There are many ways to configure the test runs. Refer to the full pytest documentation for more information.

Current tests

Test name	Main functions	Notes
test_01_get_device_id	Verify that the device is registered.
test_02_get_resource	Verify that the device responds to GET.	Uses Resource /1/0/1
test_03_put_resource	Verify that the device responds to PUT.	Uses Resource /1/0/1
test_04_subscribe_resource	Verify the notification from the subscribed resource.	Uses Resource /1/0/1
test_05_factory_reset	Verify the client's factory reset behaviour.
test_06_update_device	Verify the device firmware update.

License

See the license agreement.

client-e2e-python-test-library's Issues

Missing support for explicit firmware version

I am working with PSoc64 (target name: CYTFM_064B0S2_4343W).

Background:

• Supported in branch: cytfm-064b0s2-4343w at https://github.com/ARMmbed/mbed-os-example-pelion/.
• Detailed tutorial is at: https://github.com/ARMmbed/mbed-os-example-pelion-armdevsummit
• Firmware update on this platform requires manifest-tool 2.0.0.

This platform requires explicitly passing the new firmware version in 64-bit unsigned integer. The Cypress PSoC64 has it's own boot loader that also uses explicit versioning (instead of time stamp).

Updating the firmware works successfully with the following command (that includes the explicit firmware version).

manifest-dev-tool update-v1 --payload-path <payload> --fw-version <new firmware version> --device-id <device ID> --start-campaign --wait-for-completion

However, when I run the Pelion E2E tests, using the command below, there is no way to pass in the firmware version.

pytest TESTS/pelion-e2e-python-test-library/tests/dev-client-tests.py --update_bin=<path to update binary>

The device fails to start downloading the firmware payload & the service reports that the existing firmware version is at or greater than the firmware version listed in the manifest.

Could the problem be that the manifest is not getting generated with the explicit firmware version?

Is there a quick fix for this?

CC @moshe-shahar @cvasilak

Meet 'FileNotFoundError' on git-bash/Windows 10

Environment

Windows 10 64-bit
Python 3.7
Git for Windows 2.25.0 64-bit
manifest-tool 1.5.2

Description

I am running Pelion E2E tests. For non-update test, it is passed:

pytest TESTS/pelion-e2e-python-test-library/tests/dev-client-tests.py \
-k test_01_get_device_id

For update test, I meet FileNotFoundError.

pytest TESTS/pelion-e2e-python-test-library/tests/dev-client-tests.py \
-k test_06_update_device \
--update_bin=/C/CCLi8/IoT/mbed/mbed-examples/mbed-os-example-pelion/BUILD/NUMAKER_PFM_NUC472/ARMC6/mbed-os-example-pelion_update.bin \
--manifest_tool=/C/CCLi8/IoT/mbed/mbed-examples/mbed-os-example-pelion

Per preliminary check, the update image upload step has succeeded. The failure point is in manifest-tool create. I check my manifest-tool installation directory and find there's no manifest-tool.exe (only manifest-tool python script) there. I am not sure if this is the cause on Windows environment.

Log

=================================== ERRORS ====================================
___________________ ERROR at setup of test_06_update_device ___________________

cloud = <pelion_test_lib.cloud.cloud.PelionCloud object at 0x000001701AAC6C50>
client = <pelion_test_lib.tools.client_runner.Client object at 0x000001701B3BDC50>
request = <SubRequest 'update_device' for <Function test_06_update_device>>

    @pytest.fixture(scope='function')
    def update_device(cloud, client, request):
        """
        Fixture for updating device.
        This uploads firmware image, creates manifest, uploads manifest, and creates and starts the update campaign.
        :param cloud: Cloud fixture
        :param client: Client fixture
        :param request: Requests fixture
        :return: Campaign ID
        """
        binary_path = request.config.getoption('update_bin', None)
        manifest_tool_path = request.config.getoption('manifest_tool', os.getcwd())
        no_cleanup = request.config.getoption('no_cleanup', False)
        delta_manifest = request.config.getoption('delta_manifest', None)
        log.info('Update image: "{}"'.format(binary_path))
        log.info('Path for manifest-tool init: "{}"'.format(manifest_tool_path))
        if not binary_path or not manifest_tool_path:
            skip_msg = 'Provide missing binary image in startup arguments to run update case\n' \
                       '--update_bin={}'.format(binary_path if binary_path else 'MISSING!')
            log.warning(skip_msg)
            pytest.skip(skip_msg)
    
        sleep(5)
        resp = cloud.device_directory.get_device(client.endpoint_id(), expected_status_code=200).json()
        assert resp['state'] == 'registered'
    
        fw_image = cloud.update.upload_firmware_image(binary_path, expected_status_code=201).json()
        fw_image_id = fw_image['id']
        log.info('Firmware image uploaded! Image ID: {}'.format(fw_image_id))
    
        manifest_file = manifest_tool.create_manifest(path=manifest_tool_path,
                                                      firmware_url=fw_image['datafile'],
                                                      update_image_path=binary_path,
>                                                     delta_manifest=delta_manifest)

..\..\mbed-libs\pelion-e2e-python-test-library\pelion_test_lib\fixtures\cloud_fixtures.py:138: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
TESTS\pelion-e2e-python-test-library\pelion_test_lib\tools\manifest_tool.py:92: in create_manifest
    p = subprocess.Popen(cmd, cwd=path, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
c:\python37\lib\subprocess.py:775: in __init__
    restore_signals, start_new_session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <subprocess.Popen object at 0x000001701B3EB940>
args = 'manifest-tool create -u http://firmware-catalog-media-ca57.s3.dualstack.us-east-1.amazonaws.com/mbed-os-example-pelio...:/CCLi8/IoT/mbed/mbed-examples/mbed-os-example-pelion/BUILD/NUMAKER_PFM_NUC472/ARMC6/mbed-os-example-pelion_update.bin'
executable = None, preexec_fn = None, close_fds = False, pass_fds = ()
cwd = 'C:\\CCLi8\\IoT\\mbed\\mbed-examples\\mbed-os-example-pelion', env = None
startupinfo = <subprocess.STARTUPINFO object at 0x000001701B3EB0B8>
creationflags = 0, shell = False, p2cread = Handle(900), p2cwrite = -1
c2pread = 12, c2pwrite = Handle(704), errread = 13, errwrite = Handle(1000)
unused_restore_signals = True, unused_start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       unused_restore_signals, unused_start_new_session):
        """Execute program (MS Windows version)"""
    
        assert not pass_fds, "pass_fds not supported on Windows."
    
        if not isinstance(args, str):
            args = list2cmdline(args)
    
        # Process startup details
        if startupinfo is None:
            startupinfo = STARTUPINFO()
        else:
            # bpo-34044: Copy STARTUPINFO since it is modified above,
            # so the caller can reuse it multiple times.
            startupinfo = startupinfo._copy()
    
        use_std_handles = -1 not in (p2cread, c2pwrite, errwrite)
        if use_std_handles:
            startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
            startupinfo.hStdInput = p2cread
            startupinfo.hStdOutput = c2pwrite
            startupinfo.hStdError = errwrite
    
        attribute_list = startupinfo.lpAttributeList
        have_handle_list = bool(attribute_list and
                                "handle_list" in attribute_list and
                                attribute_list["handle_list"])
    
        # If we were given an handle_list or need to create one
        if have_handle_list or (use_std_handles and close_fds):
            if attribute_list is None:
                attribute_list = startupinfo.lpAttributeList = {}
            handle_list = attribute_list["handle_list"] = \
                list(attribute_list.get("handle_list", []))
    
            if use_std_handles:
                handle_list += [int(p2cread), int(c2pwrite), int(errwrite)]
    
            handle_list[:] = self._filter_handle_list(handle_list)
    
            if handle_list:
                if not close_fds:
                    warnings.warn("startupinfo.lpAttributeList['handle_list'] "
                                  "overriding close_fds", RuntimeWarning)
    
                # When using the handle_list we always request to inherit
                # handles but the only handles that will be inherited are
                # the ones in the handle_list
                close_fds = False
    
        if shell:
            startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
            startupinfo.wShowWindow = _winapi.SW_HIDE
            comspec = os.environ.get("COMSPEC", "cmd.exe")
            args = '{} /c "{}"'.format (comspec, args)
    
        # Start the process
        try:
            hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                                     # no special security
                                     None, None,
                                     int(not close_fds),
                                     creationflags,
                                     env,
                                     os.fspath(cwd) if cwd is not None else None,
>                                    startupinfo)
E                                    FileNotFoundError: [WinError 2] The system cannot find the file specified

c:\python37\lib\subprocess.py:1178: FileNotFoundError

Fail to install dependencies pytest / pytest-html

First, I try to install this tool as indicated in the docs:

python setup.py bdist_wheel
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'bdist_wheel'

Then trying to install this using install:
python setup.py install

The installation fails because these two modules can't be found (besides are in requirements.txt)

• pytest
• pytest-html

So I need to manually install them using pip.