Coder Social home page Coder Social logo

rotest's Introduction

Rotest

PyPI PyPI - Python Version https://coveralls.io/repos/github/gregoil/rotest/badge.svg?branch=master Read the Docs (version)

Watch the demo

Rotest is a resource oriented testing framework, for writing system or integration tests.

Rotest is based on Python's unittest module and on the Django framework. It enables defining simple abstracted components in the system, called resources. The resources may be DUT (devices under test) or they may help the test process. The tests look very much like tests written using the builtin module unittest.

Why Use Rotest?

  • Allowing teams to share resources without interfering with one another.
  • Easily abstracting automated components in the system.
  • Lots of useful features: multiprocess, filtering tests, variety of output handlers (and the ability to create custom ones), and much more.

Examples

For a complete step-by-step explanation about the framework, you can read our documentation at Read The Docs. If you just want to see how it looks, read further.

For our example, let's look at an example for a Calculator resource:

import os
import rpyc
from django.db import models
from rotest.management.models import resource_data
from rotest.management import base_resource


class CalculatorData(resource_data.ResourceData):
    class Meta:
        app_label = "resources"

    ip_address = models.IPAddressField()


class Calculator(base_resource.BaseResource):
    DATA_CLASS = CalculatorData

    PORT = 1357
    EXECUTABLE_PATH = os.path.join(os.path.expanduser("~"),
                                   "calc.py")

    def connect(self):
        self._rpyc = rpyc.classic.connect(self.data.ip_address,
                                          self.PORT)

    def calculate(self, expression):
        result = self._rpyc.modules.subprocess.check_output(
            ["python", self.EXECUTABLE_PATH, expression])
        return int(result.strip())

    def finalize(self):
        if self._rpyc is not None:
            self._rpyc.close()
            self._rpyc = None

The CalculatorData class is a standard Django model that exposes IP address of the calculator machine through the data attribute. Also, we're using rpyc for automating the access to those machines. Except from that, it's easy to notice how the connect method is making the connection to the machine, and how the finalize method is cleaning afterwards.

Now, an example for a test:

from rotest import main
from rotest.core import TestCase


class SimpleCalculationTest(TestCase):
    calculator = Calculator()

    def test_simple_calculation(self):
        self.assertEqual(self.calculator.calculate("1+2"), 3)


if __name__ == "__main__":
    main()

The test may include the setUp and tearDown methods of unittest as well, and it differs only in the request for resources.

Following, those are the options exposed when running the test:

$ rotest -h
Run tests in a module or directory.

Usage:
    rotest [<path>...] [options]

Options:
    -h,  --help
            Show help message and exit.
    --version
            Print version information and exit.
    -c <path>, --config <path>
            Test configuration file path.
    -s, --save-state
            Enable saving state of resources.
    -d <delta-iterations>, --delta <delta-iterations>
            Enable run of failed tests only - enter the number of times the
            failed tests should be run.
    -p <processes>, --processes <processes>
            Use multiprocess test runner - specify number of worker
            processes to be created.
    -o <outputs>, --outputs <outputs>
            Output handlers separated by comma.
    -f <query>, --filter <query>
            Run only tests that match the filter expression,
            e.g. 'Tag1* and not Tag13'.
    -n <name>, --name <name>
            Assign a name for current launch.
    -l, --list
            Print the tests hierarchy and quit.
    -F, --failfast
            Stop the run on first failure.
    -D, --debug
            Enter ipdb debug mode upon any test exception.
    -S, --skip-init
            Skip initialization and validation of resources.
    -r <query>, --resources <query>
            Specify resources to request by attributes,
            e.g. '-r res1.group=QA,res2.comment=CI'.

rotest's People

Contributors

dependabot[bot] avatar gregoil avatar iamshobe avatar meircif avatar osherdp avatar rfire01 avatar rinslow avatar undarkle avatar yakobu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

rotest's Issues

TestSuite work_dir ut fails

======================================================================
FAIL: test_working_dir (rotest.tests.core.test_suite.TestTestSuite)
Test the tests working directories creation and structure.

Traceback (most recent call last):
File "C:\gits\rotest\rotest\tests\core\test_suite.py", line 335, in test_working_dir
(test_suite.data, test_suite.work_dir, ROTEST_WORK_DIR))
AssertionError: Test MockTestSuite_None work directory 'C:/gits/work_dir/MockTestSuite_17.08.21_15_10_16(0)' is not contained in the base work directory 'C:/gits/work_dir/'

Support Python3.6

rotest should also support python3.6

  • don't lie at setup.py
  • add build system for python3.6
  • resolve dependencies

Make build less flaky

A successful build will fail every now and then for multiprocess reasons.
Here's a helpful traceback:

================================== FAILURES ===================================
____ TestMultiprocessTimeouts.test_suite_continues_after_teardown_timeout _____
self = <tests.core.multiprocess.test_timeout.TestMultiprocessTimeouts testMethod=test_suite_continues_after_teardown_timeout>
    def test_suite_continues_after_teardown_timeout(self):
        """Test that tests continue to run after a case is killed in tearDown.
    
            * Create a suite composed of a case which gets timeout during tearDown
                and a case which run successfully.
            * Validate that the first case was killed in time.
            * Validate that the second case was run.
            """
        TearDownTimeoutCase.pid_queue = self.pid_queue
        TearDownTimeoutCase.post_timeout_event = self.post_timeout_event
    
        BasicMultiprocessCase.pid_queue = self.pid_queue
        BasicMultiprocessCase.post_timeout_event = self.post_timeout_event
    
        MockSuite1.components = (TearDownTimeoutCase,)
        MockSuite2.components = (BasicMultiprocessCase,)
    
        MockTestSuite.components = (MockSuite1, MockSuite2)
    
        self.runner.run(MockTestSuite)
    
        self.assertFalse(self.post_timeout_event.is_set(),
                         "Process continued when it should have been "
                         "terminated due to timeout")
    
>       self.validate_test_processes(2)
tests\core\multiprocess\test_timeout.py:133: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests\core\multiprocess\test_runner.py:70: in validate_test_processes
    "Process %s wasn't killed" % pid)
E   AssertionError: Process 2052 wasn't killed
_________ TestMultiprocessTimeouts.test_suite_continues_after_timeout _________
self = <tests.core.multiprocess.test_timeout.TestMultiprocessTimeouts testMethod=test_suite_continues_after_timeout>
    def test_suite_continues_after_timeout(self):
        """Test that a test suite continues to run after a case is killed.
    
            * Create a suite composed of a case which gets timeout and a case which
                  run successfully.
            * Validate that the first case was killed in time.
            * Validate that the second case was run.
            """
        TimeoutCase.pid_queue = self.pid_queue
        TimeoutCase.post_timeout_event = self.post_timeout_event
    
        BasicMultiprocessCase.pid_queue = self.pid_queue
        BasicMultiprocessCase.post_timeout_event = self.post_timeout_event
    
        MockSuite1.components = (TimeoutCase,)
        MockSuite2.components = (BasicMultiprocessCase,)
    
        MockTestSuite.components = (MockSuite1, MockSuite2)
    
        self.runner.run(MockTestSuite)
    
        self.assertFalse(self.post_timeout_event.is_set(),
                         "Process continued when it should have been "
                         "terminated due to timeout")
    
>       self.validate_test_processes(2)
tests\core\multiprocess\test_timeout.py:106: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests\core\multiprocess\test_runner.py:70: in validate_test_processes
    "Process %s wasn't killed" % pid)
E   AssertionError: Process 652 wasn't killed
======== 2 failed, 202 passed, 3 skipped, 4 warnings in 279.36 seconds ========
ERROR: InvocationError: 'C:\\projects\\rotest\\.tox\\py27-win32\\Scripts\\pytest.EXE tests

Convert usage of lxml to xmltodict

We use lxml to encode and decode messages in our framework. The library has a very unpleasant installation process in Windows in oppose to xmltodict.
Change our usage to include only xmltodict. Don't forget to remove the requirement for lxml from requirements.txt and from setup.py.

ExcelHandler UT fails

Different OS and/or different versions of xlwt causes the excel files to be not bit-exact with the expected one.

The solution would be not to use bit-exact compare but to compare the content of the excel files

Enable result handler extensions

Our output handlers system can be extended with additional handlers that inherit from AbstractResultHandler.
The entrypoint's mechanism of setup.py can provide others (and us) to define customized result handlers.

delete states after unittests

tests such as test_save_state create files on the work_dir (~/.rotest) and don't delete them afterwards

Even better would be to use a fake fs

Cannot run anonymous TestFlows in multiprocess on Windows

Windows' pickle system, which is used when forking a process, cannot handle dynamic classes (in this case - classes created using the builtin 'type' function), such as those which are created using the 'rotest.core.clow.create_flow' function to create TestFlows.

Enable reading configuration from file

Right now, lots of our configuration is being kept by environment variables.
It's not a simple way to share configuration between several developers.
Enable reading configuration from a file rotest.ini, which is being read if no environment variable is defined.
Enable the following configuration:

  • working directory
  • server (host)
  • resource waiting time (should be optional)

Solve cyclic import design

There is a cyclic import between the following modules:

Restructure the code, so that it won't have that dependency issue, and remove the disabling of cyclic-import from the pylintrc file.

Use pipenv system

  • make sure the tox runs the pipenv shell
  • delete requirements.txt ?

Refactor %s to string formatting

Change all occurrences of %s, %d, %r and so on to using string formatting.
This is a preliminary step towards supporting python3.

Add important imports in sub_packages

to rotest.core add TestCase, TestSuite, TestBlock, TestFlow, request, block modes, runner.main
to rotest.management add BaseResource, ResourceData

Easier imports

I would like to have an easier import system
from rotest import BaseResource, TestBlock, TestFlow, main

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.