thombashi / allpairspy

A python library for test combinations generator. The generator allows one to create a set of tests using "pairwise combinations" method, reducing a number of combinations of variables into a lesser set that covers most situations.

allpairspy forked from bayandin/allpairs

• AllPairs test combinations generator
• Features
• Get Started
  • Basic Usage
  • Filtering
  • Data Source: OrderedDict
  • Parameterized testing pairwise by using pytest
  • Other Examples
• Installation
  • Installation: pip
  • Installation: apt
• Known issues
• Dependencies
• Sponsors

AllPairs test combinations generator

AllPairs is an open source test combinations generator written in Python, developed and maintained by MetaCommunications Engineering. The generator allows one to create a set of tests using "pairwise combinations" method, reducing a number of combinations of variables into a lesser set that covers most situations.

For more info on pairwise testing see http://www.pairwise.org.

Features

• Produces good enough dataset.
• Pythonic, iterator-style enumeration interface.
• Allows to filter out "invalid" combinations during search for the next combination.
• Goes beyond pairs! If/when required can generate n-wise combinations.

Get Started

Basic Usage

Sample Code:

from allpairspy import AllPairs parameters = [ ["Brand X", "Brand Y"], ["98", "NT", "2000", "XP"], ["Internal", "Modem"], ["Salaried", "Hourly", "Part-Time", "Contr."], [6, 10, 15, 30, 60], ] print("PAIRWISE:") for i, pairs in enumerate(AllPairs(parameters)): print("{:2d}: {}".format(i, pairs))

Output:

PAIRWISE: 0: ['Brand X', '98', 'Internal', 'Salaried', 6] 1: ['Brand Y', 'NT', 'Modem', 'Hourly', 6] 2: ['Brand Y', '2000', 'Internal', 'Part-Time', 10] 3: ['Brand X', 'XP', 'Modem', 'Contr.', 10] 4: ['Brand X', '2000', 'Modem', 'Part-Time', 15] 5: ['Brand Y', 'XP', 'Internal', 'Hourly', 15] 6: ['Brand Y', '98', 'Modem', 'Salaried', 30] 7: ['Brand X', 'NT', 'Internal', 'Contr.', 30] 8: ['Brand X', '98', 'Internal', 'Hourly', 60] 9: ['Brand Y', '2000', 'Modem', 'Contr.', 60] 10: ['Brand Y', 'NT', 'Modem', 'Salaried', 60] 11: ['Brand Y', 'XP', 'Modem', 'Part-Time', 60] 12: ['Brand Y', '2000', 'Modem', 'Hourly', 30] 13: ['Brand Y', '98', 'Modem', 'Contr.', 15] 14: ['Brand Y', 'XP', 'Modem', 'Salaried', 15] 15: ['Brand Y', 'NT', 'Modem', 'Part-Time', 15] 16: ['Brand Y', 'XP', 'Modem', 'Part-Time', 30] 17: ['Brand Y', '98', 'Modem', 'Part-Time', 6] 18: ['Brand Y', '2000', 'Modem', 'Salaried', 6] 19: ['Brand Y', '98', 'Modem', 'Salaried', 10] 20: ['Brand Y', 'XP', 'Modem', 'Contr.', 6] 21: ['Brand Y', 'NT', 'Modem', 'Hourly', 10]

Filtering

You can restrict pairs by setting a filtering function to filter_func at AllPairs constructor.

Sample Code:

from allpairspy import AllPairs def is_valid_combination(row): """ This is a filtering function. Filtering functions should return True if combination is valid and False otherwise. Test row that is passed here can be incomplete. To prevent search for unnecessary items filtering function is executed with found subset of data to validate it. """ n = len(row) if n > 1: # Brand Y does not support Windows 98 if "98" == row[1] and "Brand Y" == row[0]: return False # Brand X does not work with XP if "XP" == row[1] and "Brand X" == row[0]: return False if n > 4: # Contractors are billed in 30 min increments if "Contr." == row[3] and row[4] < 30: return False return True parameters = [ ["Brand X", "Brand Y"], ["98", "NT", "2000", "XP"], ["Internal", "Modem"], ["Salaried", "Hourly", "Part-Time", "Contr."], [6, 10, 15, 30, 60] ] print("PAIRWISE:") for i, pairs in enumerate(AllPairs(parameters, filter_func=is_valid_combination)): print("{:2d}: {}".format(i, pairs))

Output:

PAIRWISE: 0: ['Brand X', '98', 'Internal', 'Salaried', 6] 1: ['Brand Y', 'NT', 'Modem', 'Hourly', 6] 2: ['Brand Y', '2000', 'Internal', 'Part-Time', 10] 3: ['Brand X', '2000', 'Modem', 'Contr.', 30] 4: ['Brand X', 'NT', 'Internal', 'Contr.', 60] 5: ['Brand Y', 'XP', 'Modem', 'Salaried', 60] 6: ['Brand X', '98', 'Modem', 'Part-Time', 15] 7: ['Brand Y', 'XP', 'Internal', 'Hourly', 15] 8: ['Brand Y', 'NT', 'Internal', 'Part-Time', 30] 9: ['Brand X', '2000', 'Modem', 'Hourly', 10] 10: ['Brand Y', 'XP', 'Modem', 'Contr.', 30] 11: ['Brand Y', '2000', 'Modem', 'Salaried', 15] 12: ['Brand Y', 'NT', 'Modem', 'Salaried', 10] 13: ['Brand Y', 'XP', 'Modem', 'Part-Time', 6] 14: ['Brand Y', '2000', 'Modem', 'Contr.', 60]

Data Source: OrderedDict

You can use collections.OrderedDict instance as an argument for AllPairs constructor. Pairs will be returned as collections.namedtuple instances.

Sample Code:

from collections import OrderedDict from allpairspy import AllPairs parameters = OrderedDict({ "brand": ["Brand X", "Brand Y"], "os": ["98", "NT", "2000", "XP"], "minute": [15, 30, 60], }) print("PAIRWISE:") for i, pairs in enumerate(AllPairs(parameters)): print("{:2d}: {}".format(i, pairs))

Sample Code:

PAIRWISE: 0: Pairs(brand='Brand X', os='98', minute=15) 1: Pairs(brand='Brand Y', os='NT', minute=15) 2: Pairs(brand='Brand Y', os='2000', minute=30) 3: Pairs(brand='Brand X', os='XP', minute=30) 4: Pairs(brand='Brand X', os='2000', minute=60) 5: Pairs(brand='Brand Y', os='XP', minute=60) 6: Pairs(brand='Brand Y', os='98', minute=60) 7: Pairs(brand='Brand X', os='NT', minute=60) 8: Pairs(brand='Brand X', os='NT', minute=30) 9: Pairs(brand='Brand X', os='98', minute=30) 10: Pairs(brand='Brand X', os='XP', minute=15) 11: Pairs(brand='Brand X', os='2000', minute=15)

Parameterized testing pairwise by using pytest

Parameterized testing: value matrix

Sample Code:

import pytest from allpairspy import AllPairs def function_to_be_tested(brand, operating_system, minute) -> bool: # do something return True class TestParameterized(object): @pytest.mark.parametrize(["brand", "operating_system", "minute"], [ values for values in AllPairs([ ["Brand X", "Brand Y"], ["98", "NT", "2000", "XP"], [10, 15, 30, 60] ]) ]) def test(self, brand, operating_system, minute): assert function_to_be_tested(brand, operating_system, minute)

Output:

$ py.test test_parameterize.py -v ============================= test session starts ============================== ... collected 16 items test_parameterize.py::TestParameterized::test[Brand X-98-10] PASSED [ 6%] test_parameterize.py::TestParameterized::test[Brand Y-NT-10] PASSED [ 12%] test_parameterize.py::TestParameterized::test[Brand Y-2000-15] PASSED [ 18%] test_parameterize.py::TestParameterized::test[Brand X-XP-15] PASSED [ 25%] test_parameterize.py::TestParameterized::test[Brand X-2000-30] PASSED [ 31%] test_parameterize.py::TestParameterized::test[Brand Y-XP-30] PASSED [ 37%] test_parameterize.py::TestParameterized::test[Brand Y-98-60] PASSED [ 43%] test_parameterize.py::TestParameterized::test[Brand X-NT-60] PASSED [ 50%] test_parameterize.py::TestParameterized::test[Brand X-NT-30] PASSED [ 56%] test_parameterize.py::TestParameterized::test[Brand X-98-30] PASSED [ 62%] test_parameterize.py::TestParameterized::test[Brand X-XP-60] PASSED [ 68%] test_parameterize.py::TestParameterized::test[Brand X-2000-60] PASSED [ 75%] test_parameterize.py::TestParameterized::test[Brand X-2000-10] PASSED [ 81%] test_parameterize.py::TestParameterized::test[Brand X-XP-10] PASSED [ 87%] test_parameterize.py::TestParameterized::test[Brand X-98-15] PASSED [ 93%] test_parameterize.py::TestParameterized::test[Brand X-NT-15] PASSED [100%]

Parameterized testing: OrderedDict

Sample Code:

import pytest from allpairspy import AllPairs def function_to_be_tested(brand, operating_system, minute) -> bool: # do something return True class TestParameterized(object): @pytest.mark.parametrize( ["pair"], [ [pair] for pair in AllPairs( OrderedDict( { "brand": ["Brand X", "Brand Y"], "operating_system": ["98", "NT", "2000", "XP"], "minute": [10, 15, 30, 60], } ) ) ], ) def test(self, pair): assert function_to_be_tested(pair.brand, pair.operating_system, pair.minute)

Other Examples

Other examples could be found in examples directory.

Installation

Installation: pip

pip install allpairspy

Installation: apt

You can install the package by apt via a Personal Package Archive (PPA):

sudo add-apt-repository ppa:thombashi/ppa
sudo apt update
sudo apt install python3-allpairspy

Known issues

• Not optimal - there are tools that can create smaller set covering all the pairs. However, they are missing some other important features and/or do not integrate well with Python.
• Lousy written filtering function may lead to full permutation of parameters.
• Version 2.0 has become slower (a side-effect of introducing ability to produce n-wise combinations).

Dependencies

Python 3.7+ no external dependencies.

Sponsors

Become a sponsor

allpairspy's Issues

Update README file

In readme file at the Installation section you mentioned that to install your lib one should run the following command: pip install allpairpy

But you lost 's' letter that can confuse user, correct command is : pip install allpairspy

Got wrong results(miss many combinations) when using 3-wise and higher

Allpairspy is a very useful tool, thank you for your contribution. Unfortunately I have encountered some problems during use with 3-wise (also higher wise) .
Here is the code:

from allpairspy import AllPairs
parameters = [
[1,2],
[1,2],
[1,2],
[1,2],
]
for i, pairs in enumerate(AllPairs(parameters,n=3)):
print(f"{i:2d}: {pairs}")

And here is the result:

0: [1, 1, 1, 1]
1: [2, 2, 2, 1]
2: [2, 2, 1, 2]
3: [1, 1, 2, 2]
4: [1, 1, 2, 1]

It can be clearly seen that many cases are missing in the results. For example, 1 2 * * , 2 1 * * are none included. And this is the result generated by pict:

first second third fourth
2 1 1 2
2 2 2 2
1 1 2 2
1 1 1 1
1 2 2 1
2 2 1 1
2 1 2 1
1 2 1 2

There are 8 combinations here and the result in allpairspy is only 6.
And the official example 1.2 also has the same problem. Using allpairspy can only get 27 combination results, while pict gets 80 results. Obviously pict is correct.
I want to know is there a mistake in my configuration or method when I use it? or other reasons.
Any suggestions are welcome. Thanks in advance!

Insufficient combination when filter function is specified?

Hello, thank you for providing a great library!
I tried to use filter function to cut invalid combination, but it seems generated combinations are insufficient.

# Generate combinations of every supported Python + NumPy version combination.

from allpairspy import AllPairs

numpy = ['1.9', '1.10', '1.11', '1.12', '1.13', '1.14']
python = ['2.7', '3.4', '3.5', '3.6']

def _validate(numpy, python):
    valid = []
    if numpy == '1.9':
        valid = ['2.7', '3.4']
    elif numpy == '1.10':
        valid = ['2.7', '3.4']
    elif numpy == '1.11':
        valid = ['2.7', '3.4', '3.5']
    elif numpy == '1.12':
        valid = ['2.7', '3.4', '3.5', '3.6']
    elif numpy == '1.13':
        valid = ['2.7', '3.4', '3.5', '3.6']
    elif numpy == '1.14':
        valid = ['2.7', '3.4', '3.5', '3.6']
    return python in valid


def _filter_func(row):
    if len(row) == 2:
        stat = _validate(*row)
        #print('Valid? ', row, ' -> ', stat)
        return stat
    return True


if __name__ == '__main__':
    for i, pairs in enumerate(AllPairs([numpy, python], filter_func=_filter_func)):
        print(i, ':', pairs)

I got:

0 : ['1.9', '2.7']
1 : ['1.10', '2.7']
2 : ['1.11', '2.7']
3 : ['1.12', '2.7']
4 : ['1.13', '2.7']
5 : ['1.14', '2.7']
6 : ['1.14', '3.4']
7 : ['1.13', '3.4']
8 : ['1.12', '3.4']
9 : ['1.11', '3.4']
10 : ['1.10', '3.4']
11 : ['1.9', '3.4']

It seems Python 3.5/3.6 variations are totally ignored. Am I missing something?
Any suggestions are welcome. Thanks in advance!

incorrect distribution for one parameter

result = AllPairs(OrderedDict( {"a1": ["0", "1"], "a2": ["0", "1"], "a3": ["0", "1"], "a4": ["0", "1"], "a5": ["0", "1"], "a6": ["0", "1"]}))

generates:
[
Pairs(a1='0', a3='0', a2='0', a5='0', a4='0', a6='0'),
Pairs(a1='1', a3='1', a2='1', a5='1', a4='1', a6='0'),
Pairs(a1='1', a3='0', a2='1', a5='0', a4='1', a6='1'),
Pairs(a1='0', a3='1', a2='0', a5='1', a4='0', a6='1'),
Pairs(a1='0', a3='1', a2='1', a5='0', a4='0', a6='1'),
Pairs(a1='1', a3='0', a2='0', a5='1', a4='0', a6='1'),
Pairs(a1='0', a3='0', a2='0', a5='1', a4='1', a6='1')]

as you may see the a6 has 5 "1" while the other 3 or 4.

similar picture for 8 params:
[Pairs(a1='0', a3='0', a2='0', a5='0', a4='0', a7='0', a6='0', a8='0'),
Pairs(a1='1', a3='1', a2='1', a5='1', a4='1', a7='1', a6='1', a8='0'),
Pairs(a1='1', a3='0', a2='1', a5='0', a4='1', a7='0', a6='1', a8='1'),
Pairs(a1='0', a3='1', a2='0', a5='1', a4='0', a7='1', a6='0', a8='1'),
Pairs(a1='0', a3='1', a2='1', a5='0', a4='0', a7='1', a6='1', a8='1'),
Pairs(a1='1', a3='0', a2='0', a5='1', a4='1', a7='1', a6='0', a8='1'),
Pairs(a1='1', a3='1', a2='0', a5='1', a4='0', a7='0', a6='1', a8='1'),
Pairs(a1='0', a3='1', a2='1', a5='1', a4='1', a7='1', a6='0', a8='1')]