Coder Social home page Coder Social logo

pytest-regressions's Introduction

pytest-regressions

PyPI version

image

Python versions

image

image

image

pre-commit.ci status

Fixtures to write regression tests.

Features

This plugin makes it simple to test general data, images, files, and numeric tables by saving expected data in a data directory (courtesy of pytest-datadir) that can be used to verify that future runs produce the same data.

See the docs for examples and API usage.

Requirements

  • pytest>=3.5
  • Python 3.6+.

Installation

You can install "pytest-regressions" via pip from PyPI:

$ pip install pytest-regressions

Contributing

Contributions are very welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the MIT license, "pytest-regressions" is free and open source software

Issues

If you encounter any problems, please file an issue along with a detailed description.


This pytest plugin was generated with Cookiecutter along with @hackebrot's cookiecutter-pytest-plugin template.

pytest-regressions's People

Contributors

12rambau avatar 648trindade avatar alexandrebbruno avatar borellim avatar csantosss avatar drgfreeman avatar edetec avatar erihe251 avatar gabrieldemarmiesse avatar jmargeta avatar jpn-- avatar macelai avatar nicoddemus avatar nobreconfrade avatar pre-commit-ci[bot] avatar renefritze avatar ronarbo avatar seanv507 avatar sjvrijn avatar tadeu avatar tarcisiofischer avatar tgs avatar tovrstra avatar williamjamir avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pytest-regressions's Issues

num_regression outputs garbage to its file if arrays are of different size

If you write arrays of different sizes on the data dict, num_regression fills the missing positions with garbage (or at least it seems garbage):

def testNumRegression(num_regression):
    data = {}
    for index in range(4):
        data[f'data_{index}'] = np.array([index])

    data['other'] = np.array([10, 20, 30, 40, 50, 60])

    num_regression.check(data)

Output:

,data_0,data_1,data_2,data_3,other
0,0,1,2,3,10
1,-2147483648,-2147483648,-2147483648,-2147483648,20
2,-2147483648,-2147483648,-2147483648,-2147483648,30
3,-2147483648,-2147483648,-2147483648,-2147483648,40
4,-2147483648,-2147483648,-2147483648,-2147483648,50
5,-2147483648,-2147483648,-2147483648,-2147483648,60

Output converted to a markdown table so it's easier to visualize:

data_0 data_1 data_2 data_3 other
0 0 1 2 3 10
1 -2147483648 -2147483648 -2147483648 -2147483648 20
2 -2147483648 -2147483648 -2147483648 -2147483648 30
3 -2147483648 -2147483648 -2147483648 -2147483648 40
4 -2147483648 -2147483648 -2147483648 -2147483648 50
5 -2147483648 -2147483648 -2147483648 -2147483648 60

I can think of some ways of solving this:

  1. Raise an error suggesting that the user write the output on different files because the arrays have different sizes.
  2. Separate it in two tables, but I don't know how difficult would be for num_regression to process multiple tables on the same file.
  3. Fill the missing items with the space character.

Support for "ignoring a list of specific fields"

I was using this tool to generate our regression baselines, the result is a JSON string like:

{
"id": "4163B50C4DD9911804894F827078B218",
"instanceId": "020B24C54D891A89B44AAABDBCFCE057",
 "status": 1
}

The problem is that there are some fields such as the instanceId field that is using UUID that will always change, and I don't want this false alert.
One workaround would be:

  • We provide the ignores=["instanceId",] to the data_regression function, and the value of this field will be ignored when comparing the result.

  • Or, we can replace the value of specific field using special string, such as "{% ignore %}" or something else;

Bool values in num_regression fail when different

When checking arrays of boolean values with num_regression, if there are any differences between expected and obtained, the code tries to do a subtraction between the two pd.Series. This seems not to be supported by Pandas, and results in the following message:

UserWarning: evaluating in Python space because the '-' operator is not supported by numexpr for the bool dtype, use '^' instead

Despite being a UserWarning, this aborts the test.

One should use a logical XOR operation instead of a subtraction when testing boolean values.

use `isinstance` instead of type

I am creating a class that inherit from pd.DataFrame. in short it creates qa specific dataframe from a specific type of data. The consequence is that in my test I cannot use the dataframe_regression because my type is pygaul.Names and not exactly pd.DataFrame. On the other hand if you were using isinstance it would work thanks to the inheritance process.

https://github.com/ESSS/pytest-regressions/blob/1bd9e7ba24f7054299d4b335b92e79c38a568cf9/src/pytest_regressions/dataframe_regression.py#L239C39-L239C39

If this is a wanted feature, I'd be happy to make a PR

file_regression fixture will unexpectedly copy file to `tmp_path`

In my test, I use both tmp_path and file_regression fixtures. However, I expect the tmp_path should be empty for a new test since it is function session scope. But when I set breakpoint and go to the template folder, it has a folder with all the files for file_regression compare.

# test_dum.py

def test_foobar(file_regression, tmp_path):
    import ipdb; ipdb.set_trace() # go to the tmp path and will have a folder named `test_dum`

Forcing Does Not Force (force-regen)

Hi everyone!

I was using file_regressions for my tests, and after some changes to the output file I decided to re-generate the dumps with force-regen flag. It did not work, so I was researching for an issue and found a previously closed one: #69.

According to the issue and to my personal findings in the code, I see that --force-regen will only work when there is an AssertionError happening here.

However, I do not really see the reason why we can't regenerate the file when we explicitly force this operation. For example, as I am using file_regression , I write my own checker function which checks several aspects of the files. And after some time I decided to add few more checks to the function, but, according to the current logic, it won't regenerate the file, because previous checks did not fail. In my opinion, if we force then even the files with no change should be regenerated.

Another sub-issue is that I am using pytest-assume, with which I could replace assert with pytest.assume, so that all the tests in the particular function are executed. However, as it is does not explicitly raise AssertionError, it then won't regenerate the files even if there is a sufficient difference.

As the proposition, may be it would be good to move if force_regen in line here to the finally: subsection of try-except clause? It could potentially solve both issues and does not require too much changes.

Order shouldn't matter in comparison

The two yaml files are equivalent to the same python dict (i. e. if they are loaded into a dict, they would be equal)

But the comparison of data_regression.check does consider the order of the fields and a test can fail using one file and pass using the other.

Is this by design or just a consequence of how the yaml files are compared?

abc: "123"
foo: "bar"
foo: "bar"
abc: "123"

DeprecationWarning raised in dataframe_regression with numpy 1.20

When running dataframe_regression.check() with numpy 1.20, the following DeprecationWarning is issued:

 ...\lib\site-packages\pytest_regressions\dataframe_regression.py:126: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    if data_type in [float, np.float, np.float16, np.float32, np.float64]:

This is documented in https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations.

Since as per the above np.float is equivalent to float, removing it from this line should eliminate the warning without affecting functionality.

if data_type in [float, np.float, np.float16, np.float32, np.float64]:

module 'pytest' has no attribute 'Parser' in 2.4.0

I haven't yet looked into all the details, but since the pytest-regressions 2.4.0 release our CI builds started failing with

  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pytest_regressions/plugin.py", line 15, in <module>
    def pytest_addoption(parser: pytest.Parser) -> None:
AttributeError: module 'pytest' has no attribute 'Parser'

And this despite we're using pytest 6.2.5 https://github.com/aiidateam/aiida-core/actions/runs/3072038424/jobs/4963247024#step:8:140

I read in the changelog that pytest >=6.2 is now required.
Is it possible that you actually require pytest 7 now?

add a suffix parameter in regression fixture check method

Most of the time when I need to use "basename parameter" I in fact just want to test 2 things in the same test (yes I know it's not best practice but I have some legit reasons).

Instead of writting down the whole basename I would like to keep the original one which is my test name and simply add a suffix to it.

something like:

def test_something(data_regression):
    data_regression.check(data1, suffix="data1")
    data_regression.check(data2, suffix="data2")

Which would create a test_something_data1.yml and a test_something_data2.yml files.
Do you think it would make sense to add it to the plugin ?

data_regression outputs and empty file if the data cannot be serialised.

If yaml encounters a value it cannot serialise it raises a yaml.representer.RepresenterError, but in that case an empty file is created. This is due to the yaml serialisation occurring during the with filename.open("wb") as f: block. Once the user has converted the data to a yaml serialisable type pytest-regressions tries to compare the correct output to an empty file, which of course differ.

I propose either:

  • moving the yaml serialisation before the with block, or
  • catching any exceptions, removing the empty file, and reraising the exception.

I think the first approach would be more robust, although there might be issues with large files.

Configure html diff output location

I've noticed that a nice html output is generated if in case the regression check fails. Is there any way of customizing where this output is saved to? At the moment it seems to be saved to a temporary directory.

Exception while comparing empty strings

Minimal, reproducible example:

def test_frame(dataframe_regression):
    df = pd.DataFrame.from_records([
        {
            'a': 'a',
            'b': 'b'
        },
        {
            'a': 'a1',
            'b': ''
        },
    ])

    dataframe_regression.check(df)

This test results in rising the following exception:
{TypeError}unsupported operand type(s) for -: 'str' and 'str'

The issue arises because the empty string is converted to nan while reading a csv file containing the dataframe.

Traceback (most recent call last):
  File ".../venv/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-b86ebfe209ab>", line 1, in <module>
    dataframe_regression.check(df)
  File ".../venv/lib/python3.7/site-packages/pytest_regressions/dataframe_regression.py", line 260, in check
    force_regen=self._force_regen,
  File ".../venv/lib/python3.7/site-packages/pytest_regressions/common.py", line 153, in perform_regression_check
    check_fn(obtained_filename, filename)
  File ".../venv/lib/python3.7/site-packages/pytest_regressions/dataframe_regression.py", line 143, in _check_fn
    diffs = np.abs(obtained_column - expected_column)[diff_ids]
  File ".../venv/lib/python3.7/site-packages/pandas/core/ops/common.py", line 65, in new_method
    return method(self, other)
  File ".../venv/lib/python3.7/site-packages/pandas/core/arraylike.py", line 97, in __sub__
    return self._arith_method(other, operator.sub)
  File ".../venv/lib/python3.7/site-packages/pandas/core/series.py", line 4998, in _arith_method
    result = ops.arithmetic_op(lvalues, rvalues, op)
  File ".../venv/lib/python3.7/site-packages/pandas/core/ops/array_ops.py", line 189, in arithmetic_op
    res_values = _na_arithmetic_op(lvalues, rvalues, op)
  File ".../venv/lib/python3.7/site-packages/pandas/core/ops/array_ops.py", line 149, in _na_arithmetic_op
    result = _masked_arith_op(left, right, op)
  File ".../venv/lib/python3.7/site-packages/pandas/core/ops/array_ops.py", line 91, in _masked_arith_op
    result[mask] = op(xrav[mask], yrav[mask])
TypeError: unsupported operand type(s) for -: 'str' and 'str'

would it be possible to export num_regression as yml files ?

It's not a matter of making the actual test, it's a matter of checking a problem if no arise.

I'm moving from data_regression to num_regression as the value I compare are all numbers and changes from time to time (at the 10th decimal so not relevant). the previously generated file was super easy to humanly review looking like this:

AOT: 72
B1: 238.7171666246534
B11: 948.6636123014868
B12: 750.7564658432063
B2: 320.9578522813208
B3: 415.14734055961685
B4: 426.16661204940726
B5: 604.8148600957898
B6: 1007.6004411393997
B7: 1140.246659944543
B8: 1217.7626543987888
B8A: 1200.12724981094
B9: 1389.4672800604985
MSK_CLDPRB: 0.27464078648853035
MSK_SNWPRB: 0
NDVI: 0.47685885754442053
QA10: 0
QA20: 0
QA60: 0
SCL: 3.324502142677085
TCI_B: 33.158558104360964
TCI_G: 42.684144189563895
TCI_R: 43.699306780942756
WVP: 681.9462692210739

Now the output is .csv and it's barely readable (from a human standpoint):

,AOT,B1,B11,B12,B2,B3,B4,B5,B6,B7,B8,B8A,B9,MSK_CLDPRB,MSK_SNWPRB,NDVI,QA10,QA20,QA60,SCL,TCI_B,TCI_G,TCI_R,WVP
0,72,238.7171666246534,948.66361230148675,750.75646584320634,320.95785228132081,415.14734055961685,426.16661204940726,604.81486009578975,1007.6004411393997,1140.2466599445429,1217.7626543987888,1200.1272498109399,1389.4672800604985,0.27464078648853035,0,0.47685885754442053,0,0,0,3.3245021426770851,33.158558104360964,42.684144189563895,43.699306780942756,681.94626922107386

I would like to get the best of 2 worlds, the display of an easy to read yaml file (data_regression) and the comparaison of numercial value (num_regression). Do you think it could be an added feature ?

either an opt-in file format in num_regression or a special handle of numercal values in data_regresion ?

Multiprocessing of regression test

I am wondering if there is a good way to run these regression tests in parallel using simple joblib applications.

Running the code below:

def main_test_runner_per_market(dataframe_regression, num_regression, data_regression):
      pass


markets = ["Market_1", "Market_2"]
sub_regression_tests = [
    delayed(main_test_runner_per_market)(dataframe_regression, num_regression, data_regression, market)
    for market in markets
]
Parallel(backend="multiprocessing", n_jobs=-1)(sub_regression_tests)

Returns :

  • " Can't pickle local object 'ArgumentParser.init..identity " error when running the following:

Support for pandas DataFrame

Support for DataFrames, out of the box:

def test_foo(dataframe_regression):
    ...
    dataframe_regression.check(df)

Check non-numeric columns with dataframe_regression

It would be very useful if the dataframe_regression fixture could check non-numeric columns in dataframes.

One simple work around is to use data_regression and convert the dataframe to a dictionary:

data_regression.check(df.to_dict("records"))

However, this does not allow application of tolerances to numerical values.

As a workaround, I am currently defining a fixture in my conftest.py that leverages data_regression for the non-numeric columns and dataframe_regression for the numeric columns (with tolerances):

# conftest.py

@pytest.fixture()
def check_df(dataframe_regression, data_regression):
    """Fixture to check dataframe against expected values leveraging pytest-regression
    dataframe_regression and data_regression fixtures. This fixture allows verification
    of non-numeric columns as well as application of tolerances on numeric columns."""

    def check(df, basename=None, fullpath=None, tolerances=None, default_tolerance=None):
        data_regression.check(
            df.select_dtypes(exclude="number").to_dict("records"),
            basename=basename,
            fullpath=fullpath,
        )
        dataframe_regression.check(
            df.select_dtypes(include="number"),
            basename=basename,
            fullpath=fullpath,
            tolerances=tolerances,
            default_tolerance=default_tolerance,
        )

    yield check

# test_something.py

def test_something(check_df):
    df = some_operation()

    check_df(df, default_tolerance=dict(atol=1e-8, rtol=1e-5)

While this works, it is less elegant and requires to be run twice to generate the yaml and csv files of expected results.

file_regression check method don't accept Pathlib.Path

I wanted to use the pytest-regressions lib to check GEE generated .pkl files but it seems the check method only accept str. As pathlib.Path offers lots of flexibility and cross-platform solution to file management issues, I'm using it systematically and I thing it an acceptable input for a file checking method.

Is it a wanted feature ?

Test of num_regression user message failing

tests/test_num_regression.py:test_common_cases currently fails when running from a virtual environment on Windows because it produces this message:

data1:
          obtained_data1         ...                          diff
500  1.20000000000000018         ...           0.10000000000000009

While the expected was this one:

data1:
          obtained_data1       expected_data1                 diff
500  1.20000000000000018  1.10000000000000009  0.10000000000000009

It seems there's some issue with how pandas formats the tables.

The weird thing is that the same test passes when run inside our conda environment.

The test has been marked as xfail for now.

RFE: is it possible to start making github releases?๐Ÿค”

Is it possible next time on release new version make the github release to have entry on https://github.com/ESSS/pytest-regressions/releases? ๐Ÿค”

I'm asking because only on make gh release is spread notification about new release to those who have set watch->releases.
My automation process those notification trying make preliminary automated upgrade of building packages which allow save some time on maintaining packaging procedures.

More about gh releases is possible to find on
https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository
https://github.com/marketplace/actions/github-release
https://pgjones.dev/blog/trusted-plublishing-2023/

The serialization of a DataFrame with floats automatically casts floats into integers

def _dump_fn

import pandas as pd

df = pd.Dataframe({
                        'col_1': [1.0, 2.0, 3.0], 
                        'col_2': [21.0, 31.0, 32.0],
                        'col_3': [101.0, 102.0, 103.0]
         })

df.to_csv("test.csv", float_format=f"%.17g")       

df
#   col_1  col_2  col_3
# 0    1.0   21.0  101.0
# 1    2.0   31.0  102.0
# 2    3.0   32.0  103.0

df.dtypes
# col_1    float64
# col_2    float64
# col_3    float64
# dtype: object

The serialized csv:

,col_1,col_2,col_3
0,1,21,101
1,2,31,102
2,3,32,103

https://docs.python.org/3/library/string.html#format-specification-mini-language

According to the python docs the presentation type 'g' to format strings as a flaw.

... In both cases insignificant trailing zeros are removed from the significand, and the decimal point is also removed if there are no remaining digits following it, unless the '#' option is used. ...

Any particular reason why the scientific notation is not used? 'e'

Scientific notation. For a given precision p, formats the number in scientific notation with the letter โ€˜eโ€™ separating the coefficient from the exponent. The coefficient has one digit before and p digits after the decimal point, for a total of p + 1 significant digits. With no precision given, uses a precision of 6 digits after the decimal point for float, and shows all coefficient digits for Decimal. If no digits follow the decimal point, the decimal point is also removed unless the # option is used.

Improve `num_regression` with changes in percent

Add relative change, use sign on errors

The first idea would be to add a fourth column besides obtained_..., expected_... and diff: relative change. It would print the relative error of the obtained, in relation with the expected. It would also change to use a sign on the error.

Example:

    obtained_U_gas [m/s]  expected_U_gas [m/s]              abs. diff         rel. diff (%)
E       1   0.00000526006928701   0.00092027745031029  -0.000915017381023 -99.428425711698500
E       2   0.00109742144185932   0.05314025093428264  -0.052042829492423 -97.934858374650000
E       3   0.10233425003713974   0.83767265449607820  -0.735338404458939 -87.783503557400900
E       4   1.50587844448218111   2.46949341632139818  -0.963614971839210 -39.020754842692900
E       5   3.85292939530965661   4.17226524036200175  -0.319335845052350  -7.653776225996680
E       6   6.48537098006507318   6.44999832744570334   0.035372652619370   0.548413361114436
E       7  10.33851199361759932   9.96422211906309130   0.374289874554410   3.756338127372090
E       8  15.91086496866478051  15.83790607275804696   0.072958895906700   0.460659986058339
E       9  28.43953100878306728  28.31184178336182455   0.127689225421200   0.451009956887510

Note: using rel_error_in_percent = (obtained - expected) / abs(expected) * 100. There may be other better equations.

How it is currently:

E          obtained_U_gas [m/s]  expected_U_gas [m/s]                 diff
E       1   0.00000526006928701   0.00092027745031029  0.00091501738102328
E       2   0.00109742144185932   0.05314025093428264  0.05204282949242332
E       3   0.10233425003713974   0.83767265449607820  0.73533840445893850
E       4   1.50587844448218111   2.46949341632139818  0.96361497183921707
E       5   3.85292939530965661   4.17226524036200175  0.31933584505234514
E       6   6.48537098006507318   6.44999832744570334  0.03537265261936984
E       7  10.33851199361759932   9.96422211906309130  0.37428987455450802
E       8  15.91086496866478051  15.83790607275804696  0.07295889590673355
E       9  28.43953100878306728  28.31184178336182455  0.12768922542124272

Add changes on statistical properties

Sometimes it maybe useful to only look at variations in statistical properties, so this could be added to the output:

    obtained_U_gas [m/s]  expected_U_gas [m/s]              abs. diff         rel. diff (%)
E       1   0.00000526006928701   0.00092027745031029  -0.000915017381023 -99.428425711698500
(...)
E       9  28.43953100878306728  28.31184178336182455   0.127689225421200   0.451009956887510
E     min   0.000005260069287     0.000920277450310    -0.000915017381023 -99.428425711698500
E     max  28.439531008783000    28.311841783361800     0.127689225421200   0.451009956887510
E     avg   7.404058191385600     7.566384460243630    -0.162326268858032  -2.145361099623600
E     std   9.582861811894490     9.369144876763070     0.213716935131419   2.281071943518240

I'll mention @tadeu here since he's the original author of this request ๐Ÿ˜ƒ

delete files that are no longer needed

It would be supernice, if there was a way to automatically remove files (and directories) that are leftover from tests that do not exist anymore.
Tests are renamed regularly and after that you (or at least I) have a lot of leftover datafiles that I need to remove manually.

Test failure in test_different_data_types on s390x

hi,

as part of a recent update of the Debian package of pytest-regressions, the testsuite is now being run as part of the Debian CI. Unfortunately, there's a test failure on s390x only. Tests were fine on all other hardware architectures for which the CI is enabled (amd64, arm64, armel, armhf, i386, ppc64el).

IIRC s390x is the only big-endian architecture of the bunch, and that <i8 is the equivalent of (little-endian) int64. I suspect something is trying to be very explicit about the endianness on that platform, which then causes the output to not match the notation expected by the test.

Log excerpt:

107s ============================= test session starts ==============================
107s platform linux -- Python 3.11.7, pytest-7.4.3, pluggy-1.3.0
107s rootdir: /tmp/autopkgtest-lxc.fvwen33l/downtmp/autopkgtest_tmp/build
107s plugins: datadir-1.4.1+ds, regressions-2.5.0+ds
107s collected 72 items
107s 
107s tests/test_data_regression.py ........                                   [ 11%]
107s tests/test_dataframe_regression.py ..............                        [ 30%]
107s tests/test_file_regression.py ....                                       [ 36%]
107s tests/test_filenames.py ...                                              [ 40%]
107s tests/test_grids.py ..                                                   [ 43%]
108s tests/test_image_regression.py ..                                        [ 45%]
108s tests/test_ndarrays_regression.py ....F.................                 [ 76%]
108s tests/test_num_regression.py .................                           [100%]
108s 
108s =================================== FAILURES ===================================
108s __________________________ test_different_data_types ___________________________
108s 
108s ndarrays_regression = <pytest_regressions.ndarrays_regression.NDArraysRegressionFixture object at 0x3ff29a06b90>
108s no_regen = None
108s 
108s     def test_different_data_types(ndarrays_regression, no_regen):
108s         # Generate data with integer array.
108s         data = {"data1": np.array([1] * 10)}
108s         ndarrays_regression.check(data)
108s     
108s         # Run check with incompatible type.
108s         data = {"data1": np.array([True] * 10)}
108s         with pytest.raises(AssertionError) as excinfo:
108s             ndarrays_regression.check(data)
108s         obtained_error_msg = str(excinfo.value)
108s         expected = "\n".join(
108s             [
108s                 "Data types are not the same.",
108s                 "  key: data1",
108s                 "  Obtained: bool",
108s                 "  Expected: int64",
108s             ]
108s         )
108s >       assert expected in obtained_error_msg
108s E       AssertionError: assert 'Data types are not the same.\n  key: data1\n  Obtained: bool\n  Expected: int64' in 'Data types are not the same.\n  key: data1\n  Obtained: bool\n  Expected: <i8\n'
108s 
108s tests/test_ndarrays_regression.py:297: AssertionError
108s =========================== short test summary info ============================
108s FAILED tests/test_ndarrays_regression.py::test_different_data_types - Asserti...
108s ========================= 1 failed, 71 passed in 2.33s =========================

Full logs available via the following links for both Debian Ubuntu.

2.2.0: pytest is failing

Please ignore that unit which is failing because missing matplotlib
Just normal build, install and test cycle used on building package from non-root account:

  • "setup.py build"
  • "setup.py install --root </install/prefix>"
  • "pytest with PYTHONPATH pointing to setearch and sitelib inside </install/prefix>
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-regressions-2.2.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-regressions-2.2.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=2227736132
rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-regressions-2.2.0, configfile: tox.ini
plugins: regressions-2.2.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, pyfakefs-4.5.0, cases-3.6.1, flaky-3.7.0, hypothesis-6.14.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, randomly-3.8.0, Faker-8.8.2, datadir-1.3.1
collected 44 items

tests/test_grids.py ..                                                                                                                                               [  4%]
tests/test_image_regression.py F.                                                                                                                                    [  9%]
tests/test_num_regression.py ................                                                                                                                        [ 45%]
tests/test_file_regression.py ....                                                                                                                                   [ 54%]
. F                                                                                                                                                                  [ 56%]
tests/test_data_regression.py .......                                                                                                                                [ 72%]
tests/test_dataframe_regression.py ......F...FF                                                                                                                      [100%]

================================================================================= FAILURES =================================================================================
__________________________________________________________________________ test_image_regression ___________________________________________________________________________

image_regression = <pytest_regressions.image_regression.ImageRegressionFixture object at 0x7f78e2fd5880>
datadir = PosixPath('/tmp/pytest-of-tkloczko/pytest-204/test_image_regression0/test_image_regression')

    def test_image_regression(image_regression, datadir):
>       import matplotlib
E       ModuleNotFoundError: No module named 'matplotlib'

tests/test_image_regression.py:9: ModuleNotFoundError
_______________________________________________________________________________ test session _______________________________________________________________________________

cls = <class '_pytest.runner.CallInfo'>, func = <function call_runtest_hook.<locals>.<lambda> at 0x7f78e28fd940>, when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

/usr/lib/python3.8/site-packages/_pytest/runner.py:311:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>       lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    )

/usr/lib/python3.8/site-packages/_pytest/runner.py:255:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_HookCaller 'pytest_runtest_call'>, args = (), kwargs = {'item': <CheckdocsItem project>}, notincall = set()

    def __call__(self, *args, **kwargs):
        if args:
            raise TypeError("hook calling supports only keyword arguments")
        assert not self.is_historic()
        if self.spec and self.spec.argnames:
            notincall = (
                set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
            )
            if notincall:
                warnings.warn(
                    "Argument(s) {} which are declared in the hookspec "
                    "can not be found in this hook call".format(tuple(notincall)),
                    stacklevel=2,
                )
>       return self._hookexec(self, self.get_hookimpls(), kwargs)

/usr/lib/python3.8/site-packages/pluggy/hooks.py:286:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest.config.PytestPluginManager object at 0x7f78eab62f70>, hook = <_HookCaller 'pytest_runtest_call'>
methods = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/usr/lib/python3.8/site-packages/_pytest/runner...pper name='/dev/null' mode='r' encoding='UTF-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>>, ...]
kwargs = {'item': <CheckdocsItem project>}

    def _hookexec(self, hook, methods, kwargs):
        # called from all hookcaller instances.
        # enable_tracing will set its own wrapping function at self._inner_hookexec
>       return self._inner_hookexec(hook, methods, kwargs)

/usr/lib/python3.8/site-packages/pluggy/manager.py:93:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

hook = <_HookCaller 'pytest_runtest_call'>
methods = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/usr/lib/python3.8/site-packages/_pytest/runner...pper name='/dev/null' mode='r' encoding='UTF-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>>, ...]
kwargs = {'item': <CheckdocsItem project>}

>   self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
        methods,
        kwargs,
        firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    )

/usr/lib/python3.8/site-packages/pluggy/manager.py:84:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

hook_impls = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/usr/lib/python3.8/site-packages/_pytest/runner...pper name='/dev/null' mode='r' encoding='UTF-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>>, ...]
caller_kwargs = {'item': <CheckdocsItem project>}, firstresult = False

    def _multicall(hook_impls, caller_kwargs, firstresult=False):
        """Execute a call into multiple python functions/methods and return the
        result(s).

        ``caller_kwargs`` comes from _HookCaller.__call__().
        """
        __tracebackhide__ = True
        results = []
        excinfo = None
        try:  # run impl and wrapper setup functions in a loop
            teardowns = []
            try:
                for hook_impl in reversed(hook_impls):
                    try:
                        args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                    except KeyError:
                        for argname in hook_impl.argnames:
                            if argname not in caller_kwargs:
                                raise HookCallError(
                                    "hook call must provide argument %r" % (argname,)
                                )

                    if hook_impl.hookwrapper:
                        try:
                            gen = hook_impl.function(*args)
                            next(gen)  # first yield
                            teardowns.append(gen)
                        except StopIteration:
                            _raise_wrapfail(gen, "did not yield")
                    else:
                        res = hook_impl.function(*args)
                        if res is not None:
                            results.append(res)
                            if firstresult:  # halt further impl calls
                                break
            except BaseException:
                excinfo = sys.exc_info()
        finally:
            if firstresult:  # first result hooks return a single value
                outcome = _Result(results[0] if results else None, excinfo)
            else:
                outcome = _Result(results, excinfo)

            # run all wrapper post-yield blocks
            for gen in reversed(teardowns):
                try:
                    gen.send(outcome)
                    _raise_wrapfail(gen, "has second yield")
                except StopIteration:
                    pass

>           return outcome.get_result()

/usr/lib/python3.8/site-packages/pluggy/callers.py:208:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pluggy.callers._Result object at 0x7f78e263f730>

    def get_result(self):
        """Get the result(s) for this hook call.

        If the hook was marked as a ``firstresult`` only a single value
        will be returned otherwise a list of results.
        """
        __tracebackhide__ = True
        if self._excinfo is None:
            return self._result
        else:
            ex = self._excinfo
            if _py3:
>               raise ex[1].with_traceback(ex[2])

/usr/lib/python3.8/site-packages/pluggy/callers.py:80:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

hook_impls = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/usr/lib/python3.8/site-packages/_pytest/runner...pper name='/dev/null' mode='r' encoding='UTF-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>>, ...]
caller_kwargs = {'item': <CheckdocsItem project>}, firstresult = False

    def _multicall(hook_impls, caller_kwargs, firstresult=False):
        """Execute a call into multiple python functions/methods and return the
        result(s).

        ``caller_kwargs`` comes from _HookCaller.__call__().
        """
        __tracebackhide__ = True
        results = []
        excinfo = None
        try:  # run impl and wrapper setup functions in a loop
            teardowns = []
            try:
                for hook_impl in reversed(hook_impls):
                    try:
                        args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                    except KeyError:
                        for argname in hook_impl.argnames:
                            if argname not in caller_kwargs:
                                raise HookCallError(
                                    "hook call must provide argument %r" % (argname,)
                                )

                    if hook_impl.hookwrapper:
                        try:
                            gen = hook_impl.function(*args)
                            next(gen)  # first yield
                            teardowns.append(gen)
                        except StopIteration:
                            _raise_wrapfail(gen, "did not yield")
                    else:
>                       res = hook_impl.function(*args)

/usr/lib/python3.8/site-packages/pluggy/callers.py:187:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

item = <CheckdocsItem project>

    def pytest_runtest_call(item: Item) -> None:
        _update_current_test_var(item, "call")
        try:
            del sys.last_type
            del sys.last_value
            del sys.last_traceback
        except AttributeError:
            pass
        try:
            item.runtest()
        except Exception as e:
            # Store trace info to allow postmortem debugging
            sys.last_type = type(e)
            sys.last_value = e
            assert e.__traceback__ is not None
            # Skip *this* frame
            sys.last_traceback = e.__traceback__.tb_next
>           raise e

/usr/lib/python3.8/site-packages/_pytest/runner.py:170:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

item = <CheckdocsItem project>

    def pytest_runtest_call(item: Item) -> None:
        _update_current_test_var(item, "call")
        try:
            del sys.last_type
            del sys.last_value
            del sys.last_traceback
        except AttributeError:
            pass
        try:
>           item.runtest()

/usr/lib/python3.8/site-packages/_pytest/runner.py:162:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <CheckdocsItem project>

    def runtest(self):
>       desc = self.get_long_description()

/usr/lib/python3.8/site-packages/pytest_checkdocs/__init__.py:29:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <CheckdocsItem project>

    def get_long_description(self):
>       return Description.from_md(ensure_clean(pep517.meta.load('.').metadata))

/usr/lib/python3.8/site-packages/pytest_checkdocs/__init__.py:60:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

root = '.'

    def load(root):
        """
        Given a source directory (root) of a package,
        return an importlib.metadata.Distribution object
        with metadata build from that package.
        """
        root = os.path.expanduser(root)
        system = compat_system(root)
        builder = functools.partial(build, source_dir=root, system=system)
>       path = Path(build_as_zip(builder))

/usr/lib/python3.8/site-packages/pep517/meta.py:71:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

builder = functools.partial(<function build at 0x7f78e7a0adc0>, source_dir='.', system={'build-backend': 'setuptools.build_meta:__legacy__', 'requires': ['setuptools', 'wheel']})

    def build_as_zip(builder=build):
        with tempdir() as out_dir:
>           builder(dest=out_dir)

/usr/lib/python3.8/site-packages/pep517/meta.py:58:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

source_dir = '.', dest = '/tmp/tmppqylg39m', system = {'build-backend': 'setuptools.build_meta:__legacy__', 'requires': ['setuptools', 'wheel']}

    def build(source_dir='.', dest=None, system=None):
        system = system or load_system(source_dir)
        dest = os.path.join(source_dir, dest or 'dist')
        mkdir_p(dest)
        validate_system(system)
        hooks = Pep517HookCaller(
            source_dir, system['build-backend'], system.get('backend-path')
        )

        with hooks.subprocess_runner(quiet_subprocess_runner):
            with BuildEnvironment() as env:
                env.pip_install(system['requires'])
>               _prep_meta(hooks, env, dest)

/usr/lib/python3.8/site-packages/pep517/meta.py:53:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

hooks = <pep517.wrappers.Pep517HookCaller object at 0x7f78e263f7f0>, env = <pep517.envbuild.BuildEnvironment object at 0x7f78e263f610>, dest = '/tmp/tmppqylg39m'

    def _prep_meta(hooks, env, dest):
        reqs = hooks.get_requires_for_build_wheel({})
        log.info('Got build requires: %s', reqs)

        env.pip_install(reqs)
        log.info('Installed dynamic build dependencies')

        with tempdir() as td:
            log.info('Trying to build metadata in %s', td)
>           filename = hooks.prepare_metadata_for_build_wheel(td, {})

/usr/lib/python3.8/site-packages/pep517/meta.py:36:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pep517.wrappers.Pep517HookCaller object at 0x7f78e263f7f0>, metadata_directory = '/tmp/tmpruv20d3u', config_settings = {}, _allow_fallback = True

    def prepare_metadata_for_build_wheel(
            self, metadata_directory, config_settings=None,
            _allow_fallback=True):
        """Prepare a ``*.dist-info`` folder with metadata for this project.

        Returns the name of the newly created folder.

        If the build backend defines a hook with this name, it will be called
        in a subprocess. If not, the backend will be asked to build a wheel,
        and the dist-info extracted from that (unless _allow_fallback is
        False).
        """
>       return self._call_hook('prepare_metadata_for_build_wheel', {
            'metadata_directory': abspath(metadata_directory),
            'config_settings': config_settings,
            '_allow_fallback': _allow_fallback,
        })

/usr/lib/python3.8/site-packages/pep517/wrappers.py:184:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pep517.wrappers.Pep517HookCaller object at 0x7f78e263f7f0>, hook_name = 'prepare_metadata_for_build_wheel'
kwargs = {'_allow_fallback': True, 'config_settings': {}, 'metadata_directory': '/tmp/tmpruv20d3u'}

    def _call_hook(self, hook_name, kwargs):
        # On Python 2, pytoml returns Unicode values (which is correct) but the
        # environment passed to check_call needs to contain string values. We
        # convert here by encoding using ASCII (the backend can only contain
        # letters, digits and _, . and : characters, and will be used as a
        # Python identifier, so non-ASCII content is wrong on Python 2 in
        # any case).
        # For backend_path, we use sys.getfilesystemencoding.
        if sys.version_info[0] == 2:
            build_backend = self.build_backend.encode('ASCII')
        else:
            build_backend = self.build_backend
        extra_environ = {'PEP517_BUILD_BACKEND': build_backend}

        if self.backend_path:
            backend_path = os.pathsep.join(self.backend_path)
            if sys.version_info[0] == 2:
                backend_path = backend_path.encode(sys.getfilesystemencoding())
            extra_environ['PEP517_BACKEND_PATH'] = backend_path

        with tempdir() as td:
            hook_input = {'kwargs': kwargs}
            compat.write_json(hook_input, pjoin(td, 'input.json'),
                              indent=2)

            # Run the hook in a subprocess
            with _in_proc_script_path() as script:
                python = self.python_executable
>               self._subprocess_runner(
                    [python, abspath(str(script)), hook_name, td],
                    cwd=self.source_dir,
                    extra_environ=extra_environ
                )

/usr/lib/python3.8/site-packages/pep517/wrappers.py:265:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = ['/usr/bin/python3', '/usr/lib/python3.8/site-packages/pep517/in_process/_in_process.py', 'prepare_metadata_for_build_wheel', '/tmp/tmps1lr660y']
cwd = '/home/tkloczko/rpmbuild/BUILD/pytest-regressions-2.2.0', extra_environ = {'PEP517_BUILD_BACKEND': 'setuptools.build_meta:__legacy__'}

    def quiet_subprocess_runner(cmd, cwd=None, extra_environ=None):
        """A method of calling the wrapper subprocess while suppressing output."""
        env = os.environ.copy()
        if extra_environ:
            env.update(extra_environ)

>       check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)

/usr/lib/python3.8/site-packages/pep517/wrappers.py:75:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

timeout = None
popenargs = (['/usr/bin/python3', '/usr/lib/python3.8/site-packages/pep517/in_process/_in_process.py', 'prepare_metadata_for_build_wheel', '/tmp/tmps1lr660y'],)
kwargs = {'cwd': '/home/tkloczko/rpmbuild/BUILD/pytest-regressions-2.2.0', 'env': {'AR': '/usr/bin/gcc-ar', 'BASH_FUNC_which%%'...sh-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none', ...}, 'stderr': -2}

    def check_output(*popenargs, timeout=None, **kwargs):
        r"""Run command with arguments and return its output.

        If the exit code was non-zero it raises a CalledProcessError.  The
        CalledProcessError object will have the return code in the returncode
        attribute and output in the output attribute.

        The arguments are the same as for the Popen constructor.  Example:

        >>> check_output(["ls", "-l", "/dev/null"])
        b'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

        The stdout argument is not allowed as it is used internally.
        To capture standard error in the result, use stderr=STDOUT.

        >>> check_output(["/bin/sh", "-c",
        ...               "ls -l non_existent_file ; exit 0"],
        ...              stderr=STDOUT)
        b'ls: non_existent_file: No such file or directory\n'

        There is an additional optional argument, "input", allowing you to
        pass a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it too will be used internally.  Example:

        >>> check_output(["sed", "-e", "s/foo/bar/"],
        ...              input=b"when in the course of fooman events\n")
        b'when in the course of barman events\n'

        By default, all communication is in bytes, and therefore any "input"
        should be bytes, and the return value will be bytes.  If in text mode,
        any "input" should be a string, and the return value will be a string
        decoded according to locale encoding, or by "encoding" if set. Text mode
        is triggered by setting any of text, encoding, errors or universal_newlines.
        """
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')

        if 'input' in kwargs and kwargs['input'] is None:
            # Explicitly passing input=None was previously equivalent to passing an
            # empty string. That is maintained here for backwards compatibility.
            if kwargs.get('universal_newlines') or kwargs.get('text'):
                empty = ''
            else:
                empty = b''
            kwargs['input'] = empty

>       return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                   **kwargs).stdout

/usr/lib64/python3.8/subprocess.py:415:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3', '/usr/lib/python3.8/site-packages/pep517/in_process/_in_process.py', 'prepare_metadata_for_build_wheel', '/tmp/tmps1lr660y'],)
kwargs = {'cwd': '/home/tkloczko/rpmbuild/BUILD/pytest-regressions-2.2.0', 'env': {'AR': '/usr/bin/gcc-ar', 'BASH_FUNC_which%%'...-fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none', ...}, 'stderr': -2, 'stdout': -1}
process = <subprocess.Popen object at 0x7f78e2554c10>
stdout = b'Traceback (most recent call last):\n  File "/usr/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line...ng pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj\n'
stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.

        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['/usr/bin/python3', '/usr/lib/python3.8/site-packages/pep517/in_process/_in_process.py', 'prepare_metadata_for_build_wheel', '/tmp/tmps1lr660y']' returned non-zero exit status 1.

/usr/lib64/python3.8/subprocess.py:516: CalledProcessError
______________________________________________________________________ test_non_numeric_data[array1] _______________________________________________________________________

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f78c97d2ee0>
array = [<test_dataframe_regression.Foo object at 0x7f78e354b520>, <test_dataframe_regression.Foo object at 0x7f78e3547100>, <test_dataframe_regression.Foo object at 0x7f78e35474f0>, <test_dataframe_regression.Foo object at 0x7f78e3547880>]
no_regen = None

    @pytest.mark.parametrize(
        "array", [[np.random.randint(10, 99, 6)] * 6, [Foo(i) for i in range(4)]]
    )
    def test_non_numeric_data(dataframe_regression, array, no_regen):
        data1 = pd.DataFrame()
        data1["data1"] = array
        with pytest.raises(
            AssertionError,
            match="Only numeric data is supported on dataframe_regression fixture.\n"
            "  Array with type '%s' was given." % (str(data1["data1"].dtype),),
        ):
>           dataframe_regression.check(data1)

tests/test_dataframe_regression.py:184:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f78c97d2ee0>
data_frame =                                                data1
0  <test_dataframe_regression.Foo object at 0x7f7...
1  <test_dat...t at 0x7f7...
2  <test_dataframe_regression.Foo object at 0x7f7...
3  <test_dataframe_regression.Foo object at 0x7f7...
basename = None, fullpath = None, tolerances = None, default_tolerance = None

    def check(
        self,
        data_frame,
        basename=None,
        fullpath=None,
        tolerances=None,
        default_tolerance=None,
    ):
        """
        Checks the given pandas dataframe against a previously recorded version, or generate a new file.

        Example::

            data_frame = pandas.DataFrame.from_dict({
                'U_gas': U[0][positions],
                'U_liquid': U[1][positions],
                'gas_vol_frac [-]': vol_frac[0][positions],
                'liquid_vol_frac [-]': vol_frac[1][positions],
                'P': Pa_to_bar(P)[positions],
            })
            dataframe_regression.check(data_frame)

        :param pandas.DataFrame data_frame: pandas DataFrame containing data for regression check.

        :param str basename: basename of the file to test/record. If not given the name
            of the test is used.

        :param str fullpath: complete path to use as a reference file. This option
            will ignore embed_data completely, being useful if a reference file is located
            in the session data dir for example.

        :param dict tolerances: dict mapping keys from the data_dict to tolerance settings for the
            given data. Example::

                tolerances={'U': Tolerance(atol=1e-2)}

        :param dict default_tolerance: dict mapping the default tolerance for the current check
            call. Example::

                default_tolerance=dict(atol=1e-7, rtol=1e-18).

            If not provided, will use defaults from numpy's ``isclose`` function.

        ``basename`` and ``fullpath`` are exclusive.
        """
        try:
            import pandas as pd
        except ModuleNotFoundError:
            raise ModuleNotFoundError(import_error_message("Pandas"))

        import functools

        __tracebackhide__ = True

        assert type(data_frame) is pd.DataFrame, (
            "Only pandas DataFrames are supported on on dataframe_regression fixture.\n"
            "Object with type '%s' was given." % (str(type(data_frame)),)
        )

        for column in data_frame.columns:
            array = data_frame[column]
            # Skip assertion if an array of strings
            if (array.dtype == "O") and (type(array[0]) is str):
                continue
            # Rejected: timedelta, datetime, objects, zero-terminated bytes, unicode strings and raw data
>           assert array.dtype not in ["m", "M", "O", "S", "a", "U", "V"], (
                "Only numeric data is supported on dataframe_regression fixture.\n"
                "Array with type '%s' was given.\n" % (str(array.dtype),)
            )
E           AssertionError: Only numeric data is supported on dataframe_regression fixture.
E           Array with type 'object' was given.

../../BUILDROOT/python-pytest-regressions-2.2.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/pytest_regressions/dataframe_regression.py:235: AssertionError

During handling of the above exception, another exception occurred:

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f78c97d2ee0>
array = [<test_dataframe_regression.Foo object at 0x7f78e354b520>, <test_dataframe_regression.Foo object at 0x7f78e3547100>, <test_dataframe_regression.Foo object at 0x7f78e35474f0>, <test_dataframe_regression.Foo object at 0x7f78e3547880>]
no_regen = None

    @pytest.mark.parametrize(
        "array", [[np.random.randint(10, 99, 6)] * 6, [Foo(i) for i in range(4)]]
    )
    def test_non_numeric_data(dataframe_regression, array, no_regen):
        data1 = pd.DataFrame()
        data1["data1"] = array
        with pytest.raises(
            AssertionError,
            match="Only numeric data is supported on dataframe_regression fixture.\n"
            "  Array with type '%s' was given." % (str(data1["data1"].dtype),),
        ):
>           dataframe_regression.check(data1)
E           AssertionError: Regex pattern "Only numeric data is supported on dataframe_regression fixture.\n  Array with type 'object' was given." does not match "Only numeric data is supported on dataframe_regression fixture.\nArray with type 'object' was given.\n".

tests/test_dataframe_regression.py:184: AssertionError
______________________________________________________________________ test_non_numeric_data[array0] _______________________________________________________________________

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f78c971b850>
array = [array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91])]
no_regen = None

    @pytest.mark.parametrize(
        "array", [[np.random.randint(10, 99, 6)] * 6, [Foo(i) for i in range(4)]]
    )
    def test_non_numeric_data(dataframe_regression, array, no_regen):
        data1 = pd.DataFrame()
        data1["data1"] = array
        with pytest.raises(
            AssertionError,
            match="Only numeric data is supported on dataframe_regression fixture.\n"
            "  Array with type '%s' was given." % (str(data1["data1"].dtype),),
        ):
>           dataframe_regression.check(data1)

tests/test_dataframe_regression.py:184:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f78c971b850>
data_frame =                       data1
0  [20, 18, 37, 32, 74, 91]
1  [20, 18, 37, 32, 74, 91]
2  [20, 18, 37, 32, 74, 91]
3  [20, 18, 37, 32, 74, 91]
4  [20, 18, 37, 32, 74, 91]
5  [20, 18, 37, 32, 74, 91]
basename = None, fullpath = None, tolerances = None, default_tolerance = None

    def check(
        self,
        data_frame,
        basename=None,
        fullpath=None,
        tolerances=None,
        default_tolerance=None,
    ):
        """
        Checks the given pandas dataframe against a previously recorded version, or generate a new file.

        Example::

            data_frame = pandas.DataFrame.from_dict({
                'U_gas': U[0][positions],
                'U_liquid': U[1][positions],
                'gas_vol_frac [-]': vol_frac[0][positions],
                'liquid_vol_frac [-]': vol_frac[1][positions],
                'P': Pa_to_bar(P)[positions],
            })
            dataframe_regression.check(data_frame)

        :param pandas.DataFrame data_frame: pandas DataFrame containing data for regression check.

        :param str basename: basename of the file to test/record. If not given the name
            of the test is used.

        :param str fullpath: complete path to use as a reference file. This option
            will ignore embed_data completely, being useful if a reference file is located
            in the session data dir for example.

        :param dict tolerances: dict mapping keys from the data_dict to tolerance settings for the
            given data. Example::

                tolerances={'U': Tolerance(atol=1e-2)}

        :param dict default_tolerance: dict mapping the default tolerance for the current check
            call. Example::

                default_tolerance=dict(atol=1e-7, rtol=1e-18).

            If not provided, will use defaults from numpy's ``isclose`` function.

        ``basename`` and ``fullpath`` are exclusive.
        """
        try:
            import pandas as pd
        except ModuleNotFoundError:
            raise ModuleNotFoundError(import_error_message("Pandas"))

        import functools

        __tracebackhide__ = True

        assert type(data_frame) is pd.DataFrame, (
            "Only pandas DataFrames are supported on on dataframe_regression fixture.\n"
            "Object with type '%s' was given." % (str(type(data_frame)),)
        )

        for column in data_frame.columns:
            array = data_frame[column]
            # Skip assertion if an array of strings
            if (array.dtype == "O") and (type(array[0]) is str):
                continue
            # Rejected: timedelta, datetime, objects, zero-terminated bytes, unicode strings and raw data
>           assert array.dtype not in ["m", "M", "O", "S", "a", "U", "V"], (
                "Only numeric data is supported on dataframe_regression fixture.\n"
                "Array with type '%s' was given.\n" % (str(array.dtype),)
            )
E           AssertionError: Only numeric data is supported on dataframe_regression fixture.
E           Array with type 'object' was given.

../../BUILDROOT/python-pytest-regressions-2.2.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/pytest_regressions/dataframe_regression.py:235: AssertionError

During handling of the above exception, another exception occurred:

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f78c971b850>
array = [array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91]), array([20, 18, 37, 32, 74, 91])]
no_regen = None

    @pytest.mark.parametrize(
        "array", [[np.random.randint(10, 99, 6)] * 6, [Foo(i) for i in range(4)]]
    )
    def test_non_numeric_data(dataframe_regression, array, no_regen):
        data1 = pd.DataFrame()
        data1["data1"] = array
        with pytest.raises(
            AssertionError,
            match="Only numeric data is supported on dataframe_regression fixture.\n"
            "  Array with type '%s' was given." % (str(data1["data1"].dtype),),
        ):
>           dataframe_regression.check(data1)
E           AssertionError: Regex pattern "Only numeric data is supported on dataframe_regression fixture.\n  Array with type 'object' was given." does not match "Only numeric data is supported on dataframe_regression fixture.\nArray with type 'object' was given.\n".

tests/test_dataframe_regression.py:184: AssertionError
________________________________________________________________________ test_non_pandas_dataframe _________________________________________________________________________

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f78c970f880>

    def test_non_pandas_dataframe(dataframe_regression):
        data = np.ones(shape=(10, 10))
        with pytest.raises(
            AssertionError,
            match="Only pandas DataFrames are supported on on dataframe_regression fixture.\n"
            "  Object with type '%s' was given." % (str(type(data)),),
        ):
>           dataframe_regression.check(data)

tests/test_dataframe_regression.py:249:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f78c970f880>
data_frame = array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., ...1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])
basename = None, fullpath = None, tolerances = None, default_tolerance = None

    def check(
        self,
        data_frame,
        basename=None,
        fullpath=None,
        tolerances=None,
        default_tolerance=None,
    ):
        """
        Checks the given pandas dataframe against a previously recorded version, or generate a new file.

        Example::

            data_frame = pandas.DataFrame.from_dict({
                'U_gas': U[0][positions],
                'U_liquid': U[1][positions],
                'gas_vol_frac [-]': vol_frac[0][positions],
                'liquid_vol_frac [-]': vol_frac[1][positions],
                'P': Pa_to_bar(P)[positions],
            })
            dataframe_regression.check(data_frame)

        :param pandas.DataFrame data_frame: pandas DataFrame containing data for regression check.

        :param str basename: basename of the file to test/record. If not given the name
            of the test is used.

        :param str fullpath: complete path to use as a reference file. This option
            will ignore embed_data completely, being useful if a reference file is located
            in the session data dir for example.

        :param dict tolerances: dict mapping keys from the data_dict to tolerance settings for the
            given data. Example::

                tolerances={'U': Tolerance(atol=1e-2)}

        :param dict default_tolerance: dict mapping the default tolerance for the current check
            call. Example::

                default_tolerance=dict(atol=1e-7, rtol=1e-18).

            If not provided, will use defaults from numpy's ``isclose`` function.

        ``basename`` and ``fullpath`` are exclusive.
        """
        try:
            import pandas as pd
        except ModuleNotFoundError:
            raise ModuleNotFoundError(import_error_message("Pandas"))

        import functools

        __tracebackhide__ = True

>       assert type(data_frame) is pd.DataFrame, (
            "Only pandas DataFrames are supported on on dataframe_regression fixture.\n"
            "Object with type '%s' was given." % (str(type(data_frame)),)
        )
E       AssertionError: Only pandas DataFrames are supported on on dataframe_regression fixture.
E       Object with type '<class 'numpy.ndarray'>' was given.

../../BUILDROOT/python-pytest-regressions-2.2.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/pytest_regressions/dataframe_regression.py:224: AssertionError

During handling of the above exception, another exception occurred:

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f78c970f880>

    def test_non_pandas_dataframe(dataframe_regression):
        data = np.ones(shape=(10, 10))
        with pytest.raises(
            AssertionError,
            match="Only pandas DataFrames are supported on on dataframe_regression fixture.\n"
            "  Object with type '%s' was given." % (str(type(data)),),
        ):
>           dataframe_regression.check(data)
E           AssertionError: Regex pattern "Only pandas DataFrames are supported on on dataframe_regression fixture.\n  Object with type '<class 'numpy.ndarray'>' was given." does not match "Only pandas DataFrames are supported on on dataframe_regression fixture.\nObject with type '<class 'numpy.ndarray'>' was given.".

tests/test_dataframe_regression.py:249: AssertionError
========================================================================= short test summary info ==========================================================================
FAILED tests/test_image_regression.py::test_image_regression - ModuleNotFoundError: No module named 'matplotlib'
FAILED ::project - subprocess.CalledProcessError: Command '['/usr/bin/python3', '/usr/lib/python3.8/site-packages/pep517/in_process/_in_process.py', 'prepare_metadata_fo...
FAILED tests/test_dataframe_regression.py::test_non_numeric_data[array1] - AssertionError: Regex pattern "Only numeric data is supported on dataframe_regression fixture....
FAILED tests/test_dataframe_regression.py::test_non_numeric_data[array0] - AssertionError: Regex pattern "Only numeric data is supported on dataframe_regression fixture....
FAILED tests/test_dataframe_regression.py::test_non_pandas_dataframe - AssertionError: Regex pattern "Only pandas DataFrames are supported on on dataframe_regression fix...
====================================================================== 5 failed, 39 passed in 18.00s =======================================================================

Documentation builds broken

Heya, As you can see from https://readthedocs.org/projects/pytest-regressions/builds/21888409/:

WARNING: autodoc: failed to import method 'data_regression.DataRegressionFixture.check' from module 'pytest_regressions'; the following exception was raised:
No module named 'importlib.metadata'

So none of the API documentation is built.

Note, it seems you are missing a .readthedocs.yaml https://docs.readthedocs.io/en/stable/config-file/v2.html, to specify how to configure RTD

`array_regression` combining functionality of `num_regression`, `dataframe_regression` and `ndarrays_regression`

I'm, copying the relevant bits from #72 below:

It is possible and fairly easy to support the functionality of dataframe_regression, num_regression and ndarrays_regression in one regression fixture. E.g. to be used as follows:

# Case of 1D arrays with same length:
data = {"ar1": np.array([2.3, 9.4]) , "ar2": np.array([3, 4])}
dataframe = pd.DataFrame.from_dict(data)
array_regression(data)  # same as current num_regression
array_regression(dataframe)  # same as current dataframe_regression, based on type
array_regression(data, format="npz")  # same as ndarrays_regression
array_regression(dataframe, format="npz")  # not yet possible, but easy enough
# Case of ND arrays with different shapes
data = {"ar1": np.array([2.3, 9.4]) , "ar2": np.array([[3, 4], [2, 1]])}
dataframe = pd.DataFrame.from_dict(data)  # raises ValueError
array_regression(data)  # raises ValueError, suggesting npz format
array_regression(data, format="npz")  # same as ndarrays_regression

Future extensions could include support for HDF5 or user-defined container formats. The name array_regression is intended to be general, i.e. no reference to the dtype, leaving room for future support of more dtypes.

This could also be a good opportunity to revise default tolerances in the new API as mentioned by @tadeu:

With the introduction of a new API like array_regression, it would be a good moment to improve the default tolerances. Currently we use the same defaults of numpy.isclose, and having an atol that is not 0.0 by default is confusing (see numpy/numpy#10161 for example, we might even consider something different than isclose).

Other changes can be considered too, obviously. (E.g. NPZ as default, because it supports more array shapes.) Feel free to comment.

For code reuse, it could be preferable to rewrite fixtures dataframe_regression, num_regression using the same underlying ArrayRegressionFixture class. This is probably not strictly necessary.

The ndarrays_regression fixture can eventually be removed because it was not part of any release yet (at the moment).

Show pytest-clarity diff if installed

It would be awesome if pytest-regression would show the clarity diff provided pytest-clarity if pytest-clarity is installed.

Maybe it only has to be changed in a way to use the default pytest pytest_assertrepr_compare for displaying the diff.

Anyway that would be awesome as in our CI runner we can't easily extract the HTML diff files from the temp dir, so the CI output is often the only thing we have for failing tests.

Allow default basename to include class name

When using the *_regression fixtures within a class there is a chance of collision in the generated files due to the fact that the base name is constructed using the node name: https://github.com/ESSS/pytest-regressions/blob/master/src/pytest_regressions/common.py#L116.

For example, the following three tests (when placed in the same test file) all try to use the same basename and two out of the three will fail:

def test_main_case(data_regression):
    data_regression.check({"key": "value"})


class TestMyStuff(object):
    def test_main_case(self, data_regression):
        data_regression.check({"my_key": "my_value"})


class TestMyStuffToo(object):
    def test_main_case(self, data_regression):
        data_regression.check({"my_key_too": "my_value_too"})

Could this be fixed to also include the class name (if there is one)? I think it should be as simple as changing request.node.name to request.node.nodeId.

It's easy to work around the issue by specifying a custom basename but it would be helpful to fix the default.

Testing NaN does not work

I have a numpy array that I have to test with inside a NaN that has to be there. Version 2.2.0 works, while version 2.3.0 does not work.

output of 2.2.0:

bertossa@pippo:~/analisi/tests$ pytest -sv . -k gk
WARNING: cannot import thermocepstrum
cite as:
Riccardo Bertossa, analisi
https://github.com/rikigigi/analisi
(c) 2017-2021
=========
COMPILED AT Jan 17 2022 14:47:14 by /usr/bin/c++ whith flags (Release) -O3 -DNDEBUG -O3 (Debug)  -g -DEIGEN_INITIALIZE_MATRICES_BY_NAN    -fsanitize=leak -fsanitize=undefined (build type was Release) on a Linux-5.4.0-94-generic whith processor x86_64
With python support: /usr/lib/x86_64-linux-gnu/libpython3.8.so
With gromacs XDR file conversion support
v0.3.4
============================================================================ test session starts ============================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-0.13.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/bertossa/analisi/tests
plugins: datadir-1.3.1, regressions-2.2.0
collected 6 items / 5 deselected / 1 selected                                                                                                                               

test_gk.py::test_gk [[ 1.15338509e+05 -3.54402664e+01 -3.54402664e+01 ...  0.00000000e+00
   0.00000000e+00  0.00000000e+00]
 [ 1.13409212e+05 -3.51673102e+01 -3.49856870e+01 ...  3.53293021e-02
   7.91862554e+04             nan]
 [ 1.07866235e+05 -3.41664742e+01 -3.38088613e+01 ...  5.21211914e-02
   1.55216495e+05  3.95931277e+04]
 ...
 [-2.73309429e+03 -2.66983885e-01  1.39421901e+00 ... -2.67101979e-02
   1.62004622e+05  5.34430500e+05]
 [-2.58976152e+03 -1.40244210e-01  1.24827661e+00 ... -2.65180986e-02
   1.59872994e+05  5.34257456e+05]
 [-2.37993149e+03 -2.79107527e-02  1.07349726e+00 ... -2.63087491e-02
   1.57897803e+05  5.34083800e+05]]
PASSED

============================================================================= warnings summary ==============================================================================
../../../../usr/lib/python3/dist-packages/ipywidgets/widgets/widget_selection.py:9
  /usr/lib/python3/dist-packages/ipywidgets/widgets/widget_selection.py:9: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    from collections import Mapping

../../../../usr/lib/python3/dist-packages/ipywidgets/widgets/interaction.py:29
  /usr/lib/python3/dist-packages/ipywidgets/widgets/interaction.py:29: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    from collections import Iterable, Mapping

../../../../usr/lib/python3/dist-packages/ipywidgets/widgets/widget_link.py:19
../../../../usr/lib/python3/dist-packages/ipywidgets/widgets/widget_link.py:19
  /usr/lib/python3/dist-packages/ipywidgets/widgets/widget_link.py:19: DeprecationWarning: Traits should be given as instances, not types (for example, `Int()`, not `Int`) Passing types is deprecated in traitlets 4.1.
    super(WidgetTraitTuple, self).__init__(Instance(Widget), Unicode, **kwargs)

test_gk.py::test_gk
test_gk.py::test_gk
  /home/bertossa/.local/lib/python3.8/site-packages/pytest_regressions/dataframe_regression.py:126: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    if data_type in [float, np.float, np.float16, np.float32, np.float64]:

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================ 1 passed, 5 deselected, 6 warnings in 1.66s ================================================================
bertossa@pippo:~/analisi/tests$ 


output of 2.3.0:

bertossa@pippo:~/analisi/tests$ pytest -sv . -k gk
WARNING: cannot import thermocepstrum
cite as:
Riccardo Bertossa, analisi
https://github.com/rikigigi/analisi
(c) 2017-2021
=========
COMPILED AT Jan 17 2022 14:47:14 by /usr/bin/c++ whith flags (Release) -O3 -DNDEBUG -O3 (Debug)  -g -DEIGEN_INITIALIZE_MATRICES_BY_NAN    -fsanitize=leak -fsanitize=undefined (build type was Release) on a Linux-5.4.0-94-generic whith processor x86_64
With python support: /usr/lib/x86_64-linux-gnu/libpython3.8.so
With gromacs XDR file conversion support
v0.3.4
============================================================================ test session starts ============================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-0.13.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/bertossa/analisi/tests
plugins: datadir-1.3.1, regressions-2.3.0
collected 6 items / 5 deselected / 1 selected                                                                                                                               

test_gk.py::test_gk [[ 1.15338509e+05 -3.54402664e+01 -3.54402664e+01 ...  0.00000000e+00
   0.00000000e+00  0.00000000e+00]
 [ 1.13409212e+05 -3.51673102e+01 -3.49856870e+01 ...  3.53293021e-02
   7.91862554e+04             nan]
 [ 1.07866235e+05 -3.41664742e+01 -3.38088613e+01 ...  5.21211914e-02
   1.55216495e+05  3.95931277e+04]
 ...
 [-2.73309429e+03 -2.66983885e-01  1.39421901e+00 ... -2.67101979e-02
   1.62004622e+05  5.34430500e+05]
 [-2.58976152e+03 -1.40244210e-01  1.24827661e+00 ... -2.65180986e-02
   1.59872994e+05  5.34257456e+05]
 [-2.37993149e+03 -2.79107527e-02  1.07349726e+00 ... -2.63087491e-02
   1.57897803e+05  5.34083800e+05]]
FAILED

================================================================================= FAILURES ==================================================================================
__________________________________________________________________________________ test_gk __________________________________________________________________________________

left = array(['115338.50895090569', '-35.440266364698509', '-35.440266364698509',
       ..., '-0.026308749054506145', '157897.80272819981',
       '534083.79967621446'], dtype=object)
right = array(['115338.50895090567', '-35.440266364698509', '-35.440266364698509',
       ..., '-0.02630874905450617', '157897.80272819955',
       '534083.79967621388'], dtype=object)
op = <built-in function sub>, is_cmp = False

    def _na_arithmetic_op(left, right, op, is_cmp: bool = False):
        """
        Return the result of evaluating op on the passed in values.
    
        If native types are not compatible, try coercion to object dtype.
    
        Parameters
        ----------
        left : np.ndarray
        right : np.ndarray or scalar
        is_cmp : bool, default False
            If this a comparison operation.
    
        Returns
        -------
        array-like
    
        Raises
        ------
        TypeError : invalid operation
        """
        if isinstance(right, str):
            # can never use numexpr
            func = op
        else:
            func = partial(expressions.evaluate, op)
    
        try:
>           result = func(left, right)

../../.local/lib/python3.8/site-packages/pandas/core/ops/array_ops.py:166: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

op = <built-in function sub>
a = array(['115338.50895090569', '-35.440266364698509', '-35.440266364698509',
       ..., '-0.026308749054506145', '157897.80272819981',
       '534083.79967621446'], dtype=object)
b = array(['115338.50895090567', '-35.440266364698509', '-35.440266364698509',
       ..., '-0.02630874905450617', '157897.80272819955',
       '534083.79967621388'], dtype=object)
use_numexpr = True

    def evaluate(op, a, b, use_numexpr: bool = True):
        """
        Evaluate and return the expression of the op on a and b.
    
        Parameters
        ----------
        op : the actual operand
        a : left operand
        b : right operand
        use_numexpr : bool, default True
            Whether to try to use numexpr.
        """
        op_str = _op_str_mapping[op]
        if op_str is not None:
            if use_numexpr:
                # error: "None" not callable
>               return _evaluate(op, op_str, a, b)  # type: ignore[misc]

../../.local/lib/python3.8/site-packages/pandas/core/computation/expressions.py:239: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

op = <built-in function sub>, op_str = '-'
a = array(['115338.50895090569', '-35.440266364698509', '-35.440266364698509',
       ..., '-0.026308749054506145', '157897.80272819981',
       '534083.79967621446'], dtype=object)
b = array(['115338.50895090567', '-35.440266364698509', '-35.440266364698509',
       ..., '-0.02630874905450617', '157897.80272819955',
       '534083.79967621388'], dtype=object)

    def _evaluate_standard(op, op_str, a, b):
        """
        Standard evaluation.
        """
        if _TEST_MODE:
            _store_test_result(False)
>       return op(a, b)
E       TypeError: unsupported operand type(s) for -: 'str' and 'str'

../../.local/lib/python3.8/site-packages/pandas/core/computation/expressions.py:69: TypeError

During handling of the above exception, another exception occurred:

analisi_log = <pyanalisi.pyanalisi.ReadLog object at 0x7fdb87a47470>, num_regression = <pytest_regressions.num_regression.NumericRegressionFixture object at 0x7fdb87a28130>

    def test_gk(analisi_log,num_regression):
        import pyanalisi
        gk=pyanalisi.GreenKubo(analisi_log,'',1,['c_flux[1]','c_vcm[1][1]'], False, 2000, 2,False,0,4,False,1,100)
        gk.reset(analisi_log.getNtimesteps()-2000)
        gk.calculate(0)
        m=np.array(gk,copy=False)
        print(m)
>       num_regression.check({'gk':m.flatten()})

test_gk.py:11: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.local/lib/python3.8/site-packages/pandas/core/ops/common.py:69: in new_method
    return method(self, other)
../../.local/lib/python3.8/site-packages/pandas/core/arraylike.py:100: in __sub__
    return self._arith_method(other, operator.sub)
../../.local/lib/python3.8/site-packages/pandas/core/series.py:5526: in _arith_method
    result = ops.arithmetic_op(lvalues, rvalues, op)
../../.local/lib/python3.8/site-packages/pandas/core/ops/array_ops.py:224: in arithmetic_op
    res_values = _na_arithmetic_op(left, right, op)
../../.local/lib/python3.8/site-packages/pandas/core/ops/array_ops.py:173: in _na_arithmetic_op
    result = _masked_arith_op(left, right, op)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

x = array(['115338.50895090569', '-35.440266364698509', '-35.440266364698509',
       ..., '-0.026308749054506145', '157897.80272819981',
       '534083.79967621446'], dtype=object)
y = array(['115338.50895090567', '-35.440266364698509', '-35.440266364698509',
       ..., '-0.02630874905450617', '157897.80272819955',
       '534083.79967621388'], dtype=object)
op = <built-in function sub>

    def _masked_arith_op(x: np.ndarray, y, op):
        """
        If the given arithmetic operation fails, attempt it again on
        only the non-null elements of the input array(s).
    
        Parameters
        ----------
        x : np.ndarray
        y : np.ndarray, Series, Index
        op : binary operator
        """
        # For Series `x` is 1D so ravel() is a no-op; calling it anyway makes
        # the logic valid for both Series and DataFrame ops.
        xrav = x.ravel()
        assert isinstance(x, np.ndarray), type(x)
        if isinstance(y, np.ndarray):
            dtype = find_common_type([x.dtype, y.dtype])
            # error: Argument "dtype" to "empty" has incompatible type
            # "Union[dtype, ExtensionDtype]"; expected "Union[dtype, None, type,
            # _SupportsDtype, str, Tuple[Any, int], Tuple[Any, Union[int,
            # Sequence[int]]], List[Any], _DtypeDict, Tuple[Any, Any]]"
            result = np.empty(x.size, dtype=dtype)  # type: ignore[arg-type]
    
            if len(x) != len(y):
                raise ValueError(x.shape, y.shape)
            else:
                ymask = notna(y)
    
            # NB: ravel() is only safe since y is ndarray; for e.g. PeriodIndex
            #  we would get int64 dtype, see GH#19956
            yrav = y.ravel()
            mask = notna(xrav) & ymask.ravel()
    
            # See GH#5284, GH#5035, GH#19448 for historical reference
            if mask.any():
>               result[mask] = op(xrav[mask], yrav[mask])
E               TypeError: unsupported operand type(s) for -: 'str' and 'str'

../../.local/lib/python3.8/site-packages/pandas/core/ops/array_ops.py:112: TypeError
============================================================================= warnings summary ==============================================================================
../../../../usr/lib/python3/dist-packages/ipywidgets/widgets/widget_selection.py:9
  /usr/lib/python3/dist-packages/ipywidgets/widgets/widget_selection.py:9: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    from collections import Mapping

../../../../usr/lib/python3/dist-packages/ipywidgets/widgets/interaction.py:29
  /usr/lib/python3/dist-packages/ipywidgets/widgets/interaction.py:29: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    from collections import Iterable, Mapping

../../../../usr/lib/python3/dist-packages/ipywidgets/widgets/widget_link.py:19
../../../../usr/lib/python3/dist-packages/ipywidgets/widgets/widget_link.py:19
  /usr/lib/python3/dist-packages/ipywidgets/widgets/widget_link.py:19: DeprecationWarning: Traits should be given as instances, not types (for example, `Int()`, not `Int`) Passing types is deprecated in traitlets 4.1.
    super(WidgetTraitTuple, self).__init__(Instance(Widget), Unicode, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================== short test summary info ==========================================================================
FAILED test_gk.py::test_gk - TypeError: unsupported operand type(s) for -: 'str' and 'str'
================================================================ 1 failed, 5 deselected, 4 warnings in 1.94s ================================================================
bertossa@pippo:~/analisi/tests$ 


Failing tests

Hi,

While running the tests of your pytest fixture, I got many Failures.

IDK how to investigate these, but at least the last is due to a trivial space alignment issue between the code and the testcode of your plugin.

============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
rootdir: /tmp/autopkgtest.2XwkFn/autopkgtest_tmp
plugins: regressions-2.1.1, datadir-1.3.1+ds
collecting ... collected 39 items

tests/test_data_regression.py::test_example PASSED
tests/test_data_regression.py::test_basename PASSED
tests/test_data_regression.py::test_custom_object PASSED
tests/test_data_regression.py::test_usage_workflow ============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py F                                                           [100%]

=================================== FAILURES ===================================
____________________________________ test_1 ____________________________________

data_regression = <pytest_regressions.data_regression.DataRegressionFixture object at 0x7f490431d970>

    def test_1(data_regression):
        contents = sys.testing_get_data()
>       data_regression.check(contents)
E       Failed: File not found in data directory, created:
E       - /tmp/pytest-of-becue/pytest-0/test_usage_workflow0/test_file/test_1.yml

test_file.py:4: Failed
=========================== 1 failed in 0.04 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py .                                                           [100%]

=========================== 1 passed in 0.02 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py F                                                           [100%]

=================================== FAILURES ===================================
____________________________________ test_1 ____________________________________

data_regression = <pytest_regressions.data_regression.DataRegressionFixture object at 0x7f490428b100>

    def test_1(data_regression):
        contents = sys.testing_get_data()
>       data_regression.check(contents)
E       AssertionError: FILES DIFFER:
E       /tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-2/test_10/test_file/test_1.yml
E       /tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-2/test_10/test_file/test_1.obtained.yml
E       HTML DIFF: /tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-2/test_10/test_file/test_1.obtained.diff.html
E       --- 
E       +++ 
E       @@ -1,2 +1,2 @@
E       -contents: Foo
E       -value: 10
E       +contents: Bar
E       +value: 20

test_file.py:4: AssertionError
=========================== 1 failed in 0.01 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py F                                                           [100%]

=================================== FAILURES ===================================
____________________________________ test_1 ____________________________________

datadir = PosixPath('/tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-3/test_10/test_file')
original_datadir = PosixPath('/tmp/pytest-of-becue/pytest-0/test_usage_workflow0/test_file')
request = <SubRequest 'data_regression' for <Function test_1>>
check_fn = functools.partial(<function check_text_files at 0x7f4904507310>, encoding='UTF-8')
dump_fn = <function DataRegressionFixture.check.<locals>.dump at 0x7f4904215c10>
extension = '.yml', basename = 'test_1', fullpath = None, force_regen = True
obtained_filename = PosixPath('/tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-3/test_10/test_file/test_1.obtained.yml')
dump_aux_fn = <function <lambda> at 0x7f49045073a0>

    def perform_regression_check(
        datadir,
        original_datadir,
        request,
        check_fn,
        dump_fn,
        extension,
        basename=None,
        fullpath=None,
        force_regen=False,
        obtained_filename=None,
        dump_aux_fn=lambda filename: [],
    ):
        """
        First run of this check will generate a expected file. Following attempts will always try to
        match obtained files with that expected file.
    
        If expected file needs to be updated, just enable `force_regen` argument.
    
        :param Path datadir: Fixture embed_data.
        :param Path original_datadir: Fixture embed_data.
        :param SubRequest request: Pytest request object.
        :param callable check_fn: A function that receives as arguments, respectively, absolute path to
            obtained file and absolute path to expected file. It must assert if contents of file match.
            Function can safely assume that obtained file is already dumped and only care about
            comparison.
        :param callable dump_fn: A function that receive an absolute file path as argument. Implementor
            must dump file in this path.
        :param callable dump_aux_fn: A function that receives the same file path as ``dump_fn``, but may
            dump additional files to help diagnose this regression later (for example dumping image of
            3d views and plots to compare later). Must return the list of file names written (used to display).
        :param str extension: Extension of files compared by this check.
        :param bool force_regen: if true it will regenerate expected file.
        :param str obtained_filename: complete path to use to write the obtained file. By
            default will prepend `.obtained` before the file extension.
        ..see: `data_regression.Check` for `basename` and `fullpath` arguments.
        """
        import re
    
        assert not (basename and fullpath), "pass either basename or fullpath, but not both"
    
        __tracebackhide__ = True
    
        if basename is None:
            basename = re.sub(r"[\W]", "_", request.node.name)
    
        if fullpath:
            filename = source_filename = Path(fullpath)
        else:
            filename = datadir / (basename + extension)
            source_filename = original_datadir / (basename + extension)
    
        def make_location_message(banner, filename, aux_files):
            msg = [banner, f"- {filename}"]
            if aux_files:
                msg.append("Auxiliary:")
                msg += [f"- {x}" for x in aux_files]
            return "\n".join(msg)
    
        force_regen = force_regen or request.config.getoption("force_regen")
        if not filename.is_file():
            source_filename.parent.mkdir(parents=True, exist_ok=True)
            dump_fn(source_filename)
            aux_created = dump_aux_fn(source_filename)
    
            msg = make_location_message(
                "File not found in data directory, created:", source_filename, aux_created
            )
            pytest.fail(msg)
        else:
            if obtained_filename is None:
                if fullpath:
                    obtained_filename = (datadir / basename).with_suffix(
                        ".obtained" + extension
                    )
                else:
                    obtained_filename = filename.with_suffix(".obtained" + extension)
    
            dump_fn(obtained_filename)
    
            try:
>               check_fn(obtained_filename, filename)

/usr/lib/python3/dist-packages/pytest_regressions/common.py:153: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obtained_fn = PosixPath('/tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-3/test_10/test_file/test_1.obtained.yml')
expected_fn = PosixPath('/tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-3/test_10/test_file/test_1.yml')
fix_callback = <function <lambda> at 0x7f4904507280>, encoding = 'UTF-8'

    def check_text_files(obtained_fn, expected_fn, fix_callback=lambda x: x, encoding=None):
        """
        Compare two files contents. If the files differ, show the diff and write a nice HTML
        diff file into the data directory.
    
        :param Path obtained_fn: path to obtained file during current testing.
    
        :param Path expected_fn: path to the expected file, obtained from previous testing.
    
        :param str encoding: encoding used to open the files.
    
        :param callable fix_callback:
            A callback to "fix" the contents of the obtained (first) file.
            This callback receives a list of strings (lines) and must also return a list of lines,
            changed as needed.
            The resulting lines will be used to compare with the contents of expected_fn.
        """
        __tracebackhide__ = True
    
        obtained_fn = Path(obtained_fn)
        expected_fn = Path(expected_fn)
        obtained_lines = fix_callback(obtained_fn.read_text(encoding=encoding).splitlines())
        expected_lines = expected_fn.read_text(encoding=encoding).splitlines()
    
        if obtained_lines != expected_lines:
            diff_lines = list(
                difflib.unified_diff(expected_lines, obtained_lines, lineterm="")
            )
            if len(diff_lines) <= 500:
                html_fn = obtained_fn.with_suffix(".diff.html")
                try:
                    differ = difflib.HtmlDiff()
                    html_diff = differ.make_file(
                        fromlines=expected_lines,
                        fromdesc=expected_fn,
                        tolines=obtained_lines,
                        todesc=obtained_fn,
                    )
                except Exception as e:
                    html_fn = "(failed to generate html diff: %s)" % e
                else:
                    html_fn.write_text(html_diff, encoding="UTF-8")
    
                diff = ["FILES DIFFER:", str(expected_fn), str(obtained_fn)]
                diff += ["HTML DIFF: %s" % html_fn]
                diff += diff_lines
>               raise AssertionError("\n".join(diff))
E               AssertionError: FILES DIFFER:
E               /tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-3/test_10/test_file/test_1.yml
E               /tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-3/test_10/test_file/test_1.obtained.yml
E               HTML DIFF: /tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow0/pytest-of-becue/pytest-3/test_10/test_file/test_1.obtained.diff.html
E               --- 
E               +++ 
E               @@ -1,2 +1,2 @@
E               -contents: Foo
E               -value: 10
E               +contents: Bar
E               +value: 20

/usr/lib/python3/dist-packages/pytest_regressions/common.py:58: AssertionError

During handling of the above exception, another exception occurred:

data_regression = <pytest_regressions.data_regression.DataRegressionFixture object at 0x7f49041d5bb0>

    def test_1(data_regression):
        contents = sys.testing_get_data()
>       data_regression.check(contents)
E       Failed: Files differ and --force-regen set, regenerating file at:
E       - /tmp/pytest-of-becue/pytest-0/test_usage_workflow0/test_file/test_1.yml

test_file.py:4: Failed
=========================== 1 failed in 0.03 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py .                                                           [100%]

=========================== 1 passed in 0.01 seconds ===========================
PASSED
tests/test_data_regression.py::test_data_regression_full_path ============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_data_regression_full_path0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_foo.py F                                                            [100%]

=================================== FAILURES ===================================
_____________________________________ test _____________________________________

data_regression = <pytest_regressions.data_regression.DataRegressionFixture object at 0x7f49041767f0>

    def test(data_regression):
        contents = {'data': [1, 2]}
>       data_regression.check(contents, fullpath='/tmp/pytest-of-becue/pytest-0/test_data_regression_full_path1/full/path/to/contents.yaml')
E       Failed: File not found in data directory, created:
E       - /tmp/pytest-of-becue/pytest-0/test_data_regression_full_path1/full/path/to/contents.yaml

test_foo.py:3: Failed
=========================== 1 failed in 0.01 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_data_regression_full_path0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_foo.py .                                                            [100%]

=========================== 1 passed in 0.01 seconds ===========================
PASSED
tests/test_data_regression.py::test_data_regression_no_aliases ============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_data_regression_no_aliases0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py F                                                           [100%]

=================================== FAILURES ===================================
_____________________________________ test _____________________________________

data_regression = <pytest_regressions.data_regression.DataRegressionFixture object at 0x7f49040b12b0>

    def test(data_regression):
        red = (255, 0, 0)
        green = (0, 255, 0)
        blue = (0, 0, 255)
    
        contents = {
            'color1': red,
            'color2': green,
            'color3': blue,
            'color4': red,
            'color5': green,
            'color6': blue,
        }
>       data_regression.Check(contents)
E       Failed: File not found in data directory, created:
E       - /tmp/pytest-of-becue/pytest-0/test_data_regression_no_aliases0/test_file/test.yml

test_file.py:14: Failed
=========================== 1 failed in 0.01 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_data_regression_no_aliases0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py .                                                           [100%]

=========================== 1 passed in 0.01 seconds ===========================
PASSED
tests/test_data_regression.py::test_not_create_file_on_error ============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_not_create_file_on_error0
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py F                                                           [100%]

=================================== FAILURES ===================================
_____________________________________ test _____________________________________

data_regression = <pytest_regressions.data_regression.DataRegressionFixture object at 0x7f4904048ac0>

    def test(data_regression):
        class Scalar:
            def __init__(self, value, unit):
                self.value = value
                self.unit = unit
    
        contents = {"scalar": Scalar(10, "m")}
>       data_regression.Check(contents)

test_file.py:8: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3/dist-packages/pytest_regressions/data_regression.py:46: in dump
    dumped_str = yaml.dump_all(
/usr/lib/python3/dist-packages/yaml/__init__.py:278: in dump_all
    dumper.represent(data)
/usr/lib/python3/dist-packages/yaml/representer.py:27: in represent
    node = self.represent_data(data)
/usr/lib/python3/dist-packages/yaml/representer.py:48: in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
/usr/lib/python3/dist-packages/yaml/representer.py:207: in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
/usr/lib/python3/dist-packages/yaml/representer.py:118: in represent_mapping
    node_value = self.represent_data(item_value)
/usr/lib/python3/dist-packages/yaml/representer.py:58: in represent_data
    node = self.yaml_representers[None](self, data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pytest_regressions.data_regression.RegressionYamlDumper object at 0x7f4904048cd0>
data = <test_file.test.<locals>.Scalar object at 0x7f4904048c40>

    def represent_undefined(self, data):
>       raise RepresenterError("cannot represent an object", data)
E       yaml.representer.RepresenterError: ('cannot represent an object', <test_file.test.<locals>.Scalar object at 0x7f4904048c40>)

/usr/lib/python3/dist-packages/yaml/representer.py:231: RepresenterError
=========================== 1 failed in 0.04 seconds ===========================
PASSED
tests/test_dataframe_regression.py::test_usage_workflow ============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow1
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py F                                                           [100%]

=================================== FAILURES ===================================
____________________________________ test_1 ____________________________________

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f4903fc0f10>

    def test_1(dataframe_regression):
        contents = sys.testing_get_data()
>       dataframe_regression.check(pd.DataFrame.from_dict(contents))
E       Failed: File not found in data directory, created:
E       - /tmp/pytest-of-becue/pytest-0/test_usage_workflow1/test_file/test_1.csv

test_file.py:5: Failed
=========================== 1 failed in 0.01 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow1
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py .                                                           [100%]

=========================== 1 passed in 0.01 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow1
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py F                                                           [100%]

=================================== FAILURES ===================================
____________________________________ test_1 ____________________________________

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f4903e9f970>

    def test_1(dataframe_regression):
        contents = sys.testing_get_data()
>       dataframe_regression.check(pd.DataFrame.from_dict(contents))
E       AssertionError: Values are not sufficiently close.
E       To update values, use --force-regen option.
E       
E       data:
E                 obtained_data        expected_data                 diff
E       0   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       1   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       2   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       3   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       4   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       5   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       6   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       7   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       8   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       9   1.19999999999999996  1.10000000000000009  0.09999999999999987
E       10  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       11  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       12  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       13  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       14  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       15  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       16  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       17  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       18  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       19  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       20  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       21  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       22  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       23  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       24  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       25  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       26  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       27  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       28  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       29  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       30  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       31  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       32  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       33  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       34  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       35  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       36  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       37  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       38  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       39  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       40  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       41  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       42  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       43  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       44  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       45  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       46  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       47  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       48  1.19999999999999996  1.10000000000000009  0.09999999999999987
E       49  1.19999999999999996  1.10000000000000009  0.09999999999999987

test_file.py:5: AssertionError
=========================== 1 failed in 0.02 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow1
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py F                                                           [100%]

=================================== FAILURES ===================================
____________________________________ test_1 ____________________________________

datadir = PosixPath('/tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow1/pytest-of-becue/pytest-3/test_10/test_file')
original_datadir = PosixPath('/tmp/pytest-of-becue/pytest-0/test_usage_workflow1/test_file')
request = <SubRequest 'dataframe_regression' for <Function test_1>>
check_fn = <bound method DataFrameRegressionFixture._check_fn of <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f4903e8c970>>
dump_fn = functools.partial(<bound method DataFrameRegressionFixture._dump_fn of <pytest_regressions.dataframe_regression.DataFr...2
37   1.2
38   1.2
39   1.2
40   1.2
41   1.2
42   1.2
43   1.2
44   1.2
45   1.2
46   1.2
47   1.2
48   1.2
49   1.2)
extension = '.csv', basename = 'test_1', fullpath = None, force_regen = True
obtained_filename = PosixPath('/tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow1/pytest-of-becue/pytest-3/test_10/test_file/test_1.obtained.csv')
dump_aux_fn = <function <lambda> at 0x7f49045073a0>

    def perform_regression_check(
        datadir,
        original_datadir,
        request,
        check_fn,
        dump_fn,
        extension,
        basename=None,
        fullpath=None,
        force_regen=False,
        obtained_filename=None,
        dump_aux_fn=lambda filename: [],
    ):
        """
        First run of this check will generate a expected file. Following attempts will always try to
        match obtained files with that expected file.
    
        If expected file needs to be updated, just enable `force_regen` argument.
    
        :param Path datadir: Fixture embed_data.
        :param Path original_datadir: Fixture embed_data.
        :param SubRequest request: Pytest request object.
        :param callable check_fn: A function that receives as arguments, respectively, absolute path to
            obtained file and absolute path to expected file. It must assert if contents of file match.
            Function can safely assume that obtained file is already dumped and only care about
            comparison.
        :param callable dump_fn: A function that receive an absolute file path as argument. Implementor
            must dump file in this path.
        :param callable dump_aux_fn: A function that receives the same file path as ``dump_fn``, but may
            dump additional files to help diagnose this regression later (for example dumping image of
            3d views and plots to compare later). Must return the list of file names written (used to display).
        :param str extension: Extension of files compared by this check.
        :param bool force_regen: if true it will regenerate expected file.
        :param str obtained_filename: complete path to use to write the obtained file. By
            default will prepend `.obtained` before the file extension.
        ..see: `data_regression.Check` for `basename` and `fullpath` arguments.
        """
        import re
    
        assert not (basename and fullpath), "pass either basename or fullpath, but not both"
    
        __tracebackhide__ = True
    
        if basename is None:
            basename = re.sub(r"[\W]", "_", request.node.name)
    
        if fullpath:
            filename = source_filename = Path(fullpath)
        else:
            filename = datadir / (basename + extension)
            source_filename = original_datadir / (basename + extension)
    
        def make_location_message(banner, filename, aux_files):
            msg = [banner, f"- {filename}"]
            if aux_files:
                msg.append("Auxiliary:")
                msg += [f"- {x}" for x in aux_files]
            return "\n".join(msg)
    
        force_regen = force_regen or request.config.getoption("force_regen")
        if not filename.is_file():
            source_filename.parent.mkdir(parents=True, exist_ok=True)
            dump_fn(source_filename)
            aux_created = dump_aux_fn(source_filename)
    
            msg = make_location_message(
                "File not found in data directory, created:", source_filename, aux_created
            )
            pytest.fail(msg)
        else:
            if obtained_filename is None:
                if fullpath:
                    obtained_filename = (datadir / basename).with_suffix(
                        ".obtained" + extension
                    )
                else:
                    obtained_filename = filename.with_suffix(".obtained" + extension)
    
            dump_fn(obtained_filename)
    
            try:
>               check_fn(obtained_filename, filename)

/usr/lib/python3/dist-packages/pytest_regressions/common.py:153: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f4903e8c970>
obtained_filename = PosixPath('/tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow1/pytest-of-becue/pytest-3/test_10/test_file/test_1.obtained.csv')
expected_filename = PosixPath('/tmp/pytest-of-becue/pytest-0/tmp-test_usage_workflow1/pytest-of-becue/pytest-3/test_10/test_file/test_1.csv')

    def _check_fn(self, obtained_filename, expected_filename):
        """
        Check if dict contents dumped to a file match the contents in expected file.
    
        :param str obtained_filename:
        :param str expected_filename:
        """
        try:
            import numpy as np
        except ModuleNotFoundError:
            raise ModuleNotFoundError(import_error_message("Numpy"))
        try:
            import pandas as pd
        except ModuleNotFoundError:
            raise ModuleNotFoundError(import_error_message("Pandas"))
    
        __tracebackhide__ = True
    
        obtained_data = pd.read_csv(str(obtained_filename))
        expected_data = pd.read_csv(str(expected_filename))
    
        comparison_tables_dict = {}
        for k in obtained_data.keys():
            obtained_column = obtained_data[k]
            expected_column = expected_data.get(k)
    
            if expected_column is None:
                error_msg = f"Could not find key '{k}' in the expected results.\n"
                error_msg += "Keys in the obtained data table: ["
                for k in obtained_data.keys():
                    error_msg += f"'{k}', "
                error_msg += "]\n"
                error_msg += "Keys in the expected data table: ["
                for k in expected_data.keys():
                    error_msg += f"'{k}', "
                error_msg += "]\n"
                error_msg += "To update values, use --force-regen option.\n\n"
                raise AssertionError(error_msg)
    
            tolerance_args = self._tolerances_dict.get(k, self._default_tolerance)
    
            self._check_data_types(k, obtained_column, expected_column)
            self._check_data_shapes(obtained_column, expected_column)
    
            data_type = obtained_column.values.dtype
            if data_type in [float, np.float, np.float16, np.float32, np.float64]:
                not_close_mask = ~np.isclose(
                    obtained_column.values,
                    expected_column.values,
                    equal_nan=True,
                    **tolerance_args,
                )
            else:
                not_close_mask = obtained_column.values != expected_column.values
    
            if np.any(not_close_mask):
                diff_ids = np.where(not_close_mask)[0]
                diff_obtained_data = obtained_column[diff_ids]
                diff_expected_data = expected_column[diff_ids]
                if data_type == np.bool:
                    diffs = np.logical_xor(obtained_column, expected_column)[diff_ids]
                else:
                    diffs = np.abs(obtained_column - expected_column)[diff_ids]
    
                comparison_table = pd.concat(
                    [diff_obtained_data, diff_expected_data, diffs], axis=1
                )
                comparison_table.columns = [f"obtained_{k}", f"expected_{k}", "diff"]
                comparison_tables_dict[k] = comparison_table
    
        if len(comparison_tables_dict) > 0:
            error_msg = "Values are not sufficiently close.\n"
            error_msg += "To update values, use --force-regen option.\n\n"
            for k, comparison_table in comparison_tables_dict.items():
                error_msg += f"{k}:\n{comparison_table}\n\n"
>           raise AssertionError(error_msg)
E           AssertionError: Values are not sufficiently close.
E           To update values, use --force-regen option.
E           
E           data:
E                     obtained_data        expected_data                 diff
E           0   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           1   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           2   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           3   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           4   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           5   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           6   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           7   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           8   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           9   1.19999999999999996  1.10000000000000009  0.09999999999999987
E           10  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           11  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           12  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           13  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           14  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           15  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           16  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           17  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           18  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           19  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           20  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           21  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           22  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           23  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           24  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           25  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           26  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           27  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           28  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           29  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           30  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           31  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           32  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           33  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           34  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           35  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           36  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           37  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           38  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           39  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           40  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           41  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           42  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           43  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           44  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           45  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           46  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           47  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           48  1.19999999999999996  1.10000000000000009  0.09999999999999987
E           49  1.19999999999999996  1.10000000000000009  0.09999999999999987

/usr/lib/python3/dist-packages/pytest_regressions/dataframe_regression.py:156: AssertionError

During handling of the above exception, another exception occurred:

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f4903e8c970>

    def test_1(dataframe_regression):
        contents = sys.testing_get_data()
>       dataframe_regression.check(pd.DataFrame.from_dict(contents))
E       Failed: Files differ and --force-regen set, regenerating file at:
E       - /tmp/pytest-of-becue/pytest-0/test_usage_workflow1/test_file/test_1.csv

test_file.py:5: Failed
=========================== 1 failed in 0.04 seconds ===========================
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /tmp/pytest-of-becue/pytest-0/test_usage_workflow1
plugins: regressions-2.1.1, datadir-1.3.1+ds
collected 1 item

test_file.py .                                                           [100%]

=========================== 1 passed in 0.01 seconds ===========================
PASSED
tests/test_dataframe_regression.py::test_common_cases PASSED
tests/test_dataframe_regression.py::test_different_data_types PASSED
tests/test_dataframe_regression.py::test_non_numeric_data[array0] FAILED

=================================== FAILURES ===================================
________________________ test_non_numeric_data[array0] _________________________

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f4903dc21c0>
array = [array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19])]
no_regen = None

    @pytest.mark.parametrize(
        "array", [[np.random.randint(10, 99, 6)] * 6, [Foo(i) for i in range(4)]]
    )
    def test_non_numeric_data(dataframe_regression, array, no_regen):
        data1 = pd.DataFrame()
        data1["data1"] = array
        with pytest.raises(
            AssertionError,
            match="Only numeric data is supported on dataframe_regression fixture.\n"
            "  Array with type '%s' was given." % (str(data1["data1"].dtype),),
        ):
>           dataframe_regression.check(data1)

tests/test_dataframe_regression.py:184: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f4903dc21c0>
data_frame =                       data1
0  [39, 98, 98, 92, 58, 19]
1  [39, 98, 98, 92, 58, 19]
2  [39, 98, 98, 92, 58, 19]
3  [39, 98, 98, 92, 58, 19]
4  [39, 98, 98, 92, 58, 19]
5  [39, 98, 98, 92, 58, 19]
basename = None, fullpath = None, tolerances = None, default_tolerance = None

    def check(
        self,
        data_frame,
        basename=None,
        fullpath=None,
        tolerances=None,
        default_tolerance=None,
    ):
        """
        Checks the given pandas dataframe against a previously recorded version, or generate a new file.
    
        Example::
    
            data_frame = pandas.DataFrame.from_dict({
                'U_gas': U[0][positions],
                'U_liquid': U[1][positions],
                'gas_vol_frac [-]': vol_frac[0][positions],
                'liquid_vol_frac [-]': vol_frac[1][positions],
                'P': Pa_to_bar(P)[positions],
            })
            dataframe_regression.check(data_frame)
    
        :param pandas.DataFrame data_frame: pandas DataFrame containing data for regression check.
    
        :param str basename: basename of the file to test/record. If not given the name
            of the test is used.
    
        :param str fullpath: complete path to use as a reference file. This option
            will ignore embed_data completely, being useful if a reference file is located
            in the session data dir for example.
    
        :param dict tolerances: dict mapping keys from the data_dict to tolerance settings for the
            given data. Example::
    
                tolerances={'U': Tolerance(atol=1e-2)}
    
        :param dict default_tolerance: dict mapping the default tolerance for the current check
            call. Example::
    
                default_tolerance=dict(atol=1e-7, rtol=1e-18).
    
            If not provided, will use defaults from numpy's ``isclose`` function.
    
        ``basename`` and ``fullpath`` are exclusive.
        """
        try:
            import pandas as pd
        except ModuleNotFoundError:
            raise ModuleNotFoundError(import_error_message("Pandas"))
    
        import functools
    
        __tracebackhide__ = True
    
        assert type(data_frame) is pd.DataFrame, (
            "Only pandas DataFrames are supported on on dataframe_regression fixture.\n"
            "Object with type '%s' was given." % (str(type(data_frame)),)
        )
    
        for column in data_frame.columns:
            array = data_frame[column]
            # Skip assertion if an array of strings
            if (array.dtype == "O") and (type(array[0]) is str):
                continue
            # Rejected: timedelta, datetime, objects, zero-terminated bytes, unicode strings and raw data
>           assert array.dtype not in ["m", "M", "O", "S", "a", "U", "V"], (
                "Only numeric data is supported on dataframe_regression fixture.\n"
                "Array with type '%s' was given.\n" % (str(array.dtype),)
            )
E           AssertionError: Only numeric data is supported on dataframe_regression fixture.
E           Array with type 'object' was given.

/usr/lib/python3/dist-packages/pytest_regressions/dataframe_regression.py:235: AssertionError

During handling of the above exception, another exception occurred:

dataframe_regression = <pytest_regressions.dataframe_regression.DataFrameRegressionFixture object at 0x7f4903dc21c0>
array = [array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19]), array([39, 98, 98, 92, 58, 19])]
no_regen = None

    @pytest.mark.parametrize(
        "array", [[np.random.randint(10, 99, 6)] * 6, [Foo(i) for i in range(4)]]
    )
    def test_non_numeric_data(dataframe_regression, array, no_regen):
        data1 = pd.DataFrame()
        data1["data1"] = array
        with pytest.raises(
            AssertionError,
            match="Only numeric data is supported on dataframe_regression fixture.\n"
            "  Array with type '%s' was given." % (str(data1["data1"].dtype),),
        ):
>           dataframe_regression.check(data1)
E           AssertionError: Pattern "Only numeric data is supported on dataframe_regression fixture.\n  Array with type 'object' was given." not found in "Only numeric data is supported on dataframe_regression fixture.\nArray with type 'object' was given.\n"

tests/test_dataframe_regression.py:184: AssertionError
===================== 1 failed, 10 passed in 1.39 seconds ======================

using image_regression on bytesIO output raise an SSL socket error

I'm using the following code in my tests and I receive a weird error after completion:

    def test_plot_by_features(self, gaul, image_regression):
        fc = gaul.limit(10).select(["ADM0_CODE", "ADM1_CODE", "ADM2_CODE"])
        fig, ax = fc.geetools.plot_by_features()
        fig.savefig(buf := io.BytesIO())
        image_regression.check(buf.getvalue())
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=10, family=2, type=1, proto=6, laddr=('172.16.5.4', 40806), raddr=('142.250.180.10', 443)>

My guess is taht there are a missing context managers either on your side or mine. I'll continue to investigate but I wanted to record it before I forgot.

num_regression with storage in NPZ files?

It would be useful to store numerical results in NPZ files, because these files can contain arrays with arbitrary shapes. Also for large arrays, this format is more compact (binary + compression), which is useful when dealing with large arrays. The downside is obviously that NPZ files are not human-readable. Would this be a feature of interest for the project? Are there currently other ways of handling large (say 10^6 elements) and/or high-dimensional arrays?

num_regression fixture now requires pandas to be installed

It seems that after #35, num_regression now also requires pandas, instead of just numpy as before.

With a virtual environment with just numpy and pytest-regression, this simple test fails:

def test(num_regression):
    num_regression.check([])
        try:
            import numpy as np
        except ModuleNotFoundError:
            raise ModuleNotFoundError(import_error_message("Numpy"))
        try:
            import pandas as pd
        except ModuleNotFoundError:
>           raise ModuleNotFoundError(import_error_message("Pandas"))
E           ModuleNotFoundError: 'Pandas' library is an optional dependency and must be installed explicitly when the fixture 'check' is used

.pr\lib\site-packages\pytest_regressions\num_regression.py:73: ModuleNotFoundError

Would it be possible to remove that requirement? cc @648trindade

formatting should not matter in comparaison

I started using pytest-regressions in sphinx repository where we mostly build html content. this time I'm usinge the data_regression feature and I really like it.

THe problem is that my libs are always parsed by a prettier pre-commit hook which does the following for yaml files:

  • use double quotes
  • indent with 2 spaces

The consequence is that comparaison performed by the lib is failing because the 2 files are effectively different. I think that's the same provblem as #50, we want to compare data and in the background pytest-regressions is comparaing files (which implies that formatting and order are taken into account when they shouldn't).

Would it be possible to change paradigm and instead of transforming data to a file and then compare, read the source first as a dict and compare them with the data ?

Happy to help building up a PR if someone that knows the lib better confirms it could be possible

2.2.0: pytest with relaxed extension fails with deprecation error

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-regressions-2.2.0-2.1.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-regressions-2.2.0-2.1.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-regressions-2.2.0, configfile: tox.ini
plugins: regressions-2.2.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, toolbox-0.5, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, pyfakefs-4.5.0, flaky-3.7.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, cases-3.6.3, hypothesis-6.14.4, xprocess-0.18.1, black-0.3.12, checkdocs-2.7.1, anyio-3.3.0, Faker-8.11.0, asyncio-0.15.1, trio-0.7.0, httpbin-1.0.0, subtests-0.5.0, relaxed-1.1.5
collected 0 items / 1 error

================================================================================== ERRORS ==================================================================================
______________________________________________________________________ ERROR collecting test session _______________________________________________________________________
Direct construction of SpecModule has been deprecated, please use SpecModule.from_parent.
See https://docs.pytest.org/en/stable/deprecations.html#node-construction-changed-to-node-from-parent for more details.
========================================================================= short test summary info ==========================================================================
ERROR
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================= 1 error in 0.27s =============================================================================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.