Rationale

pyfunnel is a collection of algorithms to compare result timeseries from hybrid simulations (continuous time, discrete event). It is used in BuildingsPy's regression tests. These algorithms are particularly well suited to compare results when there are events and discontinuities in the result timeseries.

CSV Result Compare uses similar algorithms like pyfunnel.

Currently the mopyregtest.metrics collection has no similar built-in metric that allows for error localization in time and that is robust against events occuring at slightly different times.

Proposed feature

Make

• a proof of concept and
• a blueprint for other users that uses pyfunnel as a user-defined comparison metric

I think it makes more sense to fall back to an existing an proven solution instead of reproducing a similar functionality natively in MoPyRegtest.

The test test_unify_timestamps_multiplicity in test_compare_results.TestCompareResults fails in the job Unit test job.

Timestamps as computed by RegressionTest._unify_timestamps are not sorted as they should be.

Reason might the version of pandas and numpy used in the Ubuntu package repositories (python3-pandas) and in pip, producing different behavior in pandas.DataFrame.fillna.

Problem description

MoPyRegtest calls RegressionTest._unify_timestamps by default in RegressionTest.compare_result. This is convenient, but in some scenarios the implemented timestamp unification might prove unnecessary (e.g. for integral based error metrics), or unsuitable.

Desired behavior

Users shall have the option to disable timestamp unification and compare results from actual and reference results by means of their own metric definition. Even if the results have different timestamps.

Problem description

The testgen.py script is not included in the v0.3.1 release files on PyPI. Whereas the Generator class to automatically generate regression tests with a script is included.

To use the command line tool testgen.py, a user would have to get it from the sources, e.g. by cloning the repository.

Desired behavior

Give the user a command line tool in the release to generate regression test definitions.

Problem description

The command line interface fails to generate test definition files if the Modelica model can't be translated/simulated.

To Reproduce

The automated test definition generation hides simulation errors for MyLibrary.M:

package MyLibrary "Package"
  model M "Model"
    Real x(fixed=true, start=0);
  equation
    der(x) = -0.5*x + sqrt(0.5 - time);  // Breaks simulation at time=0.5
  end M;
end MyLibrary;

> mopyregtest generate .CI/mopyregtest/ "TestM" . MyLibrary.M
Simulating model MyLibrary.M using the simulation tools: omc
Using simulation tool omc

The resulting reference file .CI/mopyregtest/references/MyLibrary.M_res.csv is missing half of the simulation, but mopyregtest exited without any errors.

System information (please complete the following information):

• OS: Ubuntu 22.04.3 LTS
• OpenModelica: v1.23.0-dev-203-g5904d1ea84-cmake
• Python: v3.10.12
• MoPyRegtest: v0.5.0

Desired behavior

Return an error when the simulation for reference file generation failed.
Print or log the error message returned by OpenModelica.

So something like:

> mopyregtest generate .CI/mopyregtest/ "TestM" . MyLibrary.M
Simulating model MyLibrary.M using the simulation tools: omc
Using simulation tool omc

Failed to simulate model MyLibrary.M

LOG_STDOUT        | warning | A Modelica assert prevents the integrator to continue. For more information use -lv LOG_SOLVER
LOG_STDOUT        | warning | can't continue. time = 0.500000
LOG_STDOUT        | info    | model terminate | Integrator failed. | Simulation terminated at time 0.5

The full log can be viewed at some/path/MyLibrary.M.log
# exit code 1

Problem description

Writing a test_<...> method for every regression test can be a significant coding effort for a large number of regression tests, despite the code looking very similar each time.

Desired behavior

Instead of manually writing a test_<...> method for every regression test, it would be convenient to tell MoPyRegtest to simply turn certain elements of a Modelica library into a regression test. The resulting MoPyRegtest definition should be generated automatically.

Problem description

Right now a generated result for a model is compared to a reference result up user-defined precision. This is quite restrictive.

Desired behavior

Instead, users should be able to select from

• some predefined metrics like max-norm; euclidean distance of result vectors, i.e. $\ell^2$; $L^2$-norm
• user-defined result metrics passed as a function handle

Problem description

The setup.py currently does not include the project dependencies. So on pip install --user . the dependencies are not resolved. Thus, if the dependencies are not already satisfied, mopyregtest will not work out of the box.

Desired behavior

On installation via pip all the required dependencies should be installed automatically.

Problem description

• As of v0.2.2, the unit tests as well as the pipeline examples in .github/workflows are for on Ubuntu Linux only
• The openmodelica/openmodelica:v1.21.0-minimal is also based on Ubuntu

Desired behavior

• Create a test pipeline that runs the unit tests Windows, too

Problem description

Describe the bug
The GitHub Action for the unit tests fails, according to the logs as a consequence because of the unit tests for Python version 3.11 are failing. Apparently, pip is not found on the default installation for Python 3.11 that comes with the setup-python action.

To Reproduce
Run the unit tests associated with the file .github/workflows/job-unit-tests.yml

System information (please complete the following information):

• OS: ubuntu-latest
• OpenModelica version: 1.21
• Python version: 3.11
• MoPyRegtest version: 819b16d

Desired behavior

Unit tests shall be exectued.

Problem description

As of v0.2.2, to install MoPyRegtest users have to clone the repository and install locally by checking out the required version and then run pip install --user .

Desired behavior

• Users want to be able to install MoPyRegtest simply by running pip install mopyregtest and getting MoPyRegtest from the Python Package Index (PyPI).
• In the future, users want to be able to switch between versions using pip like pip install mopyregtest==0.2.1

Problem description

The command line interface fails to generate test definition files if the Modelica model can't be translated/simulated.

To Reproduce

The automated test definition breaks for MyLibrary.M:

package MyLibrary "Package"
  model M "Model"
    Real x(fixed=true, start=0);
    real y;  // Syntax error and number of variables not matching number of equations
  equation
    der(x) = -0.5*x;
  end M;
end MyLibrary;

> mopyregtest generate .CI/mopyregtest/ "TestM" . MyLibrary.M
Simulating model MyLibrary.M using the simulation tools: omc
Using simulation tool omc
/bin/sh: 1: ./MyLibrary.M: not found
Traceback (most recent call last):
  File "/home/USERNAME/.local/bin/mopyregtest", line 8, in <module>
    sys.exit(main())
  File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/cli.py", line 136, in main
    parse_args(sys.argv[1:])
  File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/cli.py", line 129, in parse_args
    args.func(args)
  File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/cli.py", line 48, in generate
    gen.generate_tests(test_folder, test_name, result_folder, references)
  File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/generator.py", line 237, in generate_tests
    self._generate_reference(reference_folder=test_folder / "references",
  File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/generator.py", line 175, in _generate_reference
    shutil.copyfile(pathlib.Path(tmp_res_folder) / res_file, str(pathlib.Path(reference_folder) / res_file))
  File "/usr/lib/python3.10/shutil.py", line 254, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/MyLibrary.Mae09u5nf/MyLibrary.M_res.csv'

System information (please complete the following information):

• OS: Ubuntu 22.04.3 LTS
• OpenModelica: v1.23.0-dev-203-g5904d1ea84-cmake
• Python: v3.10.12
• MoPyRegtest: v0.5.0

Desired behavior

Return a clear error message telling the user that:

• the test definitions couldn't be generated for model MyLibrary.M.
• print the error message from OpenModelica, giving a hint what the issue is and how to fix it.

The script should exit gracefully instead of failing to open some file and ideally clean up all half-generated files like .CI/mopyregtest/references/MyLibrary.M_res.csv and .CI/mopyregtest/test_testm.py.

So something like:

> mopyregtest generate .CI/mopyregtest/ "TestM" . MyLibrary.M
Failed to generate test definition for PNlib.TA:
Failed to build model:

[/path/to/MyLibrary/package.mo:4:5-4:11:writable] Error: Class real not found in scope M.

Aborting test definition generation.
Removing temporary files.
# exit code 1

Problem description

The previous de-facto standard to package Python projects using setuptools and a setup.py is now being replaced by pyproject.toml. See the official Python packaging guide.

Desired behavior

Move from setup.py to pyproject.toml to be future proof.

Problem with pyproject.toml

With the default installation of pip under Ubuntu 22.04, the local installation process pip3 install --user . fails. It installs a package called UNKOWN. This bug is already known and no other fix than installing a newer version of pip has been proposed.

A pyproject.toml for MoPyRegtest could look like

[project]
name = "MoPyRegtest" # Required
version = "0.4.0-alpha.1" # Required
description = "A Python enabled simple regression testing framework for Modelica models" # Optional
readme = "README.md" # Optional
requires-python = ">=3.8"
license = {file = "LICENSE"}
keywords = ["Modelica", "regression", "test", "automation"]  # Optional
authors = [
    {name = "Philipp Emanuel Stelzig"} # Optional
]
maintainers = [
    {name = "Philipp Emanuel Stelzig"} # Optional
]
classifiers = [  # Optional
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: Microsoft :: Windows :: Windows 10",
    "Operating System :: POSIX :: Linux"
]
dependencies = [ # Optional
    "numpy",
    "pandas"
]

[project.urls]  # Optional
"Homepage" = "https://github.com/pstelzig/mopyregtest"
"Bug Reports" = "https://github.com/pstelzig/MoPyRegtest/issues"
"Source" = "https://github.com/pstelzig/mopyregtest"

[project.scripts]  # Optional
mopyregtest = "mopyregtest.cli:main"

[tool.setuptools]
package-data = {"mopyregtest" = ["templates/omc/model_import.mos.template",
                                 "templates/omc/model_simulate.mos.template"]}

[build-system]
requires = ["setuptools>=43.0.0", "wheel"]
build-backend = "setuptools.build_meta"

Setup a CI pipeline with GitHub actions to automize calling both the unit tests and the examples

Rationale

Numpy 2 is about to be released with some breaking changes. MoPyRegtest may not break as a result of users having Numpy 2 instead of Numpy 1 installed.

Proposed feature

Remove all deprecated and obsolete elements of Numpy that are no longer present in Numpy 2, like np.trapz in test_user_defined_metrics.py

ToDos

• Remove np.trapz

Problem description

As of v0.1.2 all output from unittest when running MoPyRegtest defined tests are printed to the shell.

Desired behavior

User shall be able to create a log file with test results.

Problem description

Right now, all tests generated from a mopyregtest.Generator object use the same metric for all tests it generates. However, it could make sense to use different metrics for different library elements. Right now, this can only be achieved by creating regression test definitions manually, or use different mopyregtest.Generator objects (each of which generates its own test definition file).

Desired behavior

In mopyregtest.Generator have to option to use dedicated comparison metrics for individual Modelica models, and use a default metric for all others.

Potential solution

Use the metric passed to the construtor of mopyregtest.Generator as the default metric, and pass dedicated metrics in the mopyregtest.Generator.generate_tests method, just like with the pre-existing references as a dictionary mapping model name to metric to be used.

Problem description

In RegressionTest.compare_result, the for loop over the entries of validated_cols stops at the first column that exceeds the tolerance. All subsequent columns will not be checked.

for c in validated_cols:
    print("Comparing column \"{}\"".format(c))
    delta = metric(ref_data[["time", c]].values, result_data[["time", c]].values)
    if np.abs(delta) >= tol:
        raise AssertionError(
            f"Values in Colum {c} of results {simulation_result} and {reference_result} differ by " \
            f"{np.abs(delta)} which is larger than {tol}.")

Desired behavior

While this is efficient for continuous integration (test must fail if any column comparison fails), it desirable from a model developer perspective to

• have all columns in validated_cols checked, and
• get an error thrown if at least one column comparison has failed
• get an list of all columns in validated_cols for which the comparison has failed

Problem description

• As of v0.2.2, if a regression test fails, the output is restricted to Python's unittest standard output.
• The log output states, how many tests passed. There is no test summary for which test cases and which variables the regression failed, and the user has to manually compare the reference result and the output of the actual simulation run (if it produced any output that RegressionTest.compare_result could process)

For any failed regression tests, the output of the respective actual simulation run is not cleaned up (good behavior).

Desired behavior

• In each result folder of a failed test include a comparison csv with a tree structure like

  root
    |
    + -- reference
              |
              +...
    + -- actual
              |
              +...
    + -- failed
              |
              + var_failed_0
                  |
                  + reference
                  |
                  + actual
              |
              + var_failed_1
                  |
                  + reference
                  |
                  + actual
  

• Implement a test summary, see #2

Problem description

Right now, a user would have to go to the Actions tab to see which workflows pass and which fail.

Desired behavior

Add badges for the project status, like

• unit test status
• example passing

Problem description

If a reference result and the actual result have no columns (i.e. variable names in the result CSV files) in common and the default argument validated_cols=[] is passed to RegressionTest.compare_result is used, the intersection of column names from reference and actual result will be taken. The intersection in this case is just "time" and the test will pass.

Desired behavior

• Do not treat "time" as a result variable column name
• Throw an error if the default argument is used for validated_cols and there are no common columns
• Check if validated_cols contains elements, and, if so, that all its entries are present in the actual and the reference result. Otherwise throw an error.
• Add unit tests for all of the above new features

Problem description

As of v0.2.2, tagging, testing and building is entirely manual and there are no releases yet, just tags.

Desired behavior

• Have dedicated releases instead of just tags
• Be able to build a release and automatically push that release to PyPI (might still be manually triggered)

Problem description

As of commit b89de92, the mopyregtest command line has only one functionality, namely to automatically generate regression test definitions.

Desired behavior

Instead, mopyregtest shall be designed as a fully functional command line tool, like e.g. the conan C++ package manager's cli

Design

The mopyregtest command line tool will have other usages than just test generation, like e.g.

mopyregtest generate --metric=Lp_dist --references=Model3:Model3_res.csv /path/to/gen/target MyGenTests /path/to/package Model1,Model,Model3
mopyregtest compare --metric=Lp_dist --vars=var1,var3 /path/to/csv_1.csv /path/to/csv_2.csv

or more generally

mopyregtest generate [-h] [--metric {norm_p_dist,norm_infty_dist,Lp_dist,Linfty_dist}] [--references REFERENCES]  test_folder test_name package_folder models_in_package
mopyregtest compare [-h] [--metric {norm_p_dist,norm_infty_dist,Lp_dist,Linfty_dist}] [--vars VARS] csv_files_1 csv_file_2