modelica-tools / csv-compare Goto Github PK

Since CSVcompare is typically used in CI toolchains for verifying non-regression of results, it would make sense to also generate a Junit XML summary of the comparisons.

Some models in the MSL show chaotic motion. It's good to see the chaotic motion in the simulation, so we set StopTime long enough to see that, but when issuing reference results, these should be computed on a time span which is short enough to avoid bifurcations and the onset of chaos, which is theoretically impossible to match with finite-precision numerical integration.

An easy solution to this problem is to generate reference files that are truncated to a shorter end time than the StopTime annotation of the model.

In this case, the CSV compare tool should only compare the trajectories over the time span that overlaps both the simulation results and the reference file. It already kind of does it, but the tubes at the end of the reference file time span are warped, so the test fails. See #4341

This should be fixed.

Keeping @beutlich and @GallLeo in the loop.

> compare.exe file.csv file_base.csv
2019-07-18Z15:00:35 [       Error ] Exception during run: The path is not of a legal form.

The problem is that it tries to deduce the report directory from the first .csv file. Here there is no leading directory so Path.GetDirectoryName(options.Items[0]) is empty string. Then Path.GetFullPath("") causes an exception. You should deal with this case and in case of empty string passed to GetFullPathWithEndingSlashes you should ask for current working directory Path.GetFullPath(".")

Next is the current logic from Program.cs that needs fixing:

options.ReportDir = GetFullPathWithEndingSlashes(Path.GetDirectoryName(options.Items[0]));
....
private static string GetFullPathWithEndingSlashes(string input)
       {
           string fullPath = Path.GetFullPath(input);
           return fullPath.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) + Path.DirectorySeparatorChar;
       }

Running Compare.exe with the following input:

Compare.exe -r . --override result.csv reference.csv

where result.csv contains:

time;x
0.0;0.0
1.0;0.0
2.0;0.0

and reference.csv contains:

time;x
0.0;0.0
2.0;0.0

will give the following output:

2018-04-12Z15:51:42 [     Warning ] The resolution of the base x-axis is smaller
than the compare x-axis. The better the base resolution is, the better the
validation result will be!

It will also generate an empty file in windows temporary folder. An example file name is: tmpB053.tmp

Expected behavior is to not get any file in the temporary folder.

This was found in version 2.0.0.1

mode csvFileCompare compares files but before writing report the error "Paths do not have the same base" occurs even though paths are ok. Same setup with version 2.0.0 works...

We previously created this ticket. It was fixed with the new algorithm. However we still see some odd behaviours, probably related to input handling.

If you check the result points you see that the result points at t = 0.1 has been swapped.

Both the base and result has been swapped in the example above, however there are other places in that signal where only the result is swapped, t=~0.148, t=~0.1965, t=~2446, etc.

Command:

Compare.exe -o -m csvFileCompare -d , actual.csv expected.csv
Files can be found here
This was found using this version

First of, the new version of CSV compare is much better!

The behaviour for the error graph has changed since previous versions of csv-compare (pre 2.0?).
Now it only plot the graph for the part that has an error.
I don't know if this is intended or not but it causes problems when there's only a one or two error points:

All files that was used can be found here, and the command was:

Compare.exe -o -m csvFileCompare -r . -t 0.001 --inlineactual.csv target.csv
2015-03-20Z07:53:27 [ Warning ] aimc.idq_rr[2] is invalid! 1 errors have been found during validation.
2015-03-20Z07:53:27 [ Warning ] aimc.idq_rr[1] is invalid! 2 errors have been found during validation.
2015-03-20Z07:53:27 [ Warning ] aimc.idq_sr[1] is invalid! 1 errors have been found during validation.

Version:

Compare.exe --help
CSV File Comparison Tool 2.0.0.1
Copyright © 2015 - ITI GmbH

It would be really nice if the two graphs was synced, so if you zoomed in one, the other one was zoomed as well, in x-axis that is.

We had many false regressions in the MSL on signals that are supposed to be zero but aren't, due to numerical errors, see modelica/ModelicaStandardLibrary#4421

One solution is to remove those signals outright, but then you can't really make sure that they are close to zero, so that's not a good solution in general, unless those close-to-zero signals are somehow redundant.

@HansOlsson in that ticket suggests a pragmatic but actually quite effective fix for this problem: set the tube width not only based on a relative tolerance, but also on some a-priori nominal value for signals, e.g. 1e-3. This means that the error criterion should be something like:

(v - v_ref)/max(abs(v), nom) < tol

with nom = 1e-3 and tol = 0.002, the current setting for MSL testing.

@beutlich could you please take care of that at your earliest convenience?

Keeping @GallLeo in the loop.

from: https://trac.modelica.org/Modelica/ticket/1646

I tried running the Compare tool (r8061) with a simple csv file using the command

Compare.exe Example.csv Example.csv

This however gives an error

2015-01-20Z09:45:36 [       Error ] Exception during run: Input string was not in a correct format.

where the problematic part seems to be the tolerance as the below command works fine,

Compare.exe -t 0,001 Example.csv Example.csv

This needs to be fixed so that it works out of the box. Additionally this dependency on the locale makes it difficult to use the Compare tool in other tools.

I am working with a set of signals of varying magnitude where one signal is very small (and very small variation). Since there seem to be some limitation on the ratio of sides of the rectangle used to calculate the tube (I saw a max statement with 0.0004 in the ratio calculation) this allows really big relative differences of small results.

As I understand the setting of tolerance, it only affects the width of the tube on the x-axis, is there a way to specify the width of the tube on the y-axis instead? Preferably on a signal-to-signal basis (not all signals are small). Or can you improve the tube algorithm, there seems to be an attempt to this in the function 'SetStandardBaseAndRatio' in Modelica_ResultCompare/CurveCompare/TubeSize.cs but for some reason 'SetFormerBaseAndRatio' is used instead.

We have been seeing some issues with the false-negatives in the endpoints of some trajectory comparisons. It usually happens when the signal have a "high" derivative.
For example:

You can see that the bounds calculation doesn't account for the derivative of the trajectory at the end point. The two results are very close and should be accepted.

Example files can be found here:
https://gist.github.com/jon-modelon/656e05349415bb2776a8

Command:

Compare.exe -o -m csvFileCompare -r . -t 0.001 --inline actual.csv expected.csv

Version:

Compare.exe --help
CSV File Comparison Tool v2.0.0-rel
Copyright ? 2015 ITI GmbH

Is there any chance that the support for NaN and Infinity will improve in the near future?
In the case of NaN an error is thrown, but for Infinity the behavior is different depending where the Infinity is present, in the base.csv or in the to_compare.csv.

CSV Compare does not skip variables which aren't present in both csv files in some cases.
For example the following base.csv:

time;a;b;c
0;1;2;3
1;2;3;4
2;3;4;5
3;4;5;6
4;5;6;7

and actual.csv:

time;a;x;c
0;1;0;3
1;2;0;4
2;3;0;5
3;4;0;6
4;5;0;7

Note that b exists in base.csv but not in actual.csv, and x is missing from base.csv but exists in actual.csv. In the generated report the variable b is listed with a 100% success rate. This is a rather severe bug since the user get the impression that everything is fine and the variable produces the correct result! It does however leave a warning in the log, saying that b has been skipped (which it apparently haven't).
Log:

2015-07-07Z13:16:51 [     Warning ] b not found in "actual.csv", skipping checks.

Command used:

Compare.exe actual.csv base.csv -r out

Version:

CSV File Comparison Tool v2.0.0-rel
Copyright ? 2015 ITI GmbH

When this csv file is processed by CSVCompare as both base and target, CSVCompare report errors. The upper and lower bounds starts to behave incorrectly.

Command used:

Compare.exe -o -m csvFileCompare -t 1e-3 -d , res.csv res.csv

Version:

Compare.exe -v
CSV File Comparison Tool v2.0.0-rel
Copyright ? 2015 ITI GmbH

If the header of the first column is not named time but e.g., time[s] the Compare.exe reports missing y-values for the last column. This error message is really misleading and should be improved to point to the first column instead.

Compile and test the current code on a linux test system comparing the MSL regression testing tree with itself.

Hello,

I have an issue that almost certainly is due to a lack of knowledge on using visual studio and applications. I've downloaded the files from GitHub and once I try to open the solution and build the project, I get an error "The program '[21536] Compare.exe' has exited with code 2 (0x2)."

Is there a way that you could elucidate me a bit more on how to run the application?

Thanks

source: https://trac.modelica.org/Modelica/ticket/1486

If I compare all test cases of MSL in tree mode, compare.exe needs about 5 GB of RAM.
On my 8 GB machine, I have to close applications before being able to start the tree compare operation.

If the number of test cases increases in future, it's going to get critical on standard PCs.

It should be investigated if RAM usage can be reduced.
E.g. by writing reports to hard disk during tree compare.
Currently it seems to write all reports at the end.

Used version: compare.exe 64bit r7267

Used call:

compare.exe --mode csvTreeCompare --tolerance 2e-3 --delimiter "," --verbosity 2 C:\Work\upload\RegressionTesting\MSL\ReferenceResults\v3.2.1+build.2.release\Modelica C:\Work\upload\RegressionTesting\MSL\ReferenceResults\v3.2.1+build.2.release\Modelica --logfile log.txt --comparisonflag --reportdir 07_r7267_Modelica

We're getting "Exception during run: Index was outside the bounds of the array."
Command used is:

Compare.exe -o -m csvFileCompare -r . -t 0.001 --inline res.csv base.csv

And the files can be found here

Exact error message is:

2015-05-27Z10:58:36 [ Warning ] The resolution of the base x-axis is smaller than the compare x-axis. The better the base resolution is, the better the validation result will be!
2015-05-27Z10:58:36 [ Warning ] a is invalid! 219 errors have been found during validation.
2015-05-27Z10:58:36 [ Error ] Exception during run: Index was outside thebounds of the array.

The two warnings are rather strange, the two result files has different resolution during different time (due to events).
It would also be rather interesting to know why "a" is invalid...

Version of CSV Compare:

CSV File Comparison Tool v2.0.0-rel

CSV Compare Version 2.0.0.1 (X86)
Comparison result file for C:\temp\actual.csv
. Time:        2015-04-15T14:02:28.7713039+02:00
. Operation:   CsvFileCompare
. Tolerance:   0.001
. Result:      failed
. Biggest error: actual.a=>2,49843247137984E-07
. Failed values:
actual.a=>2,49843247137984E-07

Tolerance is locale-independent (as expected) but error value is not.

I am unable to use the --tolerance option in a meaningful way, as the mapping of the specified value to the actual shape of the tube is completely opaque (to my mind at least).

Here's a capture of a tube created with --tolerance=0.05:

Can you please explain a bit more how the --tolerance works? The short mention in the help is hardly informative enough...

When I run Compare.exe such as:
Compare.exe -o -m csvFileCompare -r ".\output" ".\data\actual.csv" ".\data\expected.csv"

The files can be found at: https://gist.github.com/JohanYli/cc6a9bb6f8d7e106ac9f

I get the error:
[ Error ] Exception during run: Index was outside the bounds of the array.

I use version 2.0.0.1 of the CSV File Comparison Tool and run it on a Windows 7 machine.

Since r8061 (on the MA svn) we get the following errors when a variable is missing in the result file.
Command used is

Compare.exe -o -m csvFileCompare res.csv base.csv

Output is:

2015-01-23Z15:02:43 [ Warning ] No report directory has been set, using ""
2015-01-23Z15:02:43 [ Warning ] actuator.armature.mass.der(s) not found in "
C:\Users\jon_041\Downloads\res.csv", skipping checks.
2015-01-23Z15:02:43 [ Error ] Exception during run: Object reference not set to an instance of an object.
CSV File Comparison Tool 2.0.0.8061
Copyright © 2014 - ITI GmbH

This software is free software and released under The BSD 3-Clause License
...

@svenruetz Is there any detailed document on the CSV-compare tool? It would be great to help us to understand purpose of each function.

We have to check if the used jqplot library supports a different way of reading its data sets to be able to set a variable with the time values and use this variable instead of repeating vectors in the report.

Maybe the series to plot could be initialized seperately.

The make file has to be changed to automatically set the correct version in AssemblyInfo.cs. Or the build rule has to be excluded from the make file.

The generated reports use absolute file:// URLs. This makes it impossible to move the files or to serve them via http. An option should be available to use relative URLs instead.

On Win8, Compare.exe reports Exception during run: Illegales Zeichen im Pfad. (for default --mode=CsvFileCompare) if the report directory specified by --reportDir has a trailing backslash. Would be nice if this could be considered as valid input.