modelica-3rdparty / externdata

:page_facing_up: Modelica library for data I/O of CSV, INI, JSON, MATLAB MAT, SSV, TIR, Excel XLS/XLSX and XML files

ExternData

Free Modelica library for data I/O of CSV, INI, JSON, MATLAB MAT, SSV, TIR, Excel XLS/XLSX and XML files.

Status

Library description

ExternData is a utility library to access data stored in CSV, INI, JSON, MATLAB MAT, SSV, TIR, Excel XLS/XLSX or XML files. The aim of this library is to provide access from Modelica simulation tools to data sets for convenient model initialization and parameterization. An overview of the library is provided in

Thomas Beutlich and Dietmar Winkler. Efficient Parameterization of Modelica Models. In: Proceedings of the 14th International Modelica Conference. Ed. by Martin Sjölund, Lena Buffoni, Adrian Pop, and Lennart Ochel. Linköping, Sweden, September 2021. DOI: 10.3384/ecp21181141.

Main features

• Read support of file formats
  • CSV
  • INI
  • JSON
  • MATLAB MAT of version v4, v6, v7 and v7.3 (HDF5)
  • SSV (System Structure Parameter Values v1.0)
  • TIR (Tire properties)
  • Excel XLS and XLSX
  • XML (incl. XPath queries)
• C (and not C++) code for external functions and objects
• Cross-platform (Windows and Linux)
• Dependency on the Modelica Standard Library v4.0.0
• Tested in Dymola, OpenModelica and SimulationX

All data I/O access is implemented using external Modelica functions.

License

ExternData is released under the terms of the Simplified BSD License.

Acknowledgement

ExternData is based on the following third-party C projects

• bsxml-json - Borislav Sapundzhiev's fairly simple XML DOM and JSON implementation
• expat - James Clark's Expat XML parser library
• hdf5 - The HDF Group's data model, library and file format for storing and managing data
• inih - Ben Hoyt's simple INI file parser in C
• libxls - David Hoerl's C library for parsing Excel files
• libxml2 - Daniel Veillard's XML C parser and toolkit of Gnome
• matio - Christopher Hulbert's C library for reading and writing MATLAB MAT-files
• minizip - Gilles Vollant's Zip and UnZip library
• parson - Krzysztof Gabis' lightweight JSON library written in C
• uthash - Troy D. Hanson's C macros for hash tables and more
• zlib - Jean-loup Gailly's and Mark Adler's massively spiffy yet delicately unobtrusive compression library
• zstring - Fehmi Noyan ISI's string processing library for C

Citing

Use the following BibTeX lines to cite the ExternData library.

@InProceedings{ExternData,
  title = {{Efficient Parameterization of Modelica Models}},
  author = {Beutlich, Thomas and Winkler, Dietmar},
  pages = {141--146},
  doi = {10.3384/ecp21181141},
  booktitle = {Proceedings of the 14th International Modelica Conference},
  location = {Link\"oping, Sweden},
  editor = {Sj\"olund, Martin and Buffoni, Lena and Pop, Adrian and Ochel, Lennart},
  isbn = {978-91-7929-027-6},
  issn = {1650-3740},
  month = sep,
  series = {Link\"oping Electronic Conference Proceedings},
  number = {181},
  publisher = {Modelica Association and Link\"oping University Electronic Press},
  year = {2021}
}

Development and contribution

You may report any issues with using the Issues button.

Contributions in shape of Pull Requests are always welcome.

externdata's Issues

Missing ModelicaExternalC.dll in Modelica

I was excited to see this code available, however I get the following error when I load the Modelica code into Dymola and run a simulation on the XMLTest1 model:

The program can't start because ModelicaExternalC.dll is missing from your computer. Try reinstalling the program to fix this problem.

I can find ModelicaExternalC.lib, but not the dynamic library. Do I need to to change something in the ExternData code in order for it to look for the static library, not the dynamic library?

System specifics:
Windows 7 Enterprise SP1
Dymola 2015 and 2016 32-bit and 64-bit versions

Read Dynamically Set Arrays from .mat File

Hi there,

I'd like to load a dynamically sized array from a Matlab file. This is a similar issue to issue #34.

Below is a minimal example that fails. It looks like Dymola fails to expand the variable.

model MATTest2 "MAT-file read test"
  extends Modelica.Icons.Example;
  inner parameter ExternData.MATFile dataSource(fileName=Modelica.Utilities.Files.loadResource("modelica://ExternData/Resources/Examples/test_v7.3.mat")) "MAT file" annotation(Placement(transformation(extent={{-80,60},{-60,80}})));
  final parameter Integer m = dataSource.getArrayRows2D("table1") "Number of rows in 2D array";
  parameter Real matTable[:,:] = dataSource.getRealArray2D("table1", m, 2);
  annotation(experiment(StopTime=1),
    Documentation(info="<html><p>This example model reads the table parameter from variable table1 of the HDF5-based MAT-file <a href=\"modelica://ExternData/Resources/Examples/test_v7.3.mat\">test_v7.3.mat</a>. The table parameter is read as Real array of dimension 3x2 by function <a href=\"modelica://ExternData.dataSource.getRealArray2D\">ExternData.dataSource.getRealArray2D</a>. The read parameter is assigned by a parameter binding to the appropriate model parameter.</p></html>"));
end MATTest2;

I've tested this in Dymola 2022x, and I get an error stating:
Failed to expand the variable matTable
and its definition equation:
dataSource.getRealArray2D_Unique'"dataSource"'(
"table1",
m,
2,
dataSource.mat)

Is it possible to use parameter Real matTable[:,:] = dataSource.getRealArray2D(tableName, m, 2); instead of parameter Real matTable[:,:] = dataSource.getRealArray2D(tableName, 3, 2);

Is there a workaround for this? Any help here would be appreciated. Thank you for your time.

Suggestion: saving .mat time series data in CombiTimeTable format

Hi,

it would be useful to have a block that saves time series data to a .mat file while simulating. The block should have arrays of real/boolean/integer inputs, and for each input a corresponding parameter for the variable name. The input values can then be saved at regular time intervals T. The output should be a .mat file in a format that can be read by Modelica.Blocks.Sources.CombiTimeTable. This allows the creation of time series data that can be used again later in a different model.

Is this something that would fit in this library? Or possibly this exists already?

Thanks!

Error compiling example with dymola 2017 FD01

Hi tbeau! Thanks for putting this together. It's just what I need for a Dymola project I'm working on.

Background info:
I'm using Dymola 2017 FD01 with Visual C++ 2015 "Community" Compiler

The Error Message in Dymola that is thrown during compilation:
Compiling and linking the model (Visual C++).

  dsmodel.c
  c:\..........\github - externdata\externdata\resources\include\msvc_compatibility.h(60): 
  error C2084: function 'FILE *__iob_func(void)' already has a body
  c:\program files (x86)\dymola 2017 fd01\source\dsutil.h(1958): 
  note: see previous definition of '__iob_func'

  Error generating Dymosim.

I can see that each file listed has the function defined in it... so it makes sense that there is a conflict, but I'm unsure of what I should do to correctly "fix" it.

Any chance you can take a look?

Cheers!

Make library working in OpenModelica

@sjoelund Whereas the library works in Dymola and SimulationX, it is currently not working with OM as stated (again) at SO. Please see the ExternData.Examples.XMLTest1 from the OM-test branch for a simple test.

What shall I do to make it compatible with OM? Or is this a issue of OM itself?

@dietmarw Thanks for the SO link.

throw error when variable not found

The line

ExternData/ExternData/Resources/C-Sources/ED_JSONFile.c

Line 136 in 2fb7395

ModelicaFormatMessage("Cannot find numeric value \"%s\" in file \"%s\"\n",

throws a message when a double cannot be found in a json file. It seems that the function then returns 0 as a default for that variable. When this message is overlooked by the user, it can result in completely wrong results that may go unnoticed. I would suggest to use ModelicaFormatError instead of ModelicaFormatMessage, or at least provide this as an optional flag. Note that for arrays an error is already thrown:

ExternData/ExternData/Resources/C-Sources/ED_JSONFile.c

Line 356 in 2fb7395

ModelicaFormatError("Cannot find array value \"%s\" in file \"%s\"\n",

Dymola 2019 model check fails on class methods

It is found that in Dymola 2019, modelcheck would fail on ExternData reader class methods, e.g. XMLFile.getReal, JSONFile.getReal etc. It seems that Dymola 2019 has changed their model check routine. It is complaining about not able to find the constructed object when doing a individual model check on the methods.

Check of ExternData.JSONFile.getReal:

Undeclared variable: ExternData.JSONFile.json
in the definition equation
json = ExternData.JSONFile.json

Failed in variable declarations of:
function ExternData.JSONFile.getReal 
input String varName "Key";
output Real y "Real value";
output Boolean exist "= true, if varName exits; = false, if it does not exist and y is set to 0.0";
input ExternData.Types.ExternJSONFile json := ExternData.JSONFile.json;
end ExternData.JSONFile.getReal;

Errors detected in functions.

Check aborted.

ERRORS have been issued.

• ExternData version: 2.3.4
• Dymola version: 2019
• OS: Windows 10 Pro

OpenModelica

Hi all,

I read somethings about the OpenModelica troubles. My question, is the library working in openmodelica at all? Or is it just the examples that don't work?

kind regards

Compilation fails, linker cannot find ED_JSONFile [linux/macos]

Hi all,

I'm trying to build an FMU from a Modelica model. Some of the parameters are read in Modelica from a .json-file, others from a .csv file.

This happens on various systems. My host system is a M1 Mac running macOS Ventura 13.2
I installed OpenModelica using brew. omc --version prints:
OMCompiler v1.18.0

Afterwards I ran this script to install the Modelica and ExternData libraries:

installPackage(Modelica);
installPackage(Complex);
installPackage(ModelicaServices);
installPackage(ExternData);

Now, I want to build my FMU with the following script (executed in the package folder):

loadModel(Modelica);getErrorString();
loadModel(ExternData);getErrorString();
loadFile("package.mo");getErrorString();
setCommandLineOptions("--fmiFlags=s:cvode --linkType=static --fmuRuntimeDepends=all");getErrorString();
cd("/home/Modelica/res");getErrorString();  // (note: the folder exists)
buildModelFMU(PVexample.Houseexample, version="2.0", fmuType="cs", fileNamePrefix="Houseexample");getErrorString();

When running the script on macOS, this is the resulting error:

[...]
mkdir -p ../binaries/aarch64-darwin
clang -shared -o Houseexample.dylib Houseexample.o Houseexample_functions.o Houseexample_records.o Houseexample_01exo.o Houseexample_02nls.o Houseexample_03lsy.o Houseexample_04set.o Houseexample_05evt.o Houseexample_06inz.o Houseexample_07dly.o Houseexample_08bnd.o Houseexample_09alg.o Houseexample_10asr.o Houseexample_11mix.o Houseexample_12jac.o Houseexample_13opt.o Houseexample_14lnz.o Houseexample_15syn.o Houseexample_16dae.o Houseexample_17inl.o Houseexample_18spd.o Houseexample_init_fmu.o Houseexample_FMU.o  simulation/solver/cvode_solver.o simulation/solver/sundials_error.o -L\"/Users/alex/Diplomarbeit/Modelica/packages/PVexample\"   \"-L/opt/homebrew/Cellar/openmodelica/1.18.0_6/lib/aarch64-darwin21.6.0/omc\" \"-L/opt/homebrew/Cellar/openmodelica/1.18.0_6/lib/\" \"-L/Users/alex/.openmodelica/binaries/PVexample\" \"-L/Users/alex/Diplomarbeit/Modelica/packages/PVexample/Resources/Library/aarch64-darwin\" \"-L/Users/alex/Diplomarbeit/Modelica/packages/PVexample/Resources/Library\" -lED_JSONFile -lparson \"-L/Users/alex/.openmodelica/binaries/Modelica\" \"-L/Users/alex/.openmodelica/libraries/Modelica 4.0.0+maint.om/Resources/Library/aarch64-darwin\" \"-L/Users/alex/.openmodelica/libraries/Modelica 4.0.0+maint.om/Resources/Library\" -lModelicaExternalC -lModelicaStandardTables -lModelicaIO -lModelicaMatIO -lzlib \"-L/Users/alex/.openmodelica/binaries/ExternData\" \"-L/Users/alex/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/aarch64-darwin\" \"-L/Users/alex/.openmodelica/libraries/ExternData 3.0.3/Resources/Library\"  -L'/opt/homebrew/Cellar/openmodelica/1.18.0_6/lib/aarch64-darwin21.6.0/omc' -Wl,-rpath,'/opt/homebrew/Cellar/openmodelica/1.18.0_6/lib/aarch64-darwin21.6.0/omc'   -lSimulationRuntimeFMI  -lopenblas -lm  -lsundials_cvode -lsundials_nvecserial
ld: warning: directory not found for option '-L/Users/alex/.openmodelica/binaries/PVexample'
ld: warning: directory not found for option '-L/Users/alex/Diplomarbeit/Modelica/packages/PVexample/Resources/Library/aarch64-darwin'
ld: warning: directory not found for option '-L/Users/alex/Diplomarbeit/Modelica/packages/PVexample/Resources/Library'
ld: warning: directory not found for option '-L/Users/alex/.openmodelica/binaries/Modelica'
ld: warning: directory not found for option '-L/Users/alex/.openmodelica/libraries/Modelica 4.0.0+maint.om/Resources/Library/aarch64-darwin'
ld: warning: directory not found for option '-L/Users/alex/.openmodelica/libraries/Modelica 4.0.0+maint.om/Resources/Library'
ld: warning: directory not found for option '-L/Users/alex/.openmodelica/binaries/ExternData'
ld: warning: directory not found for option '-L/Users/alex/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/aarch64-darwin'
ld: library not found for -lED_JSONFile
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [nozip] Error 1

I checked if there were any missing files, but ExternData folder inside .openmodelica/libraries is identical to what I see here on GitHub.

So next I tried the amd64 docker container (openmodelica/openmodelica:v1.20.0-gui).
I installed the libraries with the same script as above, then ran the same script.
It finished and produced an FMU. I could not load this though because in the modelDescription.xml, every parameter that should have been read from the .json or .csv is missing the start value, therefore it cannot be initialized.
While compiling, I get this output several times:

[/root/.openmodelica/libraries/ExternData 3.0.3/package.mo:112:23-112:178:writable] Warning: Could not find library ED_JSONFile in either of:
  /usr/bin/../lib/x86_64-linux-gnu/omc/libED_JSONFile.a
  /usr/lib/x86_64-linux-gnu/libED_JSONFile.a
  /lib/x86_64-linux-gnu/libED_JSONFile.a
  /usr/lib//libED_JSONFile.a
  /lib//libED_JSONFile.a
  /usr/bin/../lib/x86_64-linux-gnu/omc/libED_JSONFile.a
  /usr/bin/../lib//libED_JSONFile.a
  /root/.openmodelica/binaries/PVexample/libED_JSONFile.a
  /home/Modelica/PVexample/Resources/Library/x86_64-linux/libED_JSONFile.a
  /home/Modelica/PVexample/Resources/Library/linux64/libED_JSONFile.a
  /home/Modelica/PVexample/Resources/Library/libED_JSONFile.a
  /usr/bin/../lib/x86_64-linux-gnu/omc/libED_JSONFile.so
  /usr/lib/x86_64-linux-gnu/libED_JSONFile.so
  /lib/x86_64-linux-gnu/libED_JSONFile.so
  /usr/lib//libED_JSONFile.so
  /lib//libED_JSONFile.so
  /usr/bin/../lib/x86_64-linux-gnu/omc/libED_JSONFile.so
  /usr/bin/../lib//libED_JSONFile.so
  /root/.openmodelica/binaries/PVexample/libED_JSONFile.so
  /home/Modelica/PVexample/Resources/Library/x86_64-linux/libED_JSONFile.so
  /home/Modelica/PVexample/Resources/Library/linux64/libED_JSONFile.so
  /home/Modelica/PVexample/Resources/Library/libED_JSONFile.so
[/root/.openmodelica/libraries/ExternData 3.0.3/package.mo:112:23-112:178:writable] Warning: Could not find library parson in either of:
  /usr/bin/../lib/x86_64-linux-gnu/omc/libparson.a
  /usr/lib/x86_64-linux-gnu/libparson.a
  /lib/x86_64-linux-gnu/libparson.a
  /usr/lib//libparson.a
  /lib//libparson.a
  /usr/bin/../lib/x86_64-linux-gnu/omc/libparson.a
  /usr/bin/../lib//libparson.a
  /root/.openmodelica/binaries/PVexample/libparson.a
  /home/Modelica/PVexample/Resources/Library/x86_64-linux/libparson.a
  /home/Modelica/PVexample/Resources/Library/linux64/libparson.a
  /home/Modelica/PVexample/Resources/Library/libparson.a
  /usr/bin/../lib/x86_64-linux-gnu/omc/libparson.so
  /usr/lib/x86_64-linux-gnu/libparson.so
  /lib/x86_64-linux-gnu/libparson.so
  /usr/lib//libparson.so
  /lib//libparson.so
  /usr/bin/../lib/x86_64-linux-gnu/omc/libparson.so
  /usr/bin/../lib//libparson.so
  /root/.openmodelica/binaries/PVexample/libparson.so
  /home/Modelica/PVexample/Resources/Library/x86_64-linux/libparson.so
  /home/Modelica/PVexample/Resources/Library/linux64/libparson.so
  /home/Modelica/PVexample/Resources/Library/libparson.so
Warning: Could not resolve URI (uri) at compile-time; copying all loaded packages into the FMU
"

When running the executable in the terminal, I get the following:

stdout            | info    | ... loading "/home/dev/Modelica/packages/PVexample/Data/CH4.json"
stdout            | info    | ... loading "/home/dev/Modelica/packages/PVexample/Data/ExhaustGas.json"
stdout            | info    | ... loading "/home/dev/Modelica/packages/PVexample/Data/Air.json"
stdout            | info    | ... loading "/home/dev/Modelica/packages/PVexample/Data/ExhaustGas.json"
stdout            | info    | ... loading "/home/dev/Modelica/packages/PVexample/Data/CH4.json"
stdout            | info    | ... loading "table" from "/home/dev/Modelica/packages/PVexample/Data/HouseholdProfiles.csv"
assert            | debug   | Error in line 1 when reading numeric data of matrix "table(8762,5)" from file "/home/dev/Modelica/packages/PVexample/Data/HouseholdProfiles.csv"
assert            | info    | simulation terminated by an assertion at initialization

The same output is generated when using OMPython to start a simulation. When I call convertMo2Fmu(), I can see that files are being generated, but no .fmu file. The return in the console is just ' '.

I tried the same on an Ubuntu 22.04 VM using OMEdit. The results were - unsurprisingly - the same.

When I ran the package in OMEdit on a Windows machine, it worked perfectly fine. All I did was copying the package to a unix system (and I even ran dos2unix on the .csv files).
Please do not tell me that using Windows is the only way to go...

Where is my mistake here? Any help is gladly appreciated!

Thanks in advance

Fix problem with tir file example

We use the ExternData.TIRFile to read .tir files, but since the latest change to the library in revision 621 (updating binaries) this no longer works. The same error is seen in your example ExternData.Examples.TIRTest.

Issue warnings instead of ModelicaFormatError for missing paths

Personally, I actually prefer aborting the simulation if there's any missing path, but there are also some benefits to be discussed if we set a default value and continue the simulation, let's see if there's maybe a better way to handle it.

Instead of generating ModelicaFormatError for missing paths, we can use ModelicaFormatMessage function to generate warnings instead. In order to let the path search continue, we can use a default value 0. I am not quite sure about arrays though, I haven't looked into the C code.

Benefits

Users don't need to run multiple simulations to figure out all of their missing paths.

In the current implementation (i.e. XMLFile), if a certain node is not found, the simulation aborts right away due to ModelicaFormatError. Therefore, it will take a while for an end user to figure out where all his/her missing paths locates.

In Modelica, missing parameter values are default to zero in most tools

I don't know if this is specified in the spec, but the way most tools treat missing parameter values is by

• Setting its value to zero
• Generating a warning message for the missing parameter.
  ExternData could follow the same pattern.

More flexibility in XML file versioning (To be discussed)

In our case, we find that when we tried to test new models with older version XML files, new models simply won't run because of missing paths. This is, of course, an expected behavior, but it also means we need to update these old version XML files to adapt to the new changes even if most of the paths are still valid.

Please let me know if these points are valid. We can probably add a flag into the findValue function to skip errors or something like that.

GCC compatiblitiy

I see these libs are not compatible with GCC per this thread:

#32

but I'm wondering if there is anything preventing me from recompiling these libs with GCC after editing the make files? Was the decision to not include GCC compatibility intentional due to some restriction or was it just a convenience?

Library of dependencies not found

Hi

While loading this External data to SimulationX the Examples\package.mo won't load. What can I do?
And most importantly This is affecting the library usage

How to get value in array of structures in json?

If we have the next json:
{ "a" : 1.0, "b" : 2.0 }
then we can get the value "a" using getReal("a").

How to get the value "a" for example of the first element "list" for the following example:
{ "list" : [ { "a" : 1, "b" : 2 }, { "a" : 2, "b" : 1 } ] }
In the examples there is a case when an array of values is considered, but such a case is not considered, thanks.

Unit testing package

As more and more features are added to the library, I am thinking we should start thinking to add a unit testing library based on XogenyTest. Current example models are great, but they are not covering all functions and features.

getBoolean inconsistency between XMLFile and JSONFile

Is there a reason why the getBoolean function for XMLFile (and XML2File) only accepts Real values whereas JSONFile seems to take into consideration "true" and "false" statements?

CSVTest causing a translation and scripting error in OpenModelica

Hello.

I am currently using OpenModelica v1.18.0 and complex 4.0.0.
I successfully include the ExternData into my OpenModelica system libraries.
However, the CSVTest does not run and shows error messages as shown below:

[4] 16:48:44 Translation Error
[ExternData: 53:3-77:14]: Ignoring record component:
ExternalObject ExternData.Types.ExternCSVFile csv;
when building the record constructor. Records are allowed to contain only components of basic types, arrays of basic types or other records.

[5] 16:48:44 Scripting Error
[ExternData.Examples: 35:5-35:89]: The default value of csv causes a cyclic dependency.

[6] 16:48:44 Translation Error
Error occurred while flattening model ExternData.Examples.CSVTest

It seems like there's another person who's having a similar issue (https://stackoverflow.com/questions/67967920/externdata-library-with-open-modelica).
It would be very grateful if you could have a look at this issue.
Thank you :)

Best,

MATTest example files doesn't work on last Openmodelica

Dear Sir/Madam,
I shall report that running MATTest file from this library I can't get the simulation to work. The console writes:
/usr/bin/ld: /home/fake_user/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/linux64/libED_MATFile.a(ED_MATFile.o): in function Mat_H5ReadGroupInfoIterate':
ED_MATFile.c:(.text+0x303db): undefined reference to H5Oget_info_by_name' /usr/bin/ld: /home/fake_user/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/linux64/libED_MATFile.a(ED_MATFile.o): in function Mat_H5ReadGroupInfo':
ED_MATFile.c:(.text+0x30846): undefined reference to H5Oget_info_by_name' /usr/bin/ld: ED_MATFile.c:(.text+0x309f7): undefined reference to H5Oget_info_by_name'
/usr/bin/ld: ED_MATFile.c:(.text+0x30b28): undefined reference to H5Literate_by_name' /usr/bin/ld: ED_MATFile.c:(.text+0x30b99): undefined reference to H5Literate_by_name'
/usr/bin/ld: ED_MATFile.c:(.text+0x30e5e): undefined reference to H5Rdereference' /usr/bin/ld: /home/fake_user/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/linux64/libED_MATFile.a(ED_MATFile.o): in function Mat_H5ReadDatasetInfo':
ED_MATFile.c:(.text+0x314ee): undefined reference to H5Rdereference' /usr/bin/ld: /home/fake_user/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/linux64/libED_MATFile.a(ED_MATFile.o): in function Mat_VarReadNextInfoIterate':
ED_MATFile.c:(.text+0x3167e): undefined reference to H5Oget_info_by_name' /usr/bin/ld: /home/fake_user/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/linux64/libED_MATFile.a(ED_MATFile.o): in function Mat_VarRead73.part.43':
ED_MATFile.c:(.text+0x4c599): undefined reference to H5Rdereference' /usr/bin/ld: /home/fake_user/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/linux64/libED_MATFile.a(ED_MATFile.o): in function Mat_VarRead73':
ED_MATFile.c:(.text+0x4d179): undefined reference to H5Rdereference' /usr/bin/ld: /home/fake_user/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/linux64/libED_MATFile.a(ED_MATFile.o): in function Mat_VarReadNextInfo5':
ED_MATFile.c:(.text+0x8ddcb): undefined reference to H5Literate' /usr/bin/ld: /home/fake_user/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/linux64/libED_MATFile.a(ED_MATFile.o): in function readMatIO':
ED_MATFile.c:(.text+0x8e60d): undefined reference to H5Literate' /usr/bin/ld: ED_MATFile.c:(.text+0x8e763): undefined reference to H5Literate'
/usr/bin/ld: /home/fake_user/.openmodelica/libraries/ExternData 3.0.3/Resources/Library/linux64/libED_MATFile.a(ED_MATFile.o): in function ED_getDoubleArray2DFromMAT': ED_MATFile.c:(.text+0x9430a): undefined reference to H5Rdereference'
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [MATTest.makefile:37: omc_main_target] Error 1
Compilation process failed. Exited with code 2.`

The models in the other libraries work fine, so please let me know if someone can confirm this.
Thank you in advance

HDF5

The releases page mentions HDF5, but it is not listed in the ReadMe, main features section.
https://github.com/tbeu/ExternData/releases
https://en.wikipedia.org/wiki/Hierarchical_Data_Format
Is this work in progress? Or part of .mat support?

Convert class type from "model" to "record"

I'd like to recommend record instead of model as class type for external file readers.

• External files are mostly used as record replacements to store parameters.
• A warning pops up in Dymola when passing model (e.g. XMLFile) class as an argument to a function.

Evaluation at translation time and compliance with Modelica Specification

We are trying to use the JSON interface of ExternData and have a few questions.

The first one pertains to the compliance with the Modelica Language Specification, see https://specification.modelica.org/maint/3.5/class-predefined-types-and-declarations.html#specialized-classes where it is stated

The components directly declared in a record may only be of specialized class record or type.

So we wonder whether the construction of Types.ExternJSONFile json within the record ExternData.JSONFile is valid or not (although Dymola and Optimica do not complain). We noticed that a related issue is left open at modelica-3rdparty/Modelica_DeviceDrivers#93.

The second one is how to trigger the evaluation of some parameters which are assigned through calls to ExternData functions and used during flattening or symbolic processing, such as array size, and min, max or nominal attributes.
We noticed that the class annotation annotation(__Dymola_translate=true) was added in recent versions of the library. However, the following model still fails to translate with Dymola (version 2022), unless a literal value is assigned to the parameter n.

model Unnamed
  parameter ExternData.JSONFile dat(
    fileName=Modelica.Utilities.Files.loadResource(
      "modelica://ExternData/Resources/Examples/test.json"));
  parameter Integer n = dat.getArraySize1D(varName="vector") annotation(Evaluate=true);
  parameter Real array[n] = dat.getRealArray1D(varName="vector", n=n);
end Unnamed;

We did some tests (see TestModel below) with a previous version of ExternData (2.5.0 which is compatible with MSL 3.2.3) and it seems that the only case where annotation(__Dymola_translate=true) effectively triggers the evaluation at translation is when the external object is constructed within the function, as in loadAndGetReal. Otherwise we get warnings with Dymola that non-literal values are used in nominal attribute for p and r.
So is there a recommended way to force the evaluation at translation time, without resorting to a construct such as loadAndGetReal which does not seem optimal in terms of resource utilization?

package ExternDataEvaluation
  model TestModel
    extends Modelica.Icons.Example;
    parameter String fileName = Modelica.Utilities.Files.loadResource(
      "modelica://ExternData/Resources/Examples/test.json");
    parameter ExternData.JSONFile dat(fileName=fileName);
    parameter Real p_nominal = dat.getReal(varName="set1.gain.k")
      annotation(Evaluate=true);
    parameter Real q_nominal = loadAndGetReal(varName="set1.gain.k", fileName=fileName)
      annotation(Evaluate=true);
    parameter Real r_nominal = getReal(varName="set1.gain.k", json=dat.json)
      annotation(Evaluate=true);
    Real p(nominal=p_nominal) = 1;
    Real q(nominal=q_nominal) = 1;
    Real r(nominal=r_nominal) = 1;
  end TestModel;
  /* 
  Function loadAndGetReal not supported by OCT despite
  https://specification.modelica.org/maint/3.5/functions.html#external-objects
  "External objects may be a protected component (or part of one) in a function."
  And potentially memory-greedy as it loads the external file at each function call.
  */
  function loadAndGetReal
   input String varName "Key";
   input String fileName;
   output Real var "Variable value";
  protected 
   ExternData.Types.ExternJSONFile extObj = ExternData.Types.ExternJSONFile(fileName)
     "External file object";
  algorithm 
   var := ExternData.Functions.JSON.getReal(varName, extObj);
  annotation(__Dymola_translate=true);
  end loadAndGetReal;
  /* 
  Function getReal supported by OCT but does not trigger evaluation at compile 
  time with Dymola.
  */
  function getReal
    input String varName "Key";
    input ExternData.Types.ExternJSONFile json;
    output Real var "Variable value";
  algorithm 
    var := ExternData.Functions.JSON.getReal(varName, json);
  annotation(__Dymola_translate=true);
  end getReal;
end ExternDataEvaluation;

support for arm64

I tried to compile the binaries for darwin64/arm64. So far this required:

--- a/ExternData/Resources/C-Sources/libxls/src/detail_endian.h
+++ b/ExternData/Resources/C-Sources/libxls/src/detail_endian.h
@@ -65,6 +65,8 @@
 
 # define BOOST_LITTLE_ENDIAN
 # define BOOST_BYTE_ORDER 1234
+#elif __APPLE__
+#include <machine/endian.h>
 #else
 # error The file detail_endian.h needs to be set up for your CPU type.
 #endif

to compile successfully. Still working on getting a simulation to run successfully but that's not related to ExternData so chances are that the change above suffices.

Make the external object instance a parameter?

Is there a specific reason the external object instance is not a parameter? For example:
protected Types.ExternXLSFile xls=Types.ExternXLSFile(fileName, encoding);
could be:
protected parameter Types.ExternXLSFile xls=Types.ExternXLSFile(fileName, encoding);
I'm using Dymola and it doesn't allow the calling of the functions such as getReal() to set parameters if the external object isn't a parameter.

Dymola 2024x Refresh 1 Beta 1 - Experiments using ExternData will not compile

While running tests on Dymola 2024x Refresh 1 Beta 1 we found that our models referencing ExternData will no longer compile.

This is the compiler message:

__iob_func is located in msvc_compatibility.h and appears to be there to support some older versions of Visual Studio compilers.

Support for compilers 2012 (and earlier) will not be supported by Dymola 2024x Refresh1 (per the Dymola developers)

It appears that this chunk of code is gone in the newest version of dsutil.h:

Compilation fails in Dymola 2019 FD01 with MinGW GCC

Hello,
Posting under 2.4 since I'm still running MSL 3.2.2. When trying to run the examples I get errors similar to following:

Compiler message:

Compiling and linking the model (GCC).

Warning: corrupt .drectve at end of def file
Warning: corrupt .drectve at end of def file
C:/Program Files/Dymola 2019/Modelica/Library/ExternData-2.3.3/ExternData/Resources/Library/win32\ED_CSVFile.lib(Release Lib/ED_CSVFile/ED_CSVFile.obj):(.text[_ED_createCSV]+0x1d3): undefined reference to _create_locale' C:/Program Files/Dymola 2019/Modelica/Library/ExternData-2.3.3/ExternData/Resources/Library/win32\ED_CSVFile.lib(Release Lib/ED_CSVFile/ED_CSVFile.obj):(.text[_ED_destroyCSV]+0x2f): undefined reference to _free_locale'
collect2.exe: error: ld returned 1 exit status

SimulationX binaries does not fit anymore

In the meantime SimulationX has changed and also the VisualStudio used went to VS 2019 so the dll given does not run properly anymore.

Using the project file in Resources/BuildProjects/VisualStudio2010
I tried to use the 2010 tooset in VS2019 and I got the error:
error LNK2019: Verweis auf nicht aufgelöstes externes Symbol "_xmlValidNormalizeString" in Funktion "_xmlRemoveID".

I tried to compile the sources with VS 2019 compiler and I got the error
ED_XML2File.obj : error LNK2001: Nicht aufgelöstes externes Symbol "_iob".

additionally.

How to compile the Library with Visual Studio?

Writing results to different file types?

As far as I see, ExternData is an excellent tool to read parameters from different data sources.
Are there plans - or do you know a tool - to write results (at the end of a simulation) to different file types?
This would offer opportunities for some more applications ...

Simulations (incl. `Examples.XLSXTest`) with `XLSX` do not compile on OpenModelica (v1.20, MSL4.0.0)

When trying to simulate a basic model using XLSXFile under OM I get the following error:

XLSXTest_08bnd.c: In function 'XLSXTest_eqFunction_15':
XLSXTest_08bnd.c:60:122: error: too many arguments to function 'omc_ExternData_XLSXFile'
   60 |   (data->simulationInfo->realParameter[4]/* sumB PARAM */)  = omc_ExternData_Examples_XLSXTest_computeColSum(threadData, omc_ExternData_XLSXFile(threadData, (data->simulationInfo->stringParameter[0]/* dataSource.fileName PARAM */) , (data->simulationInfo->booleanParameter[0]/* dataSource.verboseRead PARAM */) , (data->simulationInfo->integerParameter[0]/* dataSource.detectMissingData PARAM */) , (data->simulationInfo->extObjs[0])), _OMC_LIT12, _OMC_LIT0, ((modelica_integer) 2), omc_ExternData_Functions_XLSX_getArrayRows2D(threadData, _OMC_LIT0, (data->simulationInfo->extObjs[0])));
      |                                                                                                                          ^~~~~~~~~~~~~~~~~~~~~~~
In file included from XLSXTest_model.h:27,
                 from XLSXTest_08bnd.c:2:
XLSXTest_functions.h:46:21: note: declared here
   46 | ExternData_XLSXFile omc_ExternData_XLSXFile (threadData_t *threadData, modelica_string omc_fileName, modelica_boolean omc_verboseRead, modelica_integer omc_detectMissingData);
      |                     ^~~~~~~~~~~~~~~~~~~~~~~
At top level:
cc1: note: unrecognized command-line option '-Wno-parentheses-equality' may have been intended to silence earlier diagnostics
make: *** [<builtin>: XLSXTest_08bnd.o] Error 1
make: *** Waiting for unfinished jobs....
Compilation process failed. Exited with code 2.

This occurs with the included example model XLSXTest and similarly with XLSTest.
It appears that the OMC generated function for omc_ExternData_XLSXFile has an additional (5th) argument data->simulationInfo->extObjs[0] that is not defined in the prototype.

Might this be a library compatibility issue, and is there a known fix or workaround for this?

Read *.ssv file in ExternData

The SSP standard gives a very neat way of doing system parameterization through xml files (*.ssv). I am curious if ExternData will be enhanced in the future that it is possible to load *.ssv file directly. (There should even be a standard for system parameterization in Modelica language in my 2 cents)

I understand that the concept of ExternData and *.ssv files are quite different, however, I don't want to create a separate project if ExternData is willing to support it in the future.

PS: Good to see ExternData is now under Modelica-3rdparty.

Error reading array of reals from XML file

A little background on what I'm attempting to accomplish:

I am using XML and JSON files output from NREL's PVWatts API and other web services to get solar PV data for a given geographic location, and use it in Modelica models. I thought this would be a quicker approach than trying to re-create PVWatts in Modelica. Once I get this code working, I will be using it for wind data as well.

The Modelica code I'm using is:

model PVWatts "XML read test - PVWatts file"
  extends Modelica.Icons.Example;

  // Read the PVWatts XML data file
  ExternData.XMLFile xmlfile(fileName=Modelica.Utilities.Files.loadResource("modelica://ExternData/Resources/Examples/PVWattsv5.xml"));

  // Get the measurement station latitude and longitude from the file.  This is just a test to make sure the file can be read.
  // Real stationLat = xmlfile.getReal("station-info.lat");
  // Real stationLon = xmlfile.getReal("station-info.lon");

  // Create an array to hold 12 monthly AC power values
  Real pvMonthly[12] = xmlfile.getRealArray1D("outputs.ac-monthly",12);

end PVWatts;

The two lines that read the station lat and lon work just fine. I run into trouble when attempting to read an array of 12 values from the XML file into the pvMonthly array. I've looked at the function code for getRealArray1D and I think I'm calling it correctly. Once this is working, I will be loading an array of 8760 values of hourly PV data.

The log file dump from Dymola when I run the above code is:

Log-file of program ./dymosim
(generated: Thu Jul 30 08:24:18 2015)

dymosim started
... "dsin.txt" loading (dymosim input file)
The following error was detected at time: 0
Error in line 33 when reading double value from file "C:/Users/jnangle/Desktop/Dymola/Libraries/ExternDatamaster/ExternData/Resources/Examples/PVWattsv5.xml"

The stack of functions is:
ExternData.Functions.XML.Internal.getRealArray1D
xmlfile.getRealArray1D_Unique'"xmlfile"'
xmlfile.getRealArray1D_Unique'"xmlfile"'(
"outputs.ac-monthly", 
12, 
xmlfile.xml)
Error: Failed to start model.

The XML data I'm using is:

<response>
   <inputs>...</inputs>
   <errors type="array"/>
   <warnings type="array"/>
   <version>1.2.0</version>
   <ssc-info>...</ssc-info>
   <station-info>
      <lat type="float">40.016666412353516</lat>
      <lon type="float">-105.25</lon>
      <elev type="float">1634.0</elev>
      <tz type="float">-7.0</tz>
      <location>94018</location>
      <city>BOULDER</city>
      <state>CO</state>
      <solar-resource-file>94018.tm2</solar-resource-file>
      <distance type="integer">38036</distance>
   </station-info>
   <outputs>
      <ac-monthly type="array">
      <ac-monthly type="float">9707.1943359375</ac-monthly>
      <ac-monthly type="float">10191.212890625</ac-monthly>
      <ac-monthly type="float">14889.302734375</ac-monthly>
      <ac-monthly type="float">15481.29296875</ac-monthly>
      <ac-monthly type="float">16531.017578125</ac-monthly>
      <ac-monthly type="float">16387.6953125</ac-monthly>
      <ac-monthly type="float">16299.7412109375</ac-monthly>
      <ac-monthly type="float">15908.3876953125</ac-monthly>
      <ac-monthly type="float">14523.0478515625</ac-monthly>
      <ac-monthly type="float">12817.548828125</ac-monthly>
      <ac-monthly type="float">9812.5400390625</ac-monthly>
      <ac-monthly type="float">9209.373046875</ac-monthly>
   </ac-monthly>
   </outputs>
</response>

Apologies if this is a simple fix. I'm new to Modelica, but I have experience in Java and VBA.

Thank you in advance for your help.

issue with reading data from Excel XLSX

Hi,
I am trying to read a column from an Excel file and then use it in Dymola. Could you please tell me what is the procedure? I am using the ExternData.Types.ExternXLSXFile.constructor function with as input the filename of the excel file, but I get the error:

Error: The class ExternData.Types.ExternXLSXFile is not a package since it contains the
non-constant component id. Lookup in non-package is restricted to 
encapsulated elements, but search found the function constructor.
You can turn off this check by setting Advanced.CheckPackageRestriction:=false;
Component context: ExternData.Types.ExternXLSXFile.id
Component declared as Integer id
File: C:/Program Files (x86)/Dymola 2016 FD01/Modelica/ExternalObject.mo, line 2
Must translate function ExternData.Types.ExternXLSXFile.constructor.
Error: The class ExternData.Types.ExternXLSXFile is not a package since it contains the
non-constant component id. Lookup in non-package is restricted to 
encapsulated elements, but search found the function constructor.
You can turn off this check by setting Advanced.CheckPackageRestriction:=false;
Component context: ExternData.Types.ExternXLSXFile.id
Component declared as Integer id
File: C:/Program Files (x86)/Dymola 2016 FD01/Modelica/ExternalObject.mo, line 2
Failed 
Failed to expand ExternData.Types.ExternXLSXFile.constructor("...path...").

After that, I guess I have to use the function ExternData.Functions.XLSX.getReal. Am I correct?

Thanks

CSV files

Do have any plans to implement CSV functionality in the ExternData library the future?

Read 1D or 2D arrays

If I take the JSON example:

 model JSONTest "JSON file read test"
    extends Modelica.Icons.Example;
    parameter String setName="set1" "Parameter set name" annotation(choices(choice="set1" "First parameter set", choice="set2" "Second parameter set"));
    inner parameter ExternData.JSONFile jsonfile(fileName=Modelica.Utilities.Files.loadResource("modelica://ExternData/Resources/Examples/test.json")) "JSON file" annotation(Placement(transformation(extent={{-80,60},{-60,80}})));
    Modelica.Blocks.Math.Gain gain1(k=jsonfile.getReal(setName + ".gain.k")) annotation(Placement(transformation(extent={{-15,60},{5,80}})));
    Modelica.Blocks.Math.Gain gain2(k=Modelica.Utilities.Strings.scanReal(jsonfile.getString(setName + ".gain.k"))) annotation(Placement(transformation(extent={{-15,30},{5,50}})));
    Modelica.Blocks.Sources.ContinuousClock clock annotation(Placement(transformation(extent={{-50,60},{-30,80}})));
    final parameter Integer m = jsonfile.getArrayRows2D("table1") "Number of rows in 2D array";
    Modelica.Blocks.Sources.TimeTable timeTable(table=jsonfile.getRealArray2D("table1", 3, 2));
...
  end JSONTest;

Is it possible to use table=jsonfile.getRealArray2D("table1", m, 2) instead of (table=jsonfile.getRealArray2D("table1", 3, 2)?

It looks Dymola fails to expand the variable. Is there any workaround, please?
Thank you.

cannot find cell value

I tried to change the cellAddress in the example: XLSXTests, from B2 to C2. I added a value in cell C2. However I get the message:

Cannot get cell (1,2) in sheet "set1" from file "C:/.../test.xlsx"

Failure using getRealArray1D from an element with many sub elements

Hi tbeu,

I have an issue whereby, if an element contains more than a certain number of sub elements, any call to retrieve an array will fail.

Attached are two xmls, which are simple extensions of the example released with ExternData.
The script contains one call to getRealArray1D and works when using the "test.xml" but fails when pointing to the "testFAIL.xml", which simply contains one additional element.

Calling other values from both xmls work fine, so it seems to be the method itself rather than the parsing.

Could you check to see if you see the same problem? Is what I've done fair?
Please let me know if anything is unclear.

Many thanks!

James

I'm using:
Dymola
Visual Studio 2015 Express Edition
ExternData v2.3.3

TestXMLs.zip