predictionmachines / sdslite Goto Github PK

Hi and thanks for this project.

I have tested the library on Windows and it works well.
However on MacOSX (10.14.6) it is not working, as the library path is not set at all :

I have tried to map to /usr/local/lib/libnetcdff.7.0.0.dylib and any other *.dylib but no success.

Can you help me with this ?

First of all thanks for this library. We have been using it for some time to read NetCDF files but have just started writing them.

I'm trying to set the _FillValue attribute as per CF conventions (https://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/ch02s05.html)

However, when setting this via the MetadataDictionary variable.Metadata["_FillValue"] = 99999.0 it just results in an empty string ("") for _FillValue when examining the output using ncdump -h and causes errors (fails to open) when trying to open the file in a viewer like Panoply - I believe due to it not matching the datatype of the variable.

I think the issue stems from the fact that _FillValue is used internally by the NetCDF library and it wants it set prior to any data being put into the variable.

Despite trying a few different permutations I could not get it to be set correctly using SDSLite. It would either not appear at all or be set to an empty string ("") rather than the double value I was expecting.

Hi!
Thanks for your work. Just wanted to flag that the link on this page does not seem to be working:
https://www.microsoft.com/en-us/research/project/scientific-dataset/#!downloads

Hi,
First, I wish to use this moment to say a big thank you to the team that has put together this sort-after scientific library useful for interacting with netCDF in C#.

I wrote the code below to create a new netcdf data using the classic netcdf library, that is, netcdf without hdf5. However, I got this error "NetCDF4 error" on the first line of using (DataSet ds = DataSet.Open($"msds:nc?file={outputnc}&openMode=createNew")). After tracing the error, I realised it was a netcdf error number -128 which means Attempt to use a feature that was not turned on when netCDF was built.

So I have two questions:

1. Why is the SDSlite not working with my netCDF whereas I have other programs, utilities and packages that use the same netCDF on my machine without any issue?
2. Let us assume I install a dedicated netCDF library for use with SDSLite only, how could I configure the Dataset for a specific netCDF.dll path?

`public static void TestNetCDF()
{
// Output filename
var outputnc = "my_3d_netcdf.nc";

// Define dimensions
int latSize = 10; // Number of latitude points
int lonSize = 20; // Number of longitude points
int timeSize = 5; // Number of time steps



// Create a new NetCDF file
using (DataSet ds = DataSet.Open($"msds:nc?file={outputnc}&openMode=createNew"))
{
    // Generate dimension values
    float[] lats = Enumerable.Range(0, latSize).Select(i => 0.0f + (i * 0.25f)).ToArray();
    float[] lons = Enumerable.Range(0, lonSize).Select(i => 0.0f + (i * 0.25f)).ToArray();
    DateTime[] times = Enumerable.Range(0, timeSize).Select(d => DateTime.Now.AddDays(d)).ToArray();


    // Define dimensions
    var lat_dim = ds.AddAxis<float[]>("lat", "degrees north", lats);
    var lon_dim = ds.AddAxis<float[]>("lon", "degrees east", lons);
    var time_dim = ds.AddAxis<DateTime[]>("time", "days since 01-01-2024", times, true);
    time_dim.Metadata["name"] = "Time";
    time_dim.Metadata["long_name"] = "Time";

    // Create a variable (e.g., temperature)
    var temperature = ds.Add<double[,,]>("temperature", ["time", "lat", "lon"]);
    temperature.MissingValue = -1.0e-32d;


    // Generate some sample data (you can replace this with actual data)
    double[,,] data2d;
    for (int t = 0; t < timeSize; t++)
    {
        data2d = new double[1,latSize, lonSize];
        for (int ilat = 0; ilat < latSize; ilat++)
        {
            for (int ilon = 0; ilon < lonSize; ilon++)
            {
                // Assign temperature values (e.g., random values for demonstration)
                data2d[0,ilat, ilon] = (20.0 + 2.0 * (ilat + ilon) + t);
            }
        }

        // Assign temperature values (e.g., random values for demonstration)
        temperature.PutData([t, 0, 0], data2d);
    }

    // Add metadata (optional)
    ds.Metadata["title"] = "My 3D NetCDF Dataset";
    ds.Metadata["institution"] = "My Institution";
    ds.Metadata["comment"] = "Sample 3D dataset with temperature data";

    // Save the dataset
    ds.Commit();
}

Console.WriteLine("NetCDF file created successfully!");

}`

I have been attempting to include SDSLite in a self-published dotnet core app, and I have been having linker issues with trying to include these .Net Framework libraries. This dotnet issue is the most accurate one I've found that describes my issue:

dotnet/linker#1634 (comment)

I found a fork that attempted to migrate SDSLite to .Net Core, but it does not seem to have as much activity as SDSLite and there is no guarantee it will be maintained:

https://github.com/xfischer/SDSCore

Is there a plan to migrate this library to .Net Standard, so that it can be used in .Net 5 and beyond?

there need a .net core support

Using the newest version netCDF4.9.0-NC4-64.exe, the unit test Empty_AddVariable_string() failes with follwing exception:

System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
----> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
----> Microsoft.Research.Science.Data.NetCDF4.NetCDFException : NetCDF error -132

I try to use SDS in a .net framework console project, using the package nuget SDSLite v2.0.0.$
Program contains only this :
class Program { static void Main(string[] args) { Microsoft.Research.Science.Data.NetCDF4.NetCDFDataSet.Open(@"path_to_my_nc_file"); } }

When executing this program I get this Exception:
Microsoft.Research.Science.Data.DataSetCreateException: 'Failed to create DataSet instance from uri c:\MeteoSurf\SampleFiles\32.nc: The type initializer for 'NetCDFInterop.NetCDFDynamic' threw an exception.'

Inner exception is:
FileNotFoundException: netcdf.dll not found in current directory nor on system path nor in ProgramFilesX86

What do I miss?

Hi, my following code is reflecting error.
Try
{
// open the file and print some info
string filePath = "D:\AccessNetCDF\sresa1b_ncar_ccsm3-example.nc";
using (DataSet ds = DataSet.Open("msds:nc" + filePath + "&openMode=readOnly"))
{
Console.WriteLine(ds);
Console.WriteLine(ds.Metadata["comment"]);
var lat = ds.GetData<float[]>("lat");
Console.WriteLine($"latitude: len={lat.Length}, min={lat.Min()}, max={lat.Max()}");
}

            Console.ReadLine();
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error reading NETCDF file: {ex.Message}");
            Console.ReadLine();
        }
    }

ERROR: Error reading NETCDF file: Failed to create dataset instance from URI msds;ncD:%5CAccessNetCDF%5Csresalb_ncar_ccm3-example.nc&openMode=readOnly: Provider ncD:%5Csresalb_ncar_ccm3-example.nc&openMode=readOnly is not registered.

Any suggestion where could be possibly wrong?

Thanks,
JesC

Hello, I know I am not in the right forum to post this question, but I don't know were else to post it. I am trying to access the initial starting values of the 3 dimensions in a NetCDF file. When I use the DataSet Viewer, I see that it's -94 long, 40 lat and 964240 for time. But through C# code, I can't find them in the dataset. Can you guide me?

First of all great library! Thanks for supporting it.

Recently I had to deploy my application on a Linux platform. I followed the instructions to install the libnetcdf-dev package with:
sudo apt-get install libnetcdf-dev

and would get an error saying 64 bit process cannot load library.

I found out the latest version of libnetcdf-dev, does not install the library at the expected location:
/usr/lib/libnetcdf.so.7

It is installed here (for amd64 architecture):
/usr/lib/x86_64-linux-gnu/libnetcdf.so.11.4.0

I made a symbolic link between the two locations:
ln -s /usr/lib/x86_64-linux-gnu/libnetcdf.so.11.4.0 /usr/lib/libnetcdf.so.7

and voila, the application was able to successfully run and read netCDF files on Linux.

the http://research.microsoft.com/en-us/projects/sds/ link is broken.

I just cloned this project and tried compiling it in Visual Studio Community 2017 15.8.6. I'm getting two errors:

The command ""C:\Users\jordan\Documents\sdslitevs2017.nuget\NuGet.exe" pack "C:\Users\jordan\Documents\sdslitevs2017\SDSLite\SDSLite.csproj" -Properties "Configuration=Debug;Platform=AnyCPU" -NonInteractive -OutputDirectory "C:\Users\jordan\Documents\sdslitevs2017\SDSLite\bin\Debug" -symbols" exited with code 1.

'NUnit' already has a dependency defined for 'NETStandard.Library'.

Any suggestions?

Unable to register format in AOT mode and unable to open file

I am trying to use this lib to read a netcdf-file, but having problem to set it up.
Have tried to use the 32-bit versions of netcdf 4.7.0 and the version from the link on the readme 4.3.3.1.

Howto fix this problem?

Managed Debugging Assistant 'PInvokeStackImbalance' : 'A call to PInvoke function 'NetCDFInterop!NetCDFInterop.NetCDFDynamic+nc_set_chunk_cache::Invoke' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.'

Hello
I am trying to run a global ecology model (Madingley model) under a Unix environment, and so using SDSLite. I am getting a bad chunksize exception when the model gets to creating an output netcdf file to store the simulation output when the model runs over a grid of the earth's surface (exception detail attached). This error does not occur when running the model over a single location, so I assume it is to do with the specifics of a gridded output file. Do you have any ideas what is causing this and how I might resolve it?

exception.txt

dotnet add package System.Security.Permissions

I have read out .nc data with c#, vs2019, netCDF 4.7.2 and SDS2.0.3 with succes, but when using the same code to open a nc4 dataset it fails with the exception : "Extension .nc4 is not registered in the factory".

try{ dataset = DataSet.Open(path_to_nc4_file); } catch(Exception ex) { Console.Write(ex.Message); }

Any idea how to manage that?

Tried reading a netcdf-4 file with netcdf 4.9.0 and SDS 2.0.3 in VS2022. Kept getting the error message Extension not registered.
i tried running

var uri = NetCDF4.NetCDFUri(){ FileName = "filename.nc4", OpenMode = ResourceOpenMode.ReadOnly};
var sds = DataSet.Open(uri);

tried the solution offered in #32

protected bool Register_Extension(string extension)
{
     Type dType = NetCDFDataSet.GetType("Microsoft.Research.Science.Data.NetCDF4.NetCDFDataSet");
     return DataSetFactory.RegisterExtension(extension, dType);
}

var uri = NetCDF4.NetCDFUri(){FileName="filename.nc4", OpenMode=ResourceOpenMode.ReadOnly};
var sds = DataSet.Open(uri);

var sds = DataSet.Open("msds:netcdf?fileName=filename.nc4&openMode=readOnly");

After trying this i got the error message "Failed to create DataSet instance from uri msds:nc?file=filename&openMode=readOnly NetCDFInterop.NetCDFDynamic called for exception"

I'm using VS2109 and SDSlite.

I added SDSLite to my project using the Nuget package manager, and I downloaded the installer for NetCDF binaries from UCAR at http://www.unidata.ucar.edu/software/netcdf/docs/winbin.html.

When I ran the NetCDF installer, I chose not to add anything to my system path. I wanted to understand which of the files I actually needed, so I copied them into my project bin folder and deleted anything that was not apparently in use when the app was executing.

It looks to me like these are needed:

• netcdf.dll
• msvcr100.dll
• zlib1.dll
• hdf5_hl.dll
• hdf5.dll

My code seems to runs fine with (only) these libraries from the NetCDF installer. Is this a valid way to get things set up, or will I run into problems later?

CM.

http://www.unidata.ucar.edu/downloads/netcdf/ftp/netCDF4.3.3.1-NC4-64.exe i download other version not working. How i find netCDF4.3.3.1-NC4-64.exe ?

The variable is mapped to CLR type Sustem.SByte, but an attempt to read data raises NetCDF error NC_ECHAR "Attempt to convert between text & numbers"

Hey there,

I'm using VisualStudio 2019 on a win10 machine. I'm pretty new to VS so maybe my question comes from a lack of understanding.

I tried the code from the tutorial with a csv file and it worked quite good. (except the dataset.Show() method didn't exist?).

Then I wanted to load a .nc file (netcdf) into visual studio (c# console) , so I downloaded the netCDF4.7.2-NC4-64.exe file from
https://www.unidata.ucar.edu/software/netcdf/docs/winbin.html

I installed everything and added the "C:\Program Files\netCDF 4.7.2\bin" to the system path variables.
Then I tried the code :

`using System;
using System.Linq;
using Microsoft.Research.Science.Data;
using Microsoft.Research.Science.Data.Imperative;

namespace SDS_test
{
class Program
{

    static void Main(string[] args)
    {

        // read input data
        var dataset = DataSet.Open("C:\\Users\\X\\Downloads\\sample.nc");`

But I got the following error :
ArgumentException: This 32-bit process failed to load the library C:\Users\X\source\repos\SDS_test\SDS_test\bin\Debug\netcdf.dll. Native error message is 'Das System kann die angegebene Datei nicht finden
(system can't find file)

  | Message | "Failed to create DataSet instance from uri C:\Users\X\Downloads\sample.nc: Der Typeninitialisierer für "NetCDFInterop.NetCDFDynamic" hat eine Ausnahme verursacht."

I tried to solve this by adding all the files to the project (couldn't do it as a reference because they are native dlls). :

I added the dll's, headers and .lib files via "Add existing element..." (as a link). I set the "Build Action" of each file to "Content" and "Copy to Output Directory" to "Copy if Newer".

So the file netcdf.dll is copied to the folder when the build starts and exists at "\Debug\netcdf.dll.".
Still I get the same errors as seen above .

I don't know how to solve this. Would very much appreciate any help.
Thanks in advance!

Repro:

• Write a data set with Variable<string> that contains null element to a .csv file
• Read the .csv file.

Observed: the element is string.Empty.

Expected: the element is null.

See also Empty_AddVariable_string test in SDSLiteTests/GenericFileTests.cs.

Hello, I am trying to cast the data of a NetCDF file, where the metadata of the variable indicates that the typeof is Int16 but the data is actually a float32. When I try to extract the data using this command I get wrong values

Int16[,,] dataVal = ds.GetData<Int16[,,]>(variable.Name);

When I replace the Int16 by float, double, decimal or even Int32, the command throws up an exception

Requested variable does not exist in the data set

The DataSet Viewer see's the same thing. In the attach file, is what I get and what I should be getting.

I've tried to install and use the latest SDSlite NuGet package (1.4.3) a few times, but it doesn't seem to make the Microsoft.Research namespace available. Typing "using Microsoft." doesn't have an option for "Research" in the intellisense popup, and trying to convert code that used the older full-system installed version of Microsoft Scientific DataSet shows this error on compile:

Error CS0234 The type or namespace name 'Research' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)

I have the latest NetCDF4 C libs installed, and use them frequently in other projects. I'm using Microsoft Visual Studio Community 2019 Preview, Version 16.9.0 Preview 1.0. Windows 10 x64.

Hello,

The project is licensed under MIT license according to the file https://github.com/predictionmachines/SDSlite/blob/master/Licence.txt.

However, files in the ScientificDataSet directory, for example,
https://github.com/predictionmachines/SDSlite/blob/master/ScientificDataSet/Providers/NetCDF/NetCdfVariable.cs, contain the headings which state "This code released under the terms of the Microsoft Research License Agreement (MSR-LA)". This is quite a different and rather restrictive license.

Would you please clarify the licensing conditions? It's better to fix headings if it is possible, i.e. if copyright holder permits it.

With best regards.

Repro:

#r "nuget:sdslite"

using Microsoft.Research.Science.Data;
using (var sdsout = DataSet.Open("out.csv", ResourceOpenMode.Create)){
    sdsout.AddVariable<double>("a","a");
}

Error message:

Error: System.Collections.Generic.KeyNotFoundException: The given key 'MissingValue' was not present in the dictionary.
at Microsoft.Research.Science.Data.MetadataDictionary.get_Item(String key, SchemaVersion version)
at Microsoft.Research.Science.Data.CSV.CsvDataSet.SaveMetadataForVariable(StreamWriter sw, Dictionary`2 idToColumn, Variable v)
at Microsoft.Research.Science.Data.CSV.CsvDataSet.SaveCsv(Stream outputStream, ICollection`1 variables)
at Microsoft.Research.Science.Data.CSV.CsvDataSet.SaveToFile(ICollection`1 vars, String fileName)
at Microsoft.Research.Science.Data.CSV.CsvDataSet.OnPrecommit(Changes changes)
at Microsoft.Research.Science.Data.DataSet.Precommit(Changes proposedChanges)
at Microsoft.Research.Science.Data.DataSet.ApplyChanges(Changes proposedChanges)
at Microsoft.Research.Science.Data.DataSet.Commit()
at Microsoft.Research.Science.Data.DataSet.TryCommit()
at Microsoft.Research.Science.Data.DataSet.OnDataSetChanged(DataSetChangeAction action, Changes changes, Object target)
at Microsoft.Research.Science.Data.DataSet.FireEventChanged(DataSetChangeAction action, Changes changes, Object target)
at Microsoft.Research.Science.Data.DataSet.AddVariableToCollection(Variable var)
at Microsoft.Research.Science.Data.DataSet.AddVariable[DataType](String varName, String[] dims)

This only happens when the attribute appendMetadata is true (default). Until this is fixed CSV files should be created without appending metadata, e.g. DataSet.Open("out.csv?appendMetadata=false", ResourceOpenMode.Create)

I'm using sdslite library for C# window application.

  using (Microsoft.Research.Science.Data.DataSet ds = Microsoft.Research.Science.Data.DataSet.Open("msds:nc?file=C:/Users/Min Thein Win/Downloads/myfile.nc&openMode=readOnly"))
        {
            Console.WriteLine(ds);
           // Console.WriteLine(ds.Metadata["comment"]);
          //  var lat = ds.GetData<float[]>("lat");
          //  Console.WriteLine($"latitude: len={lat.Length}, min={lat.Min()}, max={lat.Max()}");
        }

Error Log

Unhandled Exception: Microsoft.Research.Science.Data.DataSetCreateException: Failed to create DataSet instance from uri msds:nc?file=C:/Users/Min Thein Win/Downloads/myfile.nc&openMode=readOnly: Unable to load DLL 'netcdf': The specified module could not be found. (Exception from HRESULT: 0x8007007E) ---> System.DllNotFoundException: Unable to load DLL 'netcdf': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
at NetCDFInterop.NetCDFNative.nc_set_chunk_cache(IntPtr size, IntPtr nelems, Single preemption)
at NetCDFInterop.NetCDF.nc_open_chunked(String path, CreateMode mode, Int32& ncidp, IntPtr size, IntPtr nelems, Single preemption)
at Microsoft.Research.Science.Data.NetCDF4.NetCDFDataSet.InitializeFromFile(String fileName, ResourceOpenMode openMode)
at Microsoft.Research.Science.Data.NetCDF4.NetCDFDataSet..ctor(String uri)
--- End of inner exception stack trace ---
at Microsoft.Research.Science.Data.Factory.DataSetFactoryEntry.CreateDataSet(String uri)
at Microsoft.Research.Science.Data.Factory.DataSetFactory.Create(DataSetUri uri)
at Microsoft.Research.Science.Data.Factory.DataSetFactory.Create(String uri)
at Microsoft.Research.Science.Data.DataSet.Open(String uri)
at ControlConsoleApp.Program.Main(String[] args) in D:\WindowFormTest_workspace\sampleFormApp\ControlConsoleApp\ControlConsoleApp\Program.cs:line 49