CMOR failure when attempting to write large volume of data about cmor

[taylor13]

It might have something to do with use of 64 bit integers (integer*4 in FORTRAN) , which allows a maximum value of 2147483647.
Note that for T=30, the array size is 2005401600 (smaller than max integer), but T=45 is larger than the maximum value represented by a 64 bit integer.
I agree that we should trap the problem if we can and display a clear error message.

[taylor13]

If the problem could be easily corrected by judicious use of integer*8 for the allocation step, that might be an option, but only if not a lot of work since the huge array can be broken down a written in smaller segments (if Matt's supposition is correct).

[matthew-mizielinski]

Hi @taylor13, yes upping the type of appropriate variables to integer*8 is one option. Another would be deal with this in the python layer; if product of the array shape exceeds 2^31 raise an exception (ideally a dedicated one). Users can then react to this (e.g. by halving chunk size being written).

[mauzey1]

I have identified that this issue was being caused by the use of the type int in the function cmor_write_var_to_file when calculating the number of elements in the data. I have resolved this by replacing int with size_t for variables used to calculate the size of the data and indexes.

I have a branch that should be ready to merge but I am now encountering issues with the udunits2 library in the Linux builds. That's a separate issue that seems to be tied with the latest release of udunits2 on conda.