pcmdi / cmor

Climate Model Output Rewriter

CMOR

Climate Model Output Rewriter

CMOR3 documentation can be found at: http://cmor.llnl.gov

Contributors

Acknowledgement

The content of this repository is developed by climate and computer scientists from the Program for Climate Model Diagnosis and Intercomparison (PCMDI) at Lawrence Livermore National Laboratory (LLNL), with contributions from colleagues around the world. This work is sponsored by the Regional and Global Model Analysis (RGMA) program, of the Earth and Environmental Systems Sciences Division (EESSD) in the Office of Biological and Environmental Research (BER) within the Department of Energy's Office of Science. The work is performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

   

cmor's Issues

Error handling in CMOR 3.1

Thank you for making the changes required to prevent the core dumps :)

I have discovered some differing behaviours in the way CMOR 3.1 handles errors.

Example 1

Error terminates the Python interpreter:

% ipython
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cmor

In [2]: cmor.dataset_json('')
C Traceback:
! In function: cmor_is_setup
! called from: cmor_dataset_json
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: You need to run cmor_setup before calling any cmor_function
!
!!!!!!!!!!!!!!!!!!!!!!!!!
%

Example 2

Error causes a KeyboardInterrupt exception to be thrown with no message. In addition, there is no log message, unless cmor.close() is called:

% ipython
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cmor

In [2]: cmor.setup(logfile='cmor.log')

In [3]: cmor.dataset_json('')
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
KeyboardInterrupt: 

In [4]: more cmor.log

In [5]: cmor.close()
Out[5]: 0

In [6]: more cmor.log

C Traceback:
! In function: cmor_open_inpathFile
! called from: cmor_dataset_json
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Could not understand file "./" Is this a JSON CMOR table?
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_dataset_json
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: JSON resource file not found
!
!!!!!!!!!!!!!!!!!!!!!!!!!

! ------
! CMOR is now closed.
! ------
! During execution we encountered:
!   1 Warning(s)
!   2 Error(s)
! ------
! Please review them.
! ------
! 

In [7]: 

Would it be possible to:

1. not exit the python interpreter when there is an error.
2. print to the log after each command is executed, rather than only when cmor.close() is executed.
3. use the message that is sent to the log as the exception message.

This ticket provides information relevant to CMOR 3.1, so #27 can probably be closed as a duplicate.

[Edited to remove "raise a more appropriate exception than KeyboardInterrupt"; I just noticed a comment from Denis in an e-mail that says "Now you will see a new message Keyboard Interrupt. There is nothing I can do about this, if I want to trap the signal." and trapping the signal is definitely more important than the exception type :)]

Misleading error message from dataset_json?

Following from the comment in #25, I tried to load an incorrectly formatted JSON file:

% ipython
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cmor

In [2]: cmor.setup(logfile='cmor.log')

In [3]: cmor.dataset_json('cmor_dataset.json')
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
KeyboardInterrupt: 

In [4]: cmor.close()
Out[4]: 0

In [5]: more cmor.log

C Traceback:
! In function: cmor_open_inpathFile
! called from: cmor_dataset_json
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Please validate JSON File!
! USE: http://jsonlint.com/
! Syntax Error in file: cmor_dataset.json
!  {

!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_dataset_json
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: JSON resource file not found
!
!!!!!!!!!!!!!!!!!!!!!!!!!

! ------
! CMOR is now closed.
! ------
! During execution we encountered:
!   0 Warning(s)
!   2 Error(s)
! ------
! Please review them.
! ------
! 

In [6]: 

There are two errors, one of which says JSON resource file not found, even though the file was found. Is this second error message necessary?

No new line at the end of a table causes 'cmor.load_table' to hang indefinitely

If a table is missing a new line at the end of the file, cmor.load_table hangs indefinitely. I can't ctrl-c to terminate and I have to ctrl-z and kill the Python session. Would it be possible for cmor.load_table to fail in these cases, ideally with an error explaining why?

Feedback from the CMIP6 example

I ran through the CMIP6 example and I have some feedback:

1. Would it be possible to include something like the following in a comment in the example?

# If this example is not executed from the directory containing the
# CMOR code, please first complete the following steps:
#
#   1. In any directory, create 'Tables/', 'Test/' and 'CMIP6/' directories.
#   2. Download
#      https://github.com/PCMDI/cmor/blob/master/TestTables/CMIP6_Omon.json
#      and https://github.com/PCMDI/cmor/blob/master/TestTables/CMIP6_CV.json
#      to the 'Tables/' directory.
#   3. Download
#      https://github.com/PCMDI/cmor/blob/master/Test/test_python_CMIP6_experimentID.json
#      to the 'Test/' directory.

2. When loading the MIP table, should the line read table_id = cmor.load_table("CMIP6_Omon.json") (I was shown a return value when I ran the command in the example)?

3. I had no problems until the cmor.write step:

In [8]: for i in range(0,5):
   ...:     cmor.write(ivar,data[i:i])
   ...:     

C Traceback:
In function: cmor_CV_setInstitution
! called from: cmor_setGblAttr
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Your input attribute institution "GICC (Generic International Climate Center, Geneva, Switzerland)" will be replaced with 
! "GCESS,BNU,Beijing,China" as defined in your Control Vocabulary file.
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
In function: cmor_CV_checkSourceID
! called from: cmor_setGblAttr
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Your input attribute "source" with value 
! "MIROC-ESM" will be replaced with value 
! "MIROC-ESM:".
! 
! 
!  See Control Vocabulary JSON file.(Tables/CMIP6_CV.json)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your Control Vocabulary file specifies one or more
! required attributes.  CMOR found that the following
! attribute was not properly set.
! 
! Please set attribute: "branch_method" in your input JSON file.
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your Control Vocabulary file specifies one or more
! required attributes.  CMOR found that the following
! attribute was not properly set.
! 
! Please set attribute: "source_type" in your input JSON file.
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your Control Vocabulary file specifies one or more
! required attributes.  CMOR found that the following
! attribute was not properly set.
! 
! Please set attribute: "sub_experiment" in your input JSON file.
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_CV_checkGblAttributes
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Please fix required attributes mentioned in
! the warnings above and rerun. (aborting!)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-8-4723ba247acc> in <module>()
      1 for i in range(0,5):
----> 2     cmor.write(ivar,data[i:i])
      3 

/home/h01/ehogan/software/cmor/lib/python2.7/site-packages/cmor/pywrapper.pyc in write(var_id, data, ntimes_passed, file_suffix, time_vals, time_bnds, store_with)
    694         raise Exception, "Error data type must one of: 'f','d','i','l', please convert first"
    695 
--> 696     return _cmor.write(var_id,data,type,ntimes_passed,time_vals,time_bnds,store_with)
    697 
    698 def _check_time_bounds_contiguous(time_bnds):

KeyboardInterrupt: 

In [9]: 

The first two traceback messages, i.e.,

C Traceback:
In function: cmor_CV_setInstitution
! called from: cmor_setGblAttr

and

C Traceback:
In function: cmor_CV_checkSourceID
! called from: cmor_setGblAttr

were printed without the off-white background, so I couldn’t read the dark blue text against the black background of my terminal window.

Just a comment, but when I open the log file with emacs, the background colour causes characters like ^[[2;31;47mC Traceback: to be visible.

Test/cmor_speed_and_compression_01.py

wrong time on check at the end

Error in Python Sample Program 1 from cmor_users_guide

I have copy/pasted the Sample Program 1 example script from the CMOR2 documentation (page 56) and I had to use the following cmor.dataset call instead of the one supplied in the example. I don't know if I got all the meta-data right, but at least I got rid of the spurious quotes and I have added the missing parent_experiment_rip parameter

cmor.dataset('historical', 'ukmo', 'HadCM3 (2010)', '360_day',
             model_id='HadCM3',forcing='Nat',
             parent_experiment_id='N/A', parent_experiment_rip='N/A',
             branch_time=0.,
             contact='Tim Lincecum, [email protected]', institute_id='pcmdi')

Are the examples used in the documentation available somewhere in the github repo of CMOR? I have seen that there are several scripts in cmor/Test, but the documentation does not supply names for the examples and I have not opened all the scripts available in Test to check

Inconsistency in Makefile creates a "missing separator" error that breaks CMOR/UV-CDAT build on some platforms

I'm installing UV-CDAT 2.1.0 on different platforms and eventhough the Makefile generated for both platforms are essentially the same (except for slight different path names), make will succeed on th RH EL server and fail on the FC20 machine with the following error

Makefile:156: *** missing separator (did you mean TAB instead of 8 spaces?).  Stop.

Trouble is that this breaks the installation of UV-CDAT, that stops after the CMOR installation error is detected... :-(

The make versions are not the same on the machines
On our RH server (Red Hat Enterprise Linux Server release 6.6 (Santiago)): â_(ÿ^?

> which \make
    /usr/bin/make
> rpm -qf /usr/bin/make
    make-3.81-20.el6.x86_64

On a FC20 (Fedora release 20 (Heisenbug))

> which \make
    /bin/make
> rpm -qf /bin/make
    make-3.82-19.fc20.x86_64

If I check lines 154 to 157, I can see that indeed the lines up to 157 start with a TAB, and lines 156 and 157 start with 8 spaces. Apparently the newer version of make does not like this inconsistency!

#   @env TEST_NAME=Test/test_python_joerg_9.py make test_a_python
    @env TEST_NAME=Test/test_python_joerg_10.py make test_a_python
        @env TEST_NAME=Test/test_python_joerg_11.py make test_a_python
        @env TEST_NAME=Test/test_python_joerg_12.py make test_a_python

Lines 158 and 159 should be corrected in Makefile[.in] and start with a TAB. They seem to be the only lines exhibiting this problem

>\grep -n ' @env' Makefile 
156:        @env TEST_NAME=Test/test_python_joerg_11.py make test_a_python
157:        @env TEST_NAME=Test/test_python_joerg_12.py make test_a_python

After I patch these 2 lines, CMOR build succeeds and UV-CDAT reaches [100%] \o/

CMOR configure script uses -R flag (Mac OS X 10.8.5, Xcode 5.1)

Thanks @dakoop for reporting this one via UVCDAT.
The -R flag is hardcoded in the CMOR configure script (and configure.ac) under the RTAG variable; it should use the global flag or at least update its logic to use -Wl,-rpath, for compatibility with Clang 5.x.

Generation of non CF-compliant files (invalid formula_terms attribute on bounds variable)

As below:

On 12/8/15, 2:14 AM, "martin**@stfc.ac.uk" <martin**@stfc.ac.uk> wrote:

Hello Karl,

I've just scanned a sample of CMIP5 files, 5 random files from every CMOR variable,
with the CF Checker. This turns up a few errors from the data providers, but many
which appear to come from CMOR. Out of 4435 files checked, 1327 are
non-compliant. The commonest error, in 936 files, is:
"ERROR (4.3.2): formula_terms attribute only allowed on coordinate variables"
which is triggered by having a formula_terms attribute on a bounds variable
-- which the checker does not accept.

The remaining errors are mainly invalid standard_names or units, and syntax
errors in the cell_methods string. I hope I'll be able to catch such errors by
scanning the CMIP6 data request before it goes out, though we will need to
agree how to handle standard_names which are under review.

There are also a significant number of files which crash the CF Checke
 through use of scalar coordinates ... I've posted a comment to the CF list
on this point.

Is the use of formula_terms on bounds variables something which is built
into CMOR? Could it be switched off or blocked, or is there some
justification for keeping it and proposing an amendment to the convention?

regards,
Martin

Climatology fields

From: Karl Taylor
Date: Thursday, April 30, 2015 at 8:06 AM
To: "Doutriaux, Charles"
Cc: "Paul J. Durack"
Subject: CMOR climatology fields

Hi Charles,

What does CMOR do special when the time-coordinate is labeled "climatology" (in the CMOR table -- "climatology: yes")?

How does it assign the coordinate value (is it based on the mid-point of the bounds, which would not be the right thing to do), or do we just store the coordinates the user sent us? Are any checks done?

Do we have a sample test case where we've checked CMOR treats climatologies correctly?

thanks,
Karl

downgrade non-monotonic bounds check from error to warnings for climatological axes

sending climo time axis to non climo var leads to seg fault

When using uvcdat clt and computing annualcycle
trying to rewrite it using "clt" from AMon table
leads to seg fault.

add travis autobuild

Is the 'inpath' parameter of cmor.cmor_setup used?

I'm afraid there might be an error in the documentation

I have specified the location of my CMOR tables using cmor.cmor_setup(inpath='absolute_path_to_the_tables', ...), according to what the documentation says on page 12:

[inpath] = a character string specifying the path to the directory where the needed
MIP-specific tables reside.

But I got an error when executing cmor.load_table('CMIP5_Amon')

C Traceback:
In function: cmor_load_table
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                         !
! Error: Could not find table: CMIP5_Amon !
!                                         !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I have then checked more closely the documentation of cmor.load_table (page 20) and it does not mention using the path specified by cmor.cmor_setup:

CMOR will look for the table first followig the path
as specified by the “table” argument passed to this function. If it doesn’t find a file
there it will prepend the outpath defined in calling cmor_dataset. If it still doesn’t
find it, it will use the “prefix” where the library CMOR is to be installed (from configure time)
followed by share (e.g /usr/local/cmor/share). If it stills fails an
error will be raised.

obs4MIPs restructuring

There is a need to generalise CMOR so that it is easier for a user to "once off" write their observational data following the CF and CMIP5 conventions (this will require a consideration of existing usage by the CMIP community - tradeoffs will need to be carefully considered).

global_attributes will likely be MIP-dependent.

New obs4MIPs tables will likely need to be created.

Some of this information is presented on the ESGF-CoG

Specifically there are a number of working documents which outline global_attributes and identifiers:
obs4MIPs Dataset Requirements
obs4MIPs Global Attributes Requirements

Further information about getting a users data into netcdf can be added on a wiki page - potentially at the PCMDI_metrics page

Pinging @taylor13 @gleckler1 @doutriaux1 @durack1

create_output_path doesn't work from Python

create_output_path is available in pywrapper.py, but cmor._cmor has no attribute create_output_path:

% ipython
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cmor

In [2]: cmor.create_output_path('')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-fa2c46fcc4fc> in <module>()
----> 1 cmor.create_output_path('')

/home/h01/ehogan/software/cmor/lib/python2.7/site-packages/cmor/pywrapper.pyc in create_output_path(varid)
    912     Returns: the path where the file will be created
    913     """
--> 914     return _cmor.create_output_path(varid)

AttributeError: 'module' object has no attribute 'create_output_path'

In [3]: 

time axis error

I have a weird error, which I can’t tell whether it is a bug on my site or on CMOR’s side. The problem occurs in version 2.9.1 but does not in version 2.8.3, and it only occurs when writing 3d fields. The error I get is this:

! Error: axis time (table: 3D-M) has overlapping bounds values : 57701.000000, 57729.000000, 57701.000000 at index: 0 !

Of course, I use the same time values for all variables for the same month, and it does not occur in the 2d variables I save. In addition, in case it matters, it occurs when calling cmor_write during February (365 day calendar, so 28 days) and on the second 3d variable in a loop.

I hope I explained it correctly. I can easily reproduce it, so if you want me to run any test and print any output for you, I’ll be more than happy to. Again, it happens in 2.9.1, but not in 2.8.3.

pep8 cmor code

cmor crashing

I'm having an issue with cmor crashing intermittently. I am processing a number of files in a row, and cmor will produce a few files (even up to 100), then throw this error in the cmor_axis routine:

*** glibc detected *** /apps/python/2.7.3/bin/python: double free or corruption (!prev): 0x0000000004aa17d0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x760e6)[0x7f9fc7e940e6]
/lib64/libc.so.6(+0x78c13)[0x7f9fc7e96c13]
/home/599/pfu599/pythonlibs/lib/python2.7/site-packages/cmor/_cmor.so(cmor_axis+0x5a2)[0x7f9f90fcb222]
/home/599/pfu599/pythonlibs/lib/python2.7/site-packages/cmor/_cmor.so(+0xa805)[0x7f9f90faa805]
/apps/python/2.7.3/bin/python(PyEval_EvalFrameEx+0x5600)[0x4a7580]
/apps/python/2.7.3/bin/python(PyEval_EvalCodeEx+0x877)[0x4a9067]
/apps/python/2.7.3/bin/python(PyEval_EvalFrameEx+0x5294)[0x4a7214]
/apps/python/2.7.3/bin/python(PyEval_EvalFrameEx+0x6575)[0x4a84f5]

I've also seen this similar error:
*** glibc detected *** /apps/python/2.7.3/bin/python: free(): invalid pointer: 0x00000000246ee160 ***
and this:
*** glibc detected *** /apps/python/2.7.3/bin/python: corrupted double-linked list: 0x0000000003286ec0 ***

If I run the script to produce the same file again, generally it will succeed but then crash again when trying to produce a different file.

I am using the most recent version of cmor (2.9.1), with python2.7.3, netCDF4.3.0. I've tried building cmor with both intel and gnu compilers.

Errors and error messages in CMOR via Python

I have a few issues / questions about errors and error messages in CMOR via Python; if they should be split into separate issues, please let me know.

I am currently trying to catch messages from CMOR so I can log them via the Python logger. During my experiments I discovered there are cases where errors terminate the Python interpreter. For example:

% ipython
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cmor
No module named cdms2

In [2]: cmor.load_table('non_existent_table_name')
C Traceback:
In function: cmor_is_setup
called from: cmor_load_table


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                                                    !
! Error: You need to run cmor_setup before calling any cmor_function !
!                                                                    !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

% 

I wondered if this was the case for all errors from CMOR, but I think this first example is an exception (no pun intended) ;) :

% ipython
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cmor
No module named cdms2

In [2]: cmor.setup()

In [3]: cmor.load_table('non_existent_table_name')

C Traceback:
In function: cmor_load_table


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                                      !
! Error: Could not find table: non_existent_table_name !
!                                                      !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
<ipython-input-3-f6298efc3e7d> in <module>()
----> 1 cmor.load_table('non_existent_table_name')

/path/lib/python/cmor/pywrapper.pyc in load_table(table)
    748 ##     if not os.path.exists(table):
    749 ##         raise Exception, "Error, the table you specified (%s) does not exists" % table
--> 750     return _cmor.load_table(table)
    751 
    752 def dataset(experiment_id,institution,source,calendar,outpath='.',realization=1,contact="",history="",comment="",references="",leap_year=0,leap_month=0,month_lengths=None,model_id="",forcing="",initialization_method=None,physics_version=None,institute_id="",parent_experiment_id="",branch_time=None,parent_experiment_rip=""):

SystemError: error return without exception set

In [4]: 

Google tells me that the SystemError: error return without exception set exception can happen if code in a C extension module returns NULL (to indicate an error has occurred) but no exception was set. This means I can wrap the CMOR call in a try-except block, but the exception message is not the same as the error message printed by CMOR:

In [4]: try:
   ...:     cmor.load_table('non_existent_table_name')
   ...: except Exception, err:
   ...:     raise err
   ...:     

C Traceback:
In function: cmor_load_table


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                                      !
! Error: Could not find table: non_existent_table_name !
!                                                      !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
<ipython-input-5-a4c6935cf1f8> in <module>()
      2     cmor.load_table('non_existent_table_name')
      3 except Exception, err:
----> 4     raise err
      5 

SystemError: error return without exception set

In [5]: 

Additionally, it appears that the error message from the CMOR call isn't sent to stdout or stderr (my attempts to capture the messages have failed), so even if I use a try-except block I still get an error message printed to screen.

So, to summarise:

1. Would it be possible to fix the following issues:

• the cases where CMOR errors terminate the Python interpreter
• the cases where CMOR errors do not return a valid exception

2. Is there a way to return the error messages from CMOR in such a way that enables them to be captured by external programs?

Improving cmor fatal error handling

Still working with the Sample Program 1 example script, I have altered the variable definition line in order to generate a fatal error that I can use to expose several problems. So, I have used a non existing temperature unit

varid = cmor.variable('ts', 'Krap', axis_ids)

and I got the following error

CMOR - 91 >python -i sample_script_01.py

C Traceback:
In function: cmor_write_var_to_file


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                                                                  !
! Error: in udunits analyzing units from user (Krap) for variable ts (table: Amon) !
!                                                                                  !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

CMOR - 92 >

The things that should be improved:

1. I was using python -i to run my script, but I got dumped in the shell rather than back in the python interpreter after the fatal error. It would be much more useful if we could have an exception and be able to go back to the python shell to debug. Maybe there is an exit in the C code in case of errors that prevents going back to python?
2. The sample script that usually generates a ts_Amon_HadCM3_historical_r1i1p1_200001-200001.nc file (when there is no error) left a ts_Amon_HadCM3_historical_r1i1p1_28973.nc temporary file in the same directory instead. The first time I found one of these temporary files in the same directory as the correct file, I thought why is there a second file with a weird content in this directory?. It would be nice (and useful) to alter the name of the temp file to make it more explicit that it is a temp file! e.g. ts_Amon_HadCM3_historical_r1i1p1_28973_cmor_tmp.nc temporary file

limited area

Hello Charles,

I'm getting very confused as to what should and shouldn't work in cmor. I thought I had a work around to the grid problem below, but I'm still not sure CMOR is working quite right (it seems to want to reverse the longitudes when it shouldn't). When I was trying to produce a test case demonstrating my current problem I managed to generate yet a different set of errors. I've attached the test case which is throwing up the new (and I think incorrect - but please review the test) errors.

This is causing us some frustration as its delaying us distributing some data from a regional model. If you could find some time to have a look at the problem it would be appreciated.

Thanks,

Jamie

Sample code:

import cmor
import numpy

def cmor_initialisation():
    cmor.setup(inpath='/net/home/h03/hadju/Projects/MipTables/cmip5-cmor-tables/Tables',
               netcdf_file_action = cmor.CMOR_REPLACE_3,
               create_subdirectories = 0)
    cmor.dataset('pre-industrial control', 'ukmo', 'HadCM3', '360_day',
                 institute_id = 'ukmo',
                 model_id = 'HadCM3',
                 history = 'some global history',
                 forcing = 'N/A',
                 parent_experiment_id = 'N/A',
                 parent_experiment_rip = 'N/A',
                 branch_time = 0.,
                 contact = 'bob',
                 outpath = '/data/local/hadju/test')

def setup_data():
    axes = [ {'table_entry': 'time',
              'units': 'days since 2000-01-01 00:00:00'
              },
             {'table_entry': 'latitude',
              'units': 'degrees',
              'coord_vals': [0],
              'cell_bounds': [-0.5, 0.5]},
             {'table_entry': 'longitude',
              'units': 'degrees',
              'coord_vals': [359., 1.],
              'cell_bounds': [[359. - 1., 359. + 1.],
                              [1. - 1., 1. + 1.]]},
             ]

    values = numpy.array([359., 1.], numpy.float32)
    return values, axes

def cmor_define_and_write(values, axes):
    table = 'CMIP5_Amon'
    cmor.load_table(table)

    axis_ids = list()
    for axis in axes:
       axis_ids.append(cmor.axis(**axis))

    table = 'CMIP5_Amon'
    cmor.load_table(table)

    varid = cmor.variable('rlut',
                          'W m-2',
                          axis_ids,
                          history = 'variable history',
                          missing_value = -99,
                          positive = 'up'
                          )

    cmor.write(varid, values, time_vals = [15], time_bnds = [0, 30])

def version(cmor):
   return '%s.%s.%s' % (cmor.CMOR_VERSION_MAJOR,
                        cmor.CMOR_VERSION_MINOR,
                        cmor.CMOR_VERSION_PATCH)

def main():
    assert version(cmor) == '2.8.3'
    cmor_initialisation()
    values, axes = setup_data()
    cmor_define_and_write(values, axes)
    cmor.close()

if __name__ == '__main__':

    main()

Table restructuring

Currently tables include duplicate information (experiment specific and dimension/axis specific and variable specific).

These duplicated components should be split into separate tables

Add a capability to include a table within existing table call - @doutriaux1 has many ideas on this..

cmor.close(): slight error in the returned path and question about the duplicate close()

When I call cmor.close() in the Sample Program 1 example script

cmor.write(varid, [273])
path=cmor.close(varid, file_name=True)
print path
cmor.close()

I get a path that has a duplicate / before the file name

./CMIP5/output/pcmdi/HadCM3/historical/mon/atmos/ts/r1i1p1//ts_Amon_HadCM3_historical_r1i1p1_200001-200001.nc

I have also a question about the second cmor.close() (no parameters) that prints the message pasted below. Is it necessary to call cmor.close a last time at the end of the script?

------
CMOR is now closed.
------

We encountered no warnings or errors during execution
------
Congratulations!
------

Oh, I have just tried to generate a warning by calling cmor.write(varid, [400]) and the second cmor.close() printed some usage statistics. The documentation (p 38) says To be safe, before exiting any program that invokes CMOR, it is best to call this function with the argument omitted, but it would be more useful to say that the last cmor.close() will print some statistics and use something more clear than to be safe

------
CMOR is now closed.
------
During execution we encountered:
  2 Warning(s)
  0 Error(s)
------
Please review them.
------

Multiple .close(varid, preserve=True) causes a growing 'coordinates' list

It appears that closing files for a variable, but then opening a new file to continue the timeseries can result is a growing coordinates attribute:

$ ncdump -h  /data/local/hadju/test/rlut_Amon_HadCM3_piControl_r1i1p1_200001\-200001.nc  | grep coordinates
  rlut:coordinates = "lat lon" ;
$ ncdump -h  /data/local/hadju/test/rlut_Amon_HadCM3_piControl_r1i1p1_200002\-200002.nc  | grep coordinates
  rlut:coordinates = "lat lon lat lon" ;

A possible fix is to identify in the code where coordinates is appended to and then to replace the append with the C equivalent of something like:

 def add_coord_att(new_coord, coordinates_att):
    '''
    append new_coord to the coordinates_att string

    Examples:
    ---------
    >>> add_coord_att('lat', '')
    'lat'
    >>> add_coord_att('lon', 'lat')
    'lat lon'
    >>> add_coord_att('lat', 'lat lon')
    'lat lon'
    '''
    coords = coordinates_att.split()
    if new_coord not in coords:
         coords.append(new_coord)
    return ' '.join(coords)

It also looks as though the history attribute is getting appended too many times - for instance 'Inverted lat' can appear multiple times. (This is also evident from a review of some CMIP5 data - so this bug has been around for a while).

Strange and inconsistent characters for commit info in CMOR-configure-out.log

I'm installing UV-CDAT 2.1.0 on different platforms and, when comparing the output files created for cmor by cmake, I have noticed some strange characters (non ascii? control chars?) in the CMOR-configure-out.log files. And the characters are not the same on both platforms...

On our RH server (Red Hat Enterprise Linux Server release 6.6 (Santiago)): â_(ÿ^?

config.status: creating compile_line.txt
config.status: creating include/cmor_locale.h
configure: 
configure: ************************************************************************
configure:                               SUMMARY
configure:                             VERSION: 2.9.0 (commit: (â (ÿ�)
configure: ************************************************************************

On a FC20 (Fedora release 20 (Heisenbug)): ^A

configure:                               SUMMARY
configure:                             VERSION: 2.9.0 (commit: �)

Incorrect version tag for CMOR3-beta1

The following version tag is being written for CMOR3-beta1:

:cmor_version = "3.0.0"

This should align with the current version, so should be 3.0.0-beta1

Can not invoke CMOR in C++

Dear all,

I tried to invoke cmor in c++, and encountered the following link error:

$ clang++ -o test test.cpp -I/usr/local/include -I/usr/local/include/cdTime -I/usr/local/opt/ossp-uuid/include -I/usr/local/opt/udunits/include -L/usr/local/opt/ossp-uuid/lib -luuid -L/usr/local/opt/udunits/lib -ludunits2 -L/usr/local/lib -lcmor -lnetcdf
Undefined symbols for architecture x86_64:
 "cmor_setup(char*, int*, int*, int*, char*, int*)", referenced from:
     _main in test-547354.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

After some struggles, I figured out that I can solve this problem by changing all the extern in cmor_func_def.h to extern "C" so that those functions can be recognized by c++. But this makes cmor fail in c. This is just a inconsistent solution, so it would be nice that cmor will support the use in c++ natively.

Anaconda installation problems

When uninstalling CMOR2.9.2 after installing UV-CDAT from conda channel (to update to CMOR3) there is a dependency on UV-CDAT2.4.1

@dnadeau4 @doutriaux1 pinging you both here - it only seems to be an issue on ocean?

[duro@ocean ~]$ conda uninstall cmor
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....
Solving package specifications: .........

Package plan for package removal in environment /export/durack1/anaconda2:

The following packages will be REMOVED:

    cmor:   2.9.2-PCMDI
    uvcdat: 2.4.1-UVCDAT

And also:

which python
/export/duro/anaconda2/bin/python
[duro@ocean ~]$ python
Python 2.7.11 |Anaconda custom (64-bit)| (default, Dec  6 2015, 18:08:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cdms2
>>> import cmor
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/export/duro/anaconda2/lib/python2.7/site-packages/cmor/__init__.py", line 1, in <module>
    from cmor_const import *
  File "/export/duro/anaconda2/lib/python2.7/site-packages/cmor/cmor_const.py", line 1, in <module>
    import _cmor
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /export/duro/anaconda2/lib/python2.7/site-packages/cmor/_cmor.so

basin twice in file

CMIP6 wants to be able to have mutliple character type dims

Attaching a test code that fakes this.

The issue at the moment is becaus ew euse NetCDF CLASSIC format for string variables we need to define the max_length of the strings, which is sadly hardcoded to the name strlen in cmor at the moment. This leads to a netcdf erorr trying to define the dimension strlen twice.

C Traceback:
In function: cmor_write


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                                                                                                                                                                      !
! Error: NetCDF error (-42: NetCDF: String match to name in use) for dummy 'strlen' dimension definition of axis: basin (2) in metafile, while writing variable htovgyre (table: Omon) !
!                                                                                                                                                                                      !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

test_region_2_str_coord.f90.txt
CMIP5_Omon.txt

CMOR 3.1 tests are not working

I ran make test after installing CMOR 3.1 and make exited with an error:

% make test
r - cmor.o
r - cmor_axes.o
r - cmor_CV.o
r - cmor_variables.o
r - cmor_tables.o
r - cdTimeConv.o
r - cdUtil.o
r - timeConv.o
r - timeArith.o
r - cmor_grids.o
r - cmor_md5.o
r - arraylist.o
r - json_c_version.o
r - json_object_iterator.o
r - json_util.o
r - linkhash.o
r - random_seed.o
r - debug.o
r - json_object.o
r - json_tokener.o
r - libjson.o
r - printbuf.o
r - cmor_cfortran_interface.o
r - cmor_fortran_interface.o
rm -f ./ipcc_test_code ; gcc -I/home/h01/ehogan/software/netcdf/include -I/usr/local/sci/include -Iinclude/json-c -Iinclude -Iinclude/cdTime -g -O2   Test/ipcc_test_code.c -L/home/h01/ehogan/software/cmor/lib -I/home/h01/ehogan/software/cmor/include  -L. -lcmor -I/home/h01/ehogan/software/netcdf/include -I/usr/local/sci/include -L/home/h01/ehogan/software/netcdf/lib -lnetcdf    -L/home/h01/ehogan/software/udunits/lib  -Wl,-rpath=/home/h01/ehogan/software/udunits/lib -ludunits2 -lexpat  -I/home/h01/ehogan/software/udunits/include  -L/home/h01/ehogan/software/uuid/lib  -Wl,-rpath=/home/h01/ehogan/software/uuid/lib -luuid  -I/home/h01/ehogan/software/uuid/include  -lm   -o ipcc_test_code  >/dev/null 2>/dev/null; ./ipcc_test_code >/dev/null 2>/dev/null
make: *** [test_C] Error 127

I tried running ipcc_test_code directly:

% ./ipcc_test_code
ok mode is:0
Test code: ok init cmor
yep: GICCM1(2002), Nat

C Traceback:
! In function: cmor_outpath_exist
! called from: cmor_dataset_json
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: You defined your output directory to be: 'CMIP6', but this
! directory does not exist
!
!!!!!!!!!!!!!!!!!!!!!!!!!

If I make a directory called CMIP6 and try again I get:

% ./ipcc_test_code
ok mode is:0
Test code: ok init cmor
yep: GICCM1(2002), Nat
Test code: ok load cmor table(s)
Test code: defining axis region 
Test code: Redefining time/lat from O table
Test code: defining variables from table 1, down 
Test code: definig tas
Test code: ok we define hfogo positive: down 
Test code: writing time: 1 of 2
2d

C Traceback:
In function: cmor_CV_checkExperiment
! called from: cmor_setGblAttr
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Your input attribute "sub_experiment_id" with value 
! "ds1968" will be replaced with value "s1968"
! as defined for experiment_id "piControl-withism".
! 
!  See Control Vocabulary JSON file.(Tables/CMIP6_CV.json)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!

3d
writing tas
3d zfactor
writing ps
2d region
Test code: writing time: 2 of 2
2d
3d
writing tas
3d zfactor
writing ps
2d region
! ------
! CMOR is now closed.
! ------
! During execution we encountered:
!   1 Warning(s)
!   0 Error(s)
! ------
! Please review them.
! ------

Would it be possible to:

1. have the test create any required directories (i.e., CMIP6).
2. have all the tests run without make exiting with an error and produce a summary of how many tests passed / failed / errored.
3. fix the above issue; even though the message (and summary message) suggest that the message is a warning, it seems to be causing make to error.
4. add "run make test" to the release checklist in #62?

fix include of cdmsint.h

include/cmor_func_def.h has the line
#include "cdmsint.h"

• why?

Surely cdmsint.h is for internal use within libcdms only, and other packages should not be using it. A "make install" on cdms does not install the internal header file. If content is needed externally, it should be in the external header file (cdms.h).

This is causing issues not while installing CMOR itself but when subsequently compiling things against CMOR, in particular CDO. (Can work around by installing cdmsint.h to /usr/include manually, but that's not the point.)

About the "does not match the expect variable shape ([0, ny, nx])" error

@doutriaux1 , could you run my test script at the end of this mail and determine why I'm getting the following does not match the expect variable shape ([0, 10, 18]) error?

  File "/usr/local/install/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_VB_13/lib/python2.7/site-packages/cmor/pywrapper.py", line 630, in write
    raise Exception,"error your data shape (%s) does not match the expect variable shape (%s)" % (str(osh),str(ogoodshape))
Exception: error your data shape ((1, 10, 18)) does not match the expect variable shape ([0, 10, 18])

I have had this error several times, always when trying to do something with the time axis, so I guess the error is time axis related, but I could be wrong

1. It's strange to get a [0, ny, nx] shape. I wonder if this could be a side effect of #16
2. This problem is very hard to debug for the end user. I have used the following script to easily reproduce the error and it is clear that I am not passing any information about time values/bounds to cmor.write and that cmor.write should complain about it. But the error message should be more specific. And when I first encountered this problem, in another case, I had correctly specificied values for time! So I had to resort to trying to specify the time values in lots of different ways, till I eventually got rid of the error (for no obvious reason)
3. The error message (if you keep it) has a typo and should be "does not match the expected variable shape" :-)

Note: The script will work correctly if you uncomment the following lines. You can use it as an example, if need be, since it shows how udunits will automatically convert degF to K, and cmor2 will automatically update the axes to match what is requested in the CMOR tables

#               time_vals=numpy.array([15. + 30 * month]),
#               time_bnds=numpy.array([0. + 30 * month, 30. + 30 * month])

Example script

#!/usr/bin/env python

import numpy, cdms2, cmor

# Generate a dummy data matrix with '32 degF' everywhere except 2
# latitudinal bands around the equator
raw_data = numpy.ma.ones((1, 10, 18), numpy.float32) * 32
raw_data[:, 4:6, :] = numpy.ma.masked

# Generate axes for the data matrix
lat_axis = cdms2.createUniformLatitudeAxis(90, 10, -20)
lon_axis = cdms2.createUniformLongitudeAxis(-180, 18, 20)

cmor.setup(inpath='/home/users/jypeter/PMIP3_CMIP5/CMOR/cmip5-cmor-tables/Tables/',
           netcdf_file_action=cmor.CMOR_REPLACE)

cmor.dataset(experiment_id='lgm',
             institution='Institut Pierre Simon Laplace',
             institute_id='IPSL',
             model_id='ICE-MAKER-1',
             source='ICE-MAKER-1 atmosphere: JYP-007; ice: AMORINO-1',
             calendar='360_day',
             outpath='.',
             realization=1,
             forcing='N/A',
             parent_experiment_id='N/A', parent_experiment_rip='N/A',
             branch_time=0.,
             contact='JYP@LSCE',
             comment='This is some really dummy data',
             references='100 ways to freeze your ocean, The Ice Cream Journal')

cmor.load_table('CMIP5_Amon')

time_id = cmor.axis(table_entry= 'time',
                    units= 'days since 2000-01-01 00:00:00')

lat_id = cmor.axis(table_entry= 'latitude',
                   units= 'degrees_north',
                   coord_vals=lat_axis[:],
                   cell_bounds=lat_axis.getBounds())
lon_id = cmor.axis(table_entry= 'longitude',
                   units= 'degrees_east',
                   coord_vals=lon_axis[:],
                   cell_bounds=lon_axis.getBounds())

data_id = cmor.variable(table_entry='ts',
                        units='degF',
                        axis_ids=[time_id, lat_id, lon_id],
                        missing_value=raw_data.fill_value,
                        original_name='temp',
                        comment='A dummy variable from a dummy dataset')

# Write 3 time steps to the file (January to March). Add 18 degF to
# each time step (which should be converted to 10 K in the output
# file)
for month in range(3):
    cmor.write(var_id=data_id,
               data=raw_data + month * 18,
# Uncomment the 2 lines below in order to have a working script
#               time_vals=numpy.array([15. + 30 * month]),
#               time_bnds=numpy.array([0. + 30 * month, 30. + 30 * month])
)

file_path = cmor.close(data_id, file_name=True)

print '\nThe dummy data was successfully saved to', file_path

# The end

time axis set from time_bnds not provided time axis values

@dnadeau4 @doutriaux1 @taylor13 I've just come across a weird quirk with CMOR3.0.6 that I wasn't expecting. I am providing input data with the following time axis (for the last year of the full 1870-2015 period):

Note the input file doesn't have a time_bnds

netcdf amipbc_siconc_360x180_v1.1.0_187001-201512 {
dimensions:
    time = UNLIMITED ; // (1752 currently)
    lat = 180 ;
    bound = 2 ;
    lon = 360 ;
variables:
    double time(time) ;
        time:units = "days since 1870-1-1" ;
        time:long_name = "time" ;
        time:standard_name = "time" ;
        time:calendar = "gregorian" ;
        time:axis = "T" ;

    "2015-01-16 12", "2015-02-15", "2015-03-16 12", "2015-04-16", 
    "2015-05-16 12", "2015-06-16", "2015-07-16 12", "2015-08-16 12", 
    "2015-09-16", "2015-10-16 12", "2015-11-16", "2015-12-16 12" ;


[durack1@oceanonly 150219_AMIPForcingData]$ ncdump -v time 360x180_v1.1.0_san/amipbc_siconc_360x180_v1.1.0_187001-201512.nc

 time = 15.5, ...
    52975.5, 53005, 53034.5, 53065, 53095.5, 53126, 53156.5, 
    53187.5, 53218, 53248.5, 53279, 53309.5 ;

When I hand this dataset to CMOR3.0.6 I get the following time axis back in the output file (again this is just the last 12 months of the full 1870-2015 period):

netcdf siconcbcs_input4MIPs_AMIPForcing_PCMDI-AMIP-1-1-0_gs1x1_187001-201512 {
dimensions:
    time = UNLIMITED ; // (1752 currently)
    lat = 180 ;
    lon = 360 ;
    bnds = 2 ;
variables:
    double time(time) ;
        time:bounds = "time_bnds" ;
        time:units = "days since 1870-01-01" ;
        time:calendar = "gregorian" ;
        time:axis = "T" ;
        time:long_name = "time" ;
        time:standard_name = "time" ;
    double time_bnds(time, bnds) ;

    "2015-01-16 03", "2015-02-15", "2015-03-16 18", "2015-04-16", 
    "2015-05-16 12", "2015-06-16", "2015-07-16 15", "2015-08-16 09", 
    "2015-09-16", "2015-10-16 12", "2015-11-16", "2015-12-16 12" ;


[durack1@oceanonly 150219_AMIPForcingData]$ ncdump -v time CMIP6/input4MIPs/PCMDI/AMIPForcing/mon/ocean/PCMDI-AMIP-1-1-0/siconcbcs/gs1x1/v20160607/siconcbcs_input4MIPs_AMIPForcing_PCMDI-AMIP-1-1-0_gs1x1_187001-201512.nc

 time = 15.5, ...
52975.125, 53005, 
    53034.75, 53065, 53095.5, 53126, 53156.625, 53187.375, 53218, 53248.5, 
    53279, 53309.5 ;


[durack1@oceanonly 150219_AMIPForcingData]$ ncdump -v time_bnds CMIP6/input4MIPs/PCMDI/AMIPForcing/mon/ocean/PCMDI-AMIP-1-1-0/siconcbcs/gs1x1/v20160607/siconcbcs_input4MIPs_AMIPForcing_PCMDI-AMIP-1-1-0_gs1x1_187001-201512.nc

 time_bnds =
  0.75, 30.25, ...
  52960, 52990.25,
  52990.25, 53019.75,
  53019.75, 53049.75,
  53049.75, 53080.25,
  53080.25, 53110.75,
  53110.75, 53141.25,
  53141.25, 53172,
  53172, 53202.75,
  53202.75, 53233.25,
  53233.25, 53263.75,
  53263.75, 53294.25,
  53294.25, 53324.75 ;

This behavior is really not what I was expecting - I would assume that if I provide a time_axis those values are preserved, rather than CMOR recreating the time_axis from what would appear to be faulty time_bnds in the input file.

3.1 tagging

• Update master branch with cmor3 branch
• Review all issue lists and close all redundant issues (move unresolved issues to next milestone)
• Validate version info is correctly written (CMOR version should be 3.1.0)
• Validate data_specs_version written (01.beta.29)
• Update website so correct tag points (3.1.0) are noted for build notes
• Prepare and publish release notes to website
• Create PDF from website.
• Update README.md with updated Stable version info (should be 3.1.0)
• Set 3.1.0 tag point for PCMDI/cmor repo
• Obtain Zenodo DOI
• Finalize README.md with updated Zenodo DOI info
• Move 3.1.0 tag point to new commit
• After 3.1.0 tag point set, reset all master checkout tag points to point back to cmor-master tags.
• Increment cmor-master to 3.1.1 (odd minor [major.minor.patch] number indicating development version)
• Email mailing lists: [email protected], [email protected], [email protected], [email protected] including the text: Please pass on this email to anyone you know who plans to prepare CMIP6 model output for the archive.

Add min/max time limits per experiment

Problem with cmor close deallocating forcings in the MIP tables

I think there is a problem in cmor close - I started seeing seg faults at the end of last week, which I think I've tracked down to a piece of code for deallocating the forcings in the MIP tables.

Here's a patch:

--- a/Src/cmor.c
+++ b/Src/cmor.c
@@ -3917,8 +3917,8 @@ int cmor_close(void)
}
if (cmor_tables[i].nforcings>0) {
for (j=0;j<cmor_tables[i].nforcings;j++) {

•   free(cmor_tables[i].forcings[i]);
  

•   cmor_tables[i].forcings[i]=NULL;
  

•   free(cmor_tables[i].forcings[j]);
  

•   cmor_tables[i].forcings[j]=NULL;
  

  }
  free(cmor_tables[i].forcings);
  cmor_tables[i].forcings=NULL;

Could you check you agree with the patch and make the appropriate changes in the repository?

Deal with salinity unit variants better

The issue described at PCMDI/pcmdi_metrics#162 and it's solution should probably be implemented in CMOR. It would appear that CMOR already knows about psu (https://github.com/PCMDI/cmip5-cmor-tables/blob/master/Tables/CMIP5_Omon#L2925-2989) however the other realistic variants of pss* should also be considered.

As part of an update to CF in 2011, all the new salinity variants were included in the standard name table (http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2011/022581.html) these (and their units) should be incorporated into CMOR..

@doutriaux1 I'd be happy to work with you on this.. As part of a V3.0 push..

Error in SImon table

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Reading table SImon: axis name: 'angle' for variable: 'siitdsnconc' is not defined in table. Table defines dimensions: 'longitude latitude angle time' for this variable
!
!!!!!!!!!!!!!!!!!!!!!!!!!

A request to check the 'required_global_attributes' from the table before 'cmor.write'

Would it be possible to check for the presence of the required global attributes from the table (i.e., run cmor_has_required_global_attributes) after cmor.dataset / cmor_set_cur_dataset_attribute (perhaps when running cmor.load_table, assuming cmor.dataset has been previously run?) rather than waiting until cmor.write (as shown in the example below), so that any errors can be caught sooner?

The output when I run cmor_example.py (see below) is:

No module named cdms2
Running cmor.setup
Running cmor.dataset
Running cmor.load_table
Running cmor.axis
Running cmor.variable
Running cmor.write

C Traceback:
In function: cmor_has_required_global_attributes


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!                                                                                                                                              !
! Error: Required global attribute parent_experiment_rip is missing please check call(s) to cmor_dataset and/or cmor_set_cur_dataset_attribute !
!                                                                                                                                              !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Note that the example, which is based on 'Sample Program 1' from the CMOR Users Guide, doesn't work as-is, since parent_experiment_rip is not defined in cmor.dataset (but exists in the required_global_attributes in the table).

Contents of cmor_example.py:

#! /usr/bin/env python
import cmor

def main():
    """
    'Sample Program 1' from the CMOR Users Guide.

    ``CMIP5_Amon`` table from https://github.com/PCMDI/
    cmip5-cmor-tables/blob/master/Tables/CMIP5_Amon.

    Some minor edits were made:

      * ``inpath`` was removed from ``cmor.setup``.
      * ``source``, ``model_id`` and ``forcing`` in ``cmor.dataset``
        were updated.

    This sample program doesn't work as the ``CMIP5_Amon`` table has
    ``parent_experiment_rip`` in the ``required_global_attributes``.
    """
    print 'Running cmor.setup'
    cmor.setup(netcdf_file_action=cmor.CMOR_REPLACE)

    print 'Running cmor.dataset'
    cmor.dataset(
        experiment_id='historical', institution='ukmo', source='HadCM3 (2010)',
        calendar='360_day', contact='Tim Lincecum, [email protected]',
        model_id='HadCM3', forcing='N/A', institute_id='pcmdi',
        parent_experiment_id='N/A', branch_time=0.)

    print 'Running cmor.load_table'
    table = 'CMIP5_Amon'
    cmor.load_table(table)

    print 'Running cmor.axis'
    itime = cmor.axis(table_entry='time',
                      units='days since 2000-01-01 00:00:00', coord_vals=[15,],
                      cell_bounds=[0, 30])
    ilat = cmor.axis(table_entry='latitude', units='degrees_north',
                     coord_vals=[0], cell_bounds=[-1, 1])
    ilon = cmor.axis(table_entry='longitude', units='degrees_east',
                     coord_vals=[90], cell_bounds=[89, 91])
    axis_ids = [itime, ilat, ilon]

    print 'Running cmor.variable'
    varid = cmor.variable('ts', 'K', axis_ids)

    print 'Running cmor.write'
    cmor.write(varid, [273])

    print 'Running cmor.close'
    path = cmor.close(varid, file_name=True)
    print path
    cmor.close()

if __name__ == '__main__':
    main()

"rank is deprecated" warning message (in pywrapper.py?)

I got the following warning, when running the Sample Program 1 example script from the CMOR2 documentation (page 56)

/usr/local/install/cdat/versions/cdat_install_uv-2.1.0_x86_64_gcc4_VB_13/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2499: VisibleDeprecationWarning: `rank` is deprecated; use the `ndim` attribute or function instead. To find the rank of a matrix see `numpy.linalg.matrix_rank`.
  VisibleDeprecationWarning)

pywrapper does use numpy.rank() many times, so maybe it's coming from there

I'm using python 2.7.8 from UV-CDAT 2.1.0

Keeping track of the CMOR version

I was having an issue with CMOR similar to that described in issue #6. To see if the fix from issue #6 resolved my issue, I downloaded v2.9.2 from https://github.com/PCMDI/cmor/releases, since the issue was added to the v2.9.2 milestone (#6 (comment)). However, I discovered the fix wasn't included in the v2.9.2 release and that the CMOR version in this release was still v2.9.1.

Looking at the fix itself (a100f8f) I realised that it had only been tagged uvcdat-2.4.0 (why not v2.9.3?), so I downloaded this version only to discover that the CMOR version was still v2.9.1.

Would it be possible to ensure that the version number of CMOR is correctly updated for every release? This is important for reproducibility purposes (since the CMOR version is written to the headers of the output netCDF files), especially for CMIP6. In addition, would it be possible to ensure that issues are added to the correct milestone? This would be useful when determining which releases contain specific fixes.

I heard that Karl @taylor13 may be interested in hearing about this issue.

[The fix from issue #6 did resolve my issue, which I was very happy about] ;)

Importing CMOR from a Python session prints 'No module named cdms2'

When I import CMOR from a Python session, the message No module named cdms2 is printed:

% python
Python 2.7.6 (default, Jan  3 2014, 16:42:21) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cmor
No module named cdms2
>>>

I can confirm that I don't have cdms2 installed:

>>> import cdms2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named cdms2
>>>

The import cdms2 in pywrapper.py is in a try-except block but theimport cdms2 in check_CMOR_compliant.py is not. __init__.py contains:

try:
  from check_CMOR_compliant import checkCMOR
except Exception,err:
  print err
  pass

My concern is that this message could cause unnecessary worry for some users. Is there a way to suppress this message?

Test/cmor_speed_and_compression.py points to wrong location

sample_data moved in uvcdat see:
CDAT/cdat#1390

hourly climatology axis for CMIP6-CFMIP

CFMIP have added a request for a set of variables with monthly averages of each hour of the day. This corresponds to a cell_methods value of "time: mean within days time: mean over days" as illustrated in Example 7.10 of the CF Convention. I propose to add a new time coordinate to the CMIP6 data request, with label "time3" which will be similar to the "time2" record specifying the monthly climatology time axis.

Enabling netcdf deflation in CMOR output files

I am hoping to use the default netcdf4 deflation in output files written by CMOR. Is this possible using the python interface?

Updates in support of CMIP6

@taylor13 @doutriaux1 I've included an email from Martin Juckes below which outlines some key items for consideration in CMOR updates in preparation for CMIP6:

-----Original Message-----
From: "martin.juckes"
Date: Thursday, April 23, 2015 at 2:07 AM
To: "Paul J. Durack"
Cc: "Taylor, Karl Taylor", "Doutriaux, Charles"
Subject: RE: CMIP6 and CMOR2+

Hello Paul,

Glad to meet you.

Please don't start on the OMDP document yet -- that is input into the process, and we need to agree on how the tables etc will be organised in CMIP6 to deal with the expanded structure. I was planning on providing you with CMOR format MIP tables when we have mature documents.

At the moment the CMOR/CMIP6 issues that I'm aware of are:
(1) Need to support polar stereographic grids. Charles has said in an email that the code is there -- but I'm not sure what needs to be done in the MIP tables;
(2) Extend the "Conventions" attribute: pending agreement from CF committee -- and add a MIP table tag so that some variables can have "UGRID-0.9" added to the conventions attribute;
(3) Add a resolvable unique identifier: Tobias Weigel (DKRZ) is working on a system of resolvable identifiers (using the Handle Registry -- www.handle.net). Has he been in touch with you? I'll be at a meeting of one of the European project which is supporting his work next week, and I hope to find out more about his plans there.
(4) Global attributes: the WIP (WGCM Infrastructure Panel) is discussing a range of changes to global attributes -- hopefully these will not be difficult to implement;
(5) Delivering output in a flat directory structure: it currently provides a pseudo-ESGF directory structure with the result that some groups did not bother to transform to a correct directory structure.
(6) There is demand for a more robust system of testing CMOR and associated CMIP6 tables. We may be able to provide some support on this -- I'll check the status of our discussions there.
(7) Lots of new variables and dimensions -- but hopefully these will not need software changes.

For the CMIP6 data request I have a discussion board (http://dreq01.vanillaforums.com) and I have started a "CMOR" section -- we can move this somewhere else if you have a more convenient location (at the moment this board is mainly used by me to keep track of data request issues, so there is not much content).

regards,
Martin

Clarifying required attributes: mip_era as an example

@dnadeau4 @taylor13 I have found unusual behaviors when not including mip_era in my CMOR3 driver file. If mip_era is not included in the drive_input4MIPs_bcs/obs.json files (as per directory structure below):

(cmor)durack1@oceanonly:[150219_AMIPForcingData]:[468]> ls -l */*.json
-rw-r--r-- 1 durack1 climate 2.8K Jun  3 11:48 CMOR/drive_input4MIPs_bcs.json
-rw-r--r-- 1 durack1 climate 2.6K Jun  3 11:48 CMOR/drive_input4MIPs_obs.json
(cmor)durack1@oceanonly:[150219_AMIPForcingData]:[470]> ls -l */*/Tables/*.json
-rw-r--r-- 1 durack1 climate 5.7K Jun  3 13:12 CMOR/input4MIPs-cmor-tables/Tables/input4MIPs.json
-rw-r--r-- 1 durack1 climate 2.6K Jun  2 19:55 CMOR/input4MIPs-cmor-tables/Tables/input4MIPs_CV.json

The following errors are triggered:

C Traceback:
! In function: cmor_CV_checkGblAttributes
! called from: cmor_setGblAttr
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your Control Vocabulary file specifies one or more
! required attributes.  CMOR found that the following
! attribute was not properly set.
!
! Please set attribute: "mip_era" in your input JSON file.
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_get_cur_dataset_attribute
! called from: cmor_CV_ValidateAttribute
! called from: cmor_CV_checkGblAttributes
! called from: cmor_setGblAttr
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: CMOR Dataset: current dataset does not have attribute : mip_era
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_CV_ValidateAttribute
! called from: cmor_CV_checkGblAttributes
! called from: cmor_setGblAttr
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: The attribute "mip_era" could not be validated.
! The current input value is "�" which is not valid
! Valid values must match the regular expression:
!   ["CMIP6"  ...]
!
! Check your Control Vocabulary file "CMOR/input4MIPs-cmor-tables/Tables/input4MIPs_CV.json".
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_CV_checkGblAttributes
! called from: cmor_setGblAttr
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your Control Vocabulary file specifies one or more
! required attributes.  CMOR found that the following
! attribute was not properly set.
!
! Please set attribute: "mip_era" in your input JSON file.
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_get_cur_dataset_attribute
! called from: cmor_CV_ValidateAttribute
! called from: cmor_CV_checkGblAttributes
! called from: cmor_setGblAttr
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: CMOR Dataset: current dataset does not have attribute : mip_era
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_CV_ValidateAttribute
! called from: cmor_CV_checkGblAttributes
! called from: cmor_setGblAttr
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: The attribute "mip_era" could not be validated.
! The current input value is "�" which is not valid
! Valid values must match the regular expression:
!   ["CMIP6"  ...]
!
! Check your Control Vocabulary file "CMOR/input4MIPs-cmor-tables/Tables/input4MIPs_CV.json".
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: cmor_CV_checkGblAttributes
! called from: cmor_setGblAttr
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Please fix required attributes mentioned in
! the warnings above and rerun. (aborting!)
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!

Traceback (most recent call last):
  File "./sanitize.py", line 301, in <module>
    cmor.write(varid,values,time_vals = d.getTime()[:],time_bnds=d.getTime().genGenericBounds())
  File "/export/durack1/anaconda2/envs/cmor/lib/python2.7/site-packages/cmor/pywrapper.py", line 696, in write
    return _cmor.write(var_id,data,type,ntimes_passed,time_vals,time_bnds,store_with)
KeyboardInterrupt

Would it be possible for cmor.dataset_json to accept a JSON string?

I was wondering whether it would be possible to change cmor.dataset_json such that it can accept a JSON string as input as well as a JSON file name?

netcdf output configuration

I believe we've decided that we should be able to use netcdf4 and compression, so being able to configure CMOR to do this would be more than helpful (although it's not a done deal until after various WGCM and WIP perambulations).