cdat / cdtime

climate calendar manipulation tools

cdat

⚠️ WARNING: Maintenance-only mode until around the end of 2023.
The CDAT library is now in maintenance-only mode, with plans for deprecation and cease of support around the end of calendar year 2023. Until this time, the dependencies for specific CDAT packages (cdms2, cdat_info, cdutil, cdtime, genutil, libcdms) will be monitored to ensure they build and install in Conda environments. We currently support Python versions 3.7, 3.8, 3.9, and 3.10. Unfortunately, feature requests and bug fixes will no longer be addressed.
If you are interested in an alternative solution, please check out the xarray and xCDAT - Xarray Extended With Climate Data Analysis Tools projects.

CDAT builds on the following key technologies:

1. Python and its ecosystem (e.g. NumPy, Matplotlib);
2. Jupyter Notebooks and iPython;
3. A toolset developed at LLNL for the analysis, visualization, and management of large-scale distributed climate data;
4. VTK, the Visualization Toolkit, which is open source software for manipulating and displaying scientific data.

These combined tools, along with others such as the R open-source statistical analysis and plotting software and custom packages (e.g. DV3D), form CDAT and provide a synergistic approach to climate modeling, allowing researchers to advance scientific visualization of large-scale climate data sets. The CDAT framework couples powerful software infrastructures through two primary means:

1. Tightly coupled integration of the CDAT Core with the VTK infrastructure to provide high-performance, parallel-streaming data analysis and visualization of massive climate-data sets (other tighly coupled tools include VCS, DV3D, and ESMF/ESMP);
2. Loosely coupled integration to provide the flexibility of using tools quickly in the infrastructure such as ViSUS or R for data analysis and visualization as well as to apply customized data analysis applications within an integrated environment.

Within both paradigms, CDAT will provide data-provenance capture and mechanisms to support data analysis.

CDAT is licensed under the [BSD-3][bds3] license.

---

We'd love to get contributions from you! Please take a look at the Contribution Documents to see how to get your changes merged in.

cdtime's Issues

import error with freshly created vcs/cdms2

ImportError: dlopen(/Users/doutriaux1/miniconda3/envs/vcdt/lib/python3.6/site-packages/cdtime/_cdtime.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libc++.1.dylib
  Referenced from: /Users/doutriaux1/miniconda3/envs/vcdt/lib/python3.6/site-packages/cdtime/_cdtime.cpython-36m-darwin.so
  Reason: image not found

conda list:

# packages in environment at /Users/doutriaux1/miniconda3/envs/vcdt:
#
# Name                    Version                   Build  Channel
alembic                   0.9.9                      py_0    conda-forge
appnope                   0.1.0                 py36_1000    conda-forge
asn1crypto                0.24.0                py36_1003    conda-forge
async_generator           1.10                       py_0    conda-forge
attrs                     18.2.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
blas                      1.0                         mkl    defaults
bleach                    3.0.2                      py_1    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
cdat_info                 8.0.2018.11.28.19.22.gb6530fa            py_0    cdat/label/nightly
cdms2                     3.1.0.2018.11.29.11.23.g5a83dd4  py36hdde6e19_0    cdat/label/nightly
cdtime                    3.1.0.2018.11.01.10.48.gb8c370e  py36hdc02c5d_0    cdat/label/nightly
cdutil                    8.0.2018.11.12.19.20.g3bd6383            py_0    cdat/label/nightly
certifi                   2018.11.29            py36_1000    conda-forge
cffi                      1.11.5           py36h5e8e0c9_1    conda-forge
chardet                   3.0.4                 py36_1003    conda-forge
clapack                   3.2.1                h470a237_1    conda-forge
configurable-http-proxy   3.1.0                   node8_1    conda-forge
cryptography              2.3.1            py36hdffb7b8_0    conda-forge
cryptography-vectors      2.3.1                 py36_1000    conda-forge
curl                      7.63.0               h74213dd_0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
distarray                 2.12.2                     py_1    conda-forge
dv3d                      8.0.2018.12.15.10.52.g215c720            py_0    cdat/label/nightly
entrypoints               0.2.3                 py36_1002    conda-forge
esmf                      7.1.0r               h35eb876_3    conda-forge
esmpy                     7.1.0r                   py36_1    conda-forge
expat                     2.2.5                hfc679d8_2    conda-forge
ffmpeg                    4.0.2                ha0c5888_2    conda-forge
freetype                  2.9.1                h6debe1e_4    conda-forge
future                    0.17.0                py36_1000    conda-forge
g2clib                    1.6.0                         3    conda-forge
genutil                   8.1.2018.10.17.14.26.ga5756e2.npy1.14  py36h776bbcc_0    cdat/label/nightly
gettext                   0.19.8.1             h1f1d5ed_1    conda-forge
ghostscript               9.22                 hfc679d8_1    conda-forge
gmp                       6.1.2                hfc679d8_0    conda-forge
gnutls                    3.5.19               h2a4e5f8_1    conda-forge
hdf4                      4.2.13               h951d187_2    conda-forge
hdf5                      1.10.3               hc401514_2    conda-forge
idna                      2.8                   py36_1000    conda-forge
intel-openmp              2019.1                      144    defaults
ipykernel                 5.1.0           py36h24bf2e0_1001    conda-forge
ipython                   7.2.0           py36h24bf2e0_1000    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jasper                    1.900.1              hff1ad4c_5    conda-forge
jedi                      0.13.2                py36_1000    conda-forge
jinja2                    2.10                       py_1    conda-forge
jpeg                      9c                   h470a237_1    conda-forge
jsonschema                3.0.0a3               py36_1000    conda-forge
jupyter-vcdat             2.0.0.2018.12.17.23.37.g0e0419d            py_0    cdat/label/nightly
jupyter_client            5.2.4                      py_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyterhub                0.9.4                 py36_1000    conda-forge
jupyterlab                0.35.4                   py36_0    conda-forge
jupyterlab_server         0.2.0                      py_0    conda-forge
krb5                      1.16.2               hbb41f41_0    conda-forge
lapack                    3.6.1                         1    conda-forge
libcdms                   3.0.2018.10.17.14.17.g4f79235      hdc02c5d_0    cdat/label/nightly
libcf                     3.0.2018.06.27.19.10.g994aa30  py36h1de35cc_0    cdat/label/nightly
libcurl                   7.63.0               hbdb9355_0    conda-forge
libdrs                    3.0.2018.10.18.14.41.g36264ac      hdc02c5d_0    cdat/label/nightly
libdrs_f                  3.0.2018.10.18.14.41.g36264ac      hdc02c5d_0    cdat/label/nightly
libedit                   3.1.20170329         haf1bffa_1    conda-forge
libffi                    3.2.1                hfc679d8_5    conda-forge
libgfortran               3.0.1                h93005f0_2    defaults
libiconv                  1.15                 h470a237_3    conda-forge
libnetcdf                 4.6.1               h350cafa_11    conda-forge
libpng                    1.6.36               ha92aebf_0    conda-forge
libsodium                 1.0.16               h470a237_1    conda-forge
libssh2                   1.8.0                h5b517e9_3    conda-forge
libtiff                   4.0.10               he6b73bb_1    conda-forge
mako                      1.0.7                      py_1    conda-forge
markupsafe                1.1.0            py36h470a237_0    conda-forge
mistune                   0.8.4            py36h470a237_0    conda-forge
mkl                       2019.1                      144    defaults
mkl_fft                   1.0.10                   py36_0    conda-forge
mkl_random                1.0.2                    py36_0    conda-forge
mpi                       1.0                       mpich    conda-forge
mpich                     3.2.1                h26a2512_7    conda-forge
nb_conda                  2.2.1                    py36_0    conda-forge
nb_conda_kernels          2.2.0                 py36_1000    conda-forge
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1                  hfc679d8_2    conda-forge
netcdf-fortran            4.4.4               h71ea97b_10    conda-forge
nettle                    3.3                           0    conda-forge
nodejs                    8.10.0                        0    conda-forge
notebook                  5.7.4                 py36_1000    conda-forge
numpy                     1.15.4           py36hacdab7b_0    defaults
numpy-base                1.15.4           py36h6575580_0    defaults
openh264                  1.8.0                hd28b015_0    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
ossuuid                   1.6.2                hfc679d8_0    conda-forge
pamela                    0.3.0                   py_1001    conda-forge
pandoc                    2.5                           0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.3.1                      py_0    conda-forge
pexpect                   4.6.0                 py36_1000    conda-forge
pickleshare               0.7.5                 py36_1000    conda-forge
prometheus_client         0.5.0                      py_0    conda-forge
prompt_toolkit            2.0.7                      py_0    conda-forge
ptyprocess                0.6.0                 py36_1000    conda-forge
pycparser                 2.19                       py_0    conda-forge
pycurl                    7.43.0.2         py36hdbc3d79_0    defaults
pygments                  2.3.1                      py_0    conda-forge
pyopenssl                 18.0.0                py36_1000    conda-forge
pyrsistent                0.14.7           py36h470a237_0    conda-forge
pysocks                   1.6.8                 py36_1002    conda-forge
python                    3.6.7                h5001a0f_1    conda-forge
python-dateutil           2.7.5                      py_0    conda-forge
python-editor             1.0.3                      py_0    conda-forge
python-oauth2             1.1.0                    py36_0    conda-forge
pyzmq                     17.1.2           py36hae99301_1    conda-forge
readline                  7.0                  haf1bffa_1    conda-forge
requests                  2.21.0                py36_1000    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                40.6.3                   py36_0    conda-forge
six                       1.12.0                py36_1000    conda-forge
sqlalchemy                1.2.15           py36h470a237_0    conda-forge
sqlite                    3.26.0               hb1c47c0_0    conda-forge
terminado                 0.8.1                 py36_1001    conda-forge
testpath                  0.4.2                 py36_1000    conda-forge
tk                        8.6.9                ha92aebf_0    conda-forge
toolchain                 2.3.0                         0    conda-forge
tornado                   5.1.1            py36h470a237_0    conda-forge
traitlets                 4.3.2                 py36_1000    conda-forge
udunits2                  2.2.27.6             h3a4f0e9_1    conda-forge
urllib3                   1.24.1                py36_1000    conda-forge
vcs                       8.0.2018.12.18.00.02.gc7a97688            py_0    cdat/label/nightly
vtk-cdat                  8.2.0.rc2.289.8.0.2018.12.12.20.03.g14c0bb696d  py36h413fb2d_0    cdat/label/nightly
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
x264                      1!152.20180717       h470a237_1    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
zeromq                    4.2.5                hfc679d8_6    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge

Add a strftime function to cdtime object

For compatibilty with python's dateutils see: https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior

mac nightlies seem broken

ImportError: dlopen(/Users/doutriaux1/anaconda2/envs/py3_dan_vtk/lib/python3.6/site-packages/cdtime.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libc++.1.dylib
  Referenced from: /Users/doutriaux1/anaconda2/envs/py3_dan_vtk/lib/python3.6/site-packages/cdtime.cpython-36m-darwin.so
  Reason: image not found

(py3_dan_vtk) doutriaux1@loki:[recipes]:[master*]:[40331]> conda list cdtime
# packages in environment at /Users/doutriaux1/anaconda2/envs/py3_dan_vtk:
#
# Name                    Version                   Build  Channel
cdtime                    3.0.2018.09.25.18.17.gf5ef3bc  py36hdc02c5d_0    cdat/label/nightly

year 0 exists

python -c "import cdtime ; print(cdtime.reltime(-1,'days since 1').tocomp())"

0-12-31 0:0:0.0

BUT

python -c "import cdtime ; print(cdtime.reltime(-1,'days since 0')"

Traceback (most recent call last):
File "", line 1, in
File "/1Tb/miniconda3/envs/jupyter-vcdat/lib/python3.6/site-packages/cdtime/compreltime.py", line 254, in reltime
return _cdtime.reltime(value, units)
ValueError: Invalid relative time units

Fix memory leak in cdtime

As per CDAT/cdat#1424 (comment) and as known (and can be explained) by @doutriaux1

This could be a possible explanation for the memory growth in #1424

Migrated from:
CDAT/cdat#1698

latest cdtime broken on py3

        data = os.path.join(self.basedatadir, "coads_climatology.nc")
        f = cdms2.open(data)
        s = f["SST"]
        s.info()
        print(s.getTime().asComponentTime())

cdtime issue exception that seem to have no base class (python2 maybe 3)

In python 2 for sure, didn't test in python3

import cdtime
try:
   b = cdtime.reltime(4,"days since YYY-XX))")
except:
    print "Failed"

catches the exception properly

BUT

import cdtime
try:
   b = cdtime.reltime(4,"days since YYY-XX))")
except Exception:
    print "Failed"

does not capture the exception

I "think" this is because the exception is raised at the C level and does not inherit from Exception.

This is not a big deal once you know it, but flake8 will complain that you cannot use an except statement w/o an Exception type

python2 LINUX (probably MAC too) broken in py3 merge branch

env on linux created with:

conda create -n py2 -c uvcdat/label/nightly -c conda-forge -c uvcdat cdms2 cdat_info udunits2 nose flake8

gets us nightly cdtime:

c(py2) doutriaux1@drdoom:[/git/genutil]:[python3_continued]:[7080]> conda list cdtime
# packages in environment at /home/doutriaux1/anaconda2/envs/py2:
#
cdtime                    2.12.2017.10.03.11.24.47fd4557e455850a34c7fab5546e8ebe982b615a np111py27h6ee658f_0    uvcdat/label/nightly

Script to use:

import cdtime
c = cdtime.comptime(2017)
c2 = cdtime.comptime(2017)
c.cmp(c2)
c == c2

with 2.12

In [1]: import cdtime
In [2]: c = cdtime.comptime(2017)
In [3]: c2 = cdtime.comptime(2017)
In [4]: c.cmp(c2)
Out[4]: 0
In [5]: c == c2
Out[5]: True

with nightly py3ed:

>>> import cdtime
>>> c = cdtime.comptime(2017)
>>> c2 = cdtime.comptime(2017)
>>> c.cmp(c2)
0
>>> c == c2
False

Last == fails

cdtime bug on NERSC with 365 and 360 day calendars

I have encountered a cdtime bug that seriously impacts analyses of daily data. Now perhaps the bug is only on the NERSC system, I don’t know. I tested it on an old installation that Hari did for me and on jupyter.nersc.gov which Charles probably made.

The script below demonstrates it. It should print out the calendar date of the first and last entry in a nc file, which for CMIP6 daily files is in the name of the file. They do not match for the 365 and 360 day calendars. But they do for the proleptic_gregorian calendar.

About half the CMIP6 models use a 365 day calendar. To test, pick any daily file in ESGF from a CESM model. To test the 360 day calendar, use a HadCM model.

Here is the script.
import MV2 as MV, cdtime,os, cdms2 as cdms, sys, string
var=sys.argv[1]
f=cdms.open(sys.argv[2])

tim=f.dimensionarray('time')
u=f.getdimensionunits('time')

cdtime.DefaultCalendar=cdtime.NoLeapCalendar
tt=f.dimensionobject('time')
if hasattr(tt, 'calendar'):
if tt.calendar=='360_day':cdtime.DefaultCalendar=cdtime.Calendar360
if tt.calendar=='gregorian':cdtime.DefaultCalendar=cdtime.MixedCalendar
if tt.calendar=='365_day':cdtime.DefaultCalendar=cdtime.NoLeapCalendar
if tt.calendar=='noleap':cdtime.DefaultCalendar=cdtime.NoLeapCalendar
if tt.calendar=='proleptic_gregorian':cdtime.DefaultCalendar=cdtime.GregorianCalendar
if tt.calendar=='standard':cdtime.DefaultCalendar=cdtime.StandardCalendar

r = cdtime.reltime(tim[tim.shape[0]-1],u)
last_day=r.tocomp()
r = cdtime.reltime(tim[0],u)
first_day=r.tocomp()
print(first_day,last_day)

import error

import cdtime
Traceback (most recent call last):
File "", line 1, in
ImportError: libnetcdf.so.11: cannot open shared object file: No such file or directory

cdtime conda package version is 2.6.1-UVCDAT

@doutriaux1 @dnadeau4 I am trying to update my cmor conda package. When I attempt to do so, conda tells me it is going to to downgrade my cdtime package as:

cdtime: 2.10-np110py27_0 uvcdat --> 2.6.1-UVCDAT uvcdat

This looks like a bad thing. Is it?

time comparisons

Hi,
time comparisons seem now buggy.
Here are various tests:

In [5]: cdtime.reltime(0,'hours since 2000')>cdtime.reltime(1,'hours since 2000')
Out[5]: False

In [6]: cdtime.reltime(0,'hours since 2000')<cdtime.reltime(1,'hours since 2000')
Out[6]: True

In [7]: cdtime.comptime(2000)>cdtime.comptime(2001)
Out[7]: False

In [8]: cdtime.comptime(2000)<cdtime.comptime(2001)
Out[8]: False

In [9]: cdtime.reltime(0,'years since 2000')>cdtime.reltime(1,'years since 2000')
Out[9]: False

In [10]: cdtime.reltime(0,'years since 2000')<cdtime.reltime(1,'years since 2000')
Out[10]: False

In [11]: cdtime.comptime(2000,1,1,0)>cdtime.comptime(2000,1,1,1)
Out[11]: True

In [12]: cdtime.comptime(2000,1,1,0)<cdtime.comptime(2000,1,1,1)
Out[12]: True

What changed?

cdtime add broken

Example in new and old cdms manual describes how add work, as below:

>>> from cdtime import *
>>> c = comptime(1996,2,28)
>>> r = reltime(28,"days since 1996-1-1")
>>> print r.add(1,Day)
29.00 days since 1996-1-1
>>> print c.add(36,Hours)
1996-2-29 12:0:0.0

However, it is not working in my env, as below:

>>> from cdtime import *
>>> c = comptime(1996,2,28)
>>> r = reltime(28,"days since 1996-1-1")
>>> print(r.add(1,Day))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'Day' is not defined
>>> print(c.add(36,Hours))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'Hours' is not defined

It does not work for Months and Years neither.
My cdtime version is:

-bash-4.1$ conda list cdtime
# packages in environment at /export_backup/lee1043/anaconda2/envs/pmp_nightly_20181128:
#
# Name                    Version                   Build  Channel
cdtime                    3.1.0.2018.11.01.17.45.gb8c370e  py36h9ac9557_0    cdat/label/nightly

@dnadeau4 @doutriaux1 any idea? Thanks in advance.

On mac, py27, cdat8 and nightly fail when installed from cdat channels

On Mac, py27, both cdat8 and nightly installed using conda channels (not the yaml file) fail with:

(nightly) brasov:src danlipsa$ python t.py
Traceback (most recent call last):
File "t.py", line 1, in
import vcs, cdms2, cdat_info
File "/Users/danlipsa/projects/vcs/src/vcs/init.py", line 102, in
from .utils import * # noqa
File "/Users/danlipsa/projects/vcs/src/vcs/utils.py", line 17, in
import cdtime
ImportError: dlopen(/Users/danlipsa/anaconda2/envs/nightly/lib/python2.7/site-packages/cdtime.so, 2): Library not loaded: @rpath/libpng16.16.dylib
Referenced from: /Users/danlipsa/anaconda2/envs/nightly/lib/python2.7/site-packages/cdtime.so
Reason: Incompatible library version: cdtime.so requires version 52.0.0 or later, but libpng16.16.dylib provides version 51.0.0

Do you know how to fix this? (In the future, adding that pin_run_as_build might help, but its a pity you have to pin to a version to use it)

import error - upgrade to openssl-1.1.1?

It appears that conda will upgrade openssl to 1.1.1 even though it appears that the requirement is openssl-1.0.2. The result is:

import cdtime

File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/cdtime/init.py", line 8, in
from .compreltime import *
File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/cdtime/compreltime.py", line 2, in
from . import _cdtime
ImportError: libssl.so.1.0.0: cannot open shared object file: No such file or directory

Might the dependency be corrected so openssl stays at 1.0.2 or upgrade openssl to 1.1.1?

cdtime nightly not generated.

CDAT/cdms#209 (comment)