raphaeldussin / om4labs Goto Github PK

View Code? Open in Web Editor NEW

5.0 6.0 11.0 1.08 MB

scripts for OM4 analysis

Python 90.23% Jupyter Notebook 8.26% Makefile 1.51%

om4labs's Introduction

OM4-Analysis-Labs

scripts for OM4 analysis

om4labs's People

Contributors

Stargazers

Watchers

Forkers

jkrasting amoebaliz gustavo-marques becki-beadling gmacgilchrist marionalberty mdtocean breichl chanjeunlam

om4labs's Issues

Heat transport function bugs

Two issues are present in this diagnostic:

calculate() function erroneously copied and included from moc.py
rename existing compute() to calculate()

backend support

hardcoded calls to matplotlib backends, will need fix

vertical split scale seems to change direction between matplotlib versions

needs to be investigated, PR #76 fixes recent version

Reference dependency on `intake-xarray`

The current setup.cfg just includes intake:

om4labs/setup.cfg

Line 10 in 9d6836a

intake

This needs to be expanded to include intake-xarray.

CI tests are timing out

The default tests for OM4Labs are all timing out. The resolution of the environment seems to be taking a long time,

moc and seaice issue

I tired to make testing and bumped into some complaints like this.

The error from moc is as follows:

(xesmf_env) hck@hck-XPS-8900 ~/KIM/MOM6-examples/tools/om4labs/testing $ make
om4labs sst_annual_bias_1x1deg --style diff --platform testing
test_data/output/ocean_annual_z_d2_1x1deg.0001-0010.ann.nc
om4labs sst_annual_bias_1x1deg --style compare --platform testing
test_data/output/ocean_annual_z_d2_1x1deg.0001-0010.ann.nc
om4labs sss_annual_bias_1x1deg --style diff --platform testing
test_data/output/ocean_annual_z_d2_1x1deg.0001-0010.ann.nc
om4labs sss_annual_bias_1x1deg --style compare --platform testing
test_data/output/ocean_annual_z_d2_1x1deg.0001-0010.ann.nc
om4labs moc --config OM4p125 --platform testing
test_data/output/ocean_month_z_d2_refined.0036-0040.ann.nc
Traceback (most recent call last):
File "/home/hck/miniconda3/envs/xesmf_env/bin/om4labs", line 4, in
import('pkg_resources').run_script('om4labs==0.0.1', 'om4labs')
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/pkg_resources/init.py", line 650, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/pkg_resources/init.py", line 1446, in run_script
exec(code, namespace, namespace)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/EGG-INFO/scripts/om4labs", line 33, in
CLI()
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/EGG-INFO/scripts/om4labs", line 29, in init
om4labs.diags.dict[args.command].parse_and_run(subargs)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/om4labs/diags/moc/moc.py", line 399, in parse_and_run
imgbuf = run(args)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/om4labs/diags/moc/moc.py", line 370, in run
) = read(dictArgs, varname=varname)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/om4labs/diags/moc/moc.py", line 146, in read
depth = read_topography(dictArgs, coords=ds.coords, point_type="v")
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/om4labs/om4common.py", line 558, in read_topography
xedge = "outer" if (len(coords.xq) == len(coords.xh) + 1) else "right"
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/xarray/core/common.py", line 229, in getattr
"{!r} object has no attribute {!r}".format(type(self).name, name)
AttributeError: 'Dataset' object has no attribute 'xq'
Makefile:19: 'moc.png' 타겟에 대한 명령이 실패했습니다
make: *** [moc.png] 오류 1

The seaice complaint is as follows:

(xesmf_env) hck@hck-XPS-8900 ~/KIM/MOM6-examples/tools/om4labs/testing $ make
om4labs seaice --platform testing --config OM4p125
--obsfile test_data/obs/OBS_NSIDC_sat_NH_T2Ms_sic.nc
test_data/output/ice_1x1deg.003601-004012.siconc_old.nc
Create weight file: bilinear_448x304_180x360.nc
Traceback (most recent call last):
File "/home/hck/miniconda3/envs/xesmf_env/bin/om4labs", line 4, in
import('pkg_resources').run_script('om4labs==0.0.1', 'om4labs')
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/pkg_resources/init.py", line 650, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/pkg_resources/init.py", line 1446, in run_script
exec(code, namespace, namespace)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/EGG-INFO/scripts/om4labs", line 33, in
CLI()
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/EGG-INFO/scripts/om4labs", line 29, in init
om4labs.diags.dict[args.command].parse_and_run(subargs)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/om4labs/diags/seaice/seaice.py", line 423, in parse_and_run
imgbuf = run(args)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/om4labs/diags/seaice/seaice.py", line 399, in run
model, obs = calculate(ds, dobs, region=dictArgs["region"])
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/om4labs/diags/seaice/seaice.py", line 189, in calculate
obs["ac_r"] = regrid.curv_to_curv(obs["ac"], model["ac"], reuse_weights=False)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/om4labs-0.0.1-py3.7.egg/om4labs/diags/seaice/regrid.py", line 18, in curv_to_curv
regridder = xe.Regridder(src, dst, "bilinear", reuse_weights=reuse_weights)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/xesmf/frontend.py", line 235, in init
self._write_weight_file()
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/xesmf/frontend.py", line 279, in _write_weight_file
ignore_degenerate=self.ignore_degenerate)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/xesmf/backend.py", line 280, in esmf_regrid_build
ignore_degenerate=ignore_degenerate)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/ESMF/util/decorators.py", line 64, in new_func
return func(*args, **kwargs)
File "/home/hck/miniconda3/envs/xesmf_env/lib/python3.7/site-packages/ESMF/api/regrid.py", line 136, in init
raise ImportError(msg)
ImportError: Regrid(filename) requires PIO and does not work if ESMF has not been built with MPI support
Makefile:40: 'seaice.nh.png' 타겟에 대한 명령이 실패했습니다
make: *** [seaice.nh.png] 오류 1

My conda list is as follows:

(xesmf_env) hck@hck-XPS-8900 ~/KIM/MOM6-examples/tools/om4labs/testing $ conda list --explicit

This file may be used to create an environment using:

$ conda create --name --file

platform: linux-64

Default dictionary generator

It would be helpful to have a module that generates a dictionary similar to the default_diag_parser. This would help with running the om4lab diagnostics in the Jupyterlab environment. This dictionary would be an input into the run function of the diagnostic.

path = '/glade/scratch/gmarques/OM4_025_JRA/'
dict_om4 = dict(static=path+'20180101.ocean_static.nc',
               outdir='./',
               infile=path+'20??????.ocean_annual_rho2.nc',
               topog=path+'20180101.ocean_static.nc',
               interactive=False,
               hgrid=None,
               label='OM4_025',
               format='png'
               )

om4labs.diags.moc.run(dict_om4)

Organize common functions

We are starting to accumulate common functions. To avoid monolithic catch-all scripts, we could consider some organization up-front.

Add cmocean to package requirements

Recently merged stress curl diagnostic requires "cmocean" as a package dependency.

Basin mask for MOC script on different grid

The basin mask field used in the MOC script is located on t-cell points and needs to be translated to the velocity grid points. @raphaeldussin has some ideas for this.

Detect symmetric mode for MOC script

An extra land point over Antarctica appears when running the model in symmetric memory mode. A check for this condition needs to be added.

cmip_basins dependency issue

cmip_basins is being flagged as a dependency but is not being downloaded on install. URL to package likely needs to specified.

getting an import error from vcr

Getting this error:

`import xarray as xr
from vcr import util , conserve
import numpy as np
import seawater
import time
import pydap

ImportError Traceback (most recent call last)
Input In [13], in <cell line: 2>()
1 import xarray as xr
----> 2 from vcr import util , conserve
3 import numpy as np
4 import seawater

ImportError: cannot import name 'conserve' from 'vcr' (/nbhome/abs/miniconda/envs/myenv/lib/python3.10/site-packages/vcr/init.py)

Duplicate spherical area grid cell functions

standard_grid_cell_area and compute_area_regular_grid both compute cell areas for a spherical grid.

compute_area_regular_grid is avoids the for loop and is thus more efficient, but rE should be double-checked against the FMS constants. @raphaeldussin - I propose we use the FMS value as the default.

Different calendar behavior for intake-xarray

When loading the sea ice observational file (OBS_NSIDC_sat_NH_T2Ms_sic.nc) through intake-esm, time is represented as a floating point type and calendar attributes are getting lost:

"'float' object has no attribute 'year'"

Import structure unfriendly to dora

This packaging / import structure is unfriendly to Dora. It requires setup.py to be run every time and the package to be installed if updates are made to the package. The main issue is that the account where the web service runs has limited permissions capabilities to do this and our strategy with @adcroft to create a "menu of commits" won't be compatible with this.

https://github.com/raphaeldussin/OM4-Analysis-Labs/blob/510bd30da1eca66f5209b6ee607dfe1a04d83184/OM4_Analysis_Labs/annual_bias_1x1deg.py#L4-L5

Warnings issued by xESMF

Warning errors are coming from seaice.py at Line 188:

obs["ac_r"] = regrid.curv_to_curv(obs["ac"], model["ac"], reuse_weights=False)

Produces:

~/miniconda3/envs/python38/lib/python3.8/site-packages/xarray/core/dataarray.py:679: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  return key in self.data
~/miniconda3/envs/python38/lib/python3.8/site-packages/dask/array/core.py:377: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  o = func(*args, **kwargs)
~/miniconda3/envs/python38/lib/python3.8/site-packages/xesmf/frontend.py:450: FutureWarning: ``output_sizes`` should be given in the ``dask_gufunc_kwargs`` parameter. It will be removed as direct parameter in a future version.
  dr_out = xr.apply_ufunc(

Module versions:

dask                      2.30.0                     py_0    conda-forge
dask-core                 2.30.0                     py_0    conda-forge
esmf                      8.0.1           nompi_h39dff87_2    conda-forge
esmpy                     8.0.1           nompi_py38h5410a82_2    conda-forge
xarray                    0.16.1                     py_0    conda-forge
xesmf                     0.4.0              pyhd8ed1ab_0    conda-forge

Standardize diagnostic arguments

The different diagnostics have slightly different command line argument options. It would help to develop a common vocabulary and set of options across the diagnostics.

Heat transport issues warning on annual data

Annual data produces the following error:

site-packages/dask/array/numpy_compat.py:40: RuntimeWarning: invalid value encountered in true_divide
  x = np.divide(x1, x2, out)

Diagnostic is attempting to do a time average when the length of the time dimension is 1.

Fix Vertical Split Scale function for Matplotlib version > 2.x

Vertical split scale function is failing:

om4labs moc --model OM4p125 --platform testing test_data/output/ocean_month_z_d2_refined.0036-0040.ann.nc
Traceback (most recent call last):
  File "/net2/rnd/anaconda3/envs/analysis/bin/om4labs", line 4, in <module>
    __import__('pkg_resources').run_script('om4labs==0.0.1', 'om4labs')
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/pkg_resources/__init__.py", line 667, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1464, in run_script
    exec(code, namespace, namespace)
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/om4labs-0.0.1-py3.6.egg/EGG-INFO/scripts/om4labs", line 33, in <module>
    CLI()
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/om4labs-0.0.1-py3.6.egg/EGG-INFO/scripts/om4labs", line 29, in __init__
    om4labs.diags.__dict__[args.command].parse_and_run(subargs)
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/om4labs-0.0.1-py3.6.egg/om4labs/diags/moc/moc.py", line 353, in parse_and_run
    imgbuf = run(args)
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/om4labs-0.0.1-py3.6.egg/om4labs/diags/moc/moc.py", line 341, in run
    fig = plot(y, z, msftyyz, dictArgs["label"])
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/om4labs-0.0.1-py3.6.egg/om4labs/diags/moc/moc.py", line 289, in plot
    _plotPsi(yy, z, psiPlot, ci, "Atlantic MOC [Sv]", cmap=cmap)
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/om4labs-0.0.1-py3.6.egg/om4labs/diags/moc/moc.py", line 272, in _plotPsi
    plt.gca().set_yscale("splitscale", zval=[0, -2000, -6500])
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 3668, in set_yscale
    ax.yaxis._set_scale(value, **kwargs)
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/matplotlib/axis.py", line 826, in _set_scale
    self._scale = mscale.scale_factory(value, self, **kwargs)
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/matplotlib/scale.py", line 729, in scale_factory
    return _scale_mapping[scale](axis, **kwargs)
  File "/net2/rnd/anaconda3/envs/analysis/lib/python3.6/site-packages/om4labs-0.0.1-py3.6.egg/om4labs/m6plot/formatting/VerticalSplitScale.py", line 34, in __init__
    mscale.ScaleBase.__init__(self)
TypeError: __init__() missing 1 required positional argument: 'axis'
make: *** [moc.png] Error 1

timetuple fails on DORA SSS plot

@jkrasting

om4labs/om4labs/om4common.py

Line 64 in 202cb82

t1 = tuple(ds["time_bnds"].values[-1][-1].timetuple())

triggers a failure of the type:

AttributeError: 'numpy.timedelta64' object has no attribute 'timetuple'
      args = ("'numpy.timedelta64' object has no attribute 'timetuple'",)
      with_traceback = <built-in method with_traceback of AttributeError object>

when trying to plot SSS bias with odiv-75

Rotate y-ward transport to true northward transport

This would be preferred for both the MOC and heat transport diagnostics. @raphaeldussin to explore this, potentially invoking capability from xgcm.

seaice diags needs refactoring

plot function does not work:

fig = plot(model, obs, regridded, valid_mask)                                                                                                                                                                          
/net2/rnd/anaconda3/envs/repro/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py:1491: MatplotlibDeprecationWarning: shading='flat' when X and Y have the same dimensions as C is deprecated since 3.3.  Either specify the corners of the quadrilaterals with X and Y, or pass shading='auto', 'nearest' or 'gouraud', or set rcParams['pcolor.shading'].  This will become an error two minor releases later.
  X, Y, C = self._pcolorargs('pcolormesh', *args, allmatch=allmatch)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-20-8404f36099d9> in <module>
----> 1 fig = plot(model, obs, regridded, valid_mask)

/net2/rnd/anaconda3/envs/repro/lib/python3.8/site-packages/om4labs-0.0.1-py3.8.egg/om4labs/diags/seaice/seaice.py in plot(model, obs, regridded, valid_mask, label, region, month)
    272     plotdata = (model["ac"][month_index] * 100.0).to_masked_array()
    273     plotdata = np.ma.masked_where(valid_mask, plotdata)
--> 274     cb1 = _plot_map_panel(ax, x, y, plotdata, extent=extent)
    275     ax.set_title(f"Model - Years {model.time[0]} to {model.time[1]}")
    276     fig.colorbar(

/net2/rnd/anaconda3/envs/repro/lib/python3.8/site-packages/om4labs-0.0.1-py3.8.egg/om4labs/diags/seaice/seaice.py in _plot_map_panel(ax, x, y, plotdata, cmap, vmin, vmax, extent, contour)
    235     cmap.set_bad(color="#555555", alpha=1)
    236     ax.set_extent(extent, ccrs.PlateCarree())
--> 237     cb = ax.pcolormesh(
    238         x, y, plotdata, transform=ccrs.PlateCarree(), cmap=cmap, vmin=vmin, vmax=vmax
    239     )

/net2/rnd/anaconda3/envs/repro/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py in pcolormesh(self, *args, **kwargs)
   1457                              ' consider using PlateCarree/RotatedPole.')
   1458         kwargs.setdefault('transform', t)
-> 1459         result = self._pcolormesh_patched(*args, **kwargs)
   1460         self.autoscale_view()
   1461         return result

/net2/rnd/anaconda3/envs/repro/lib/python3.8/site-packages/cartopy/mpl/geoaxes.py in _pcolormesh_patched(self, *args, **kwargs)
   1489         allmatch = (shading == 'gouraud')
   1490 
-> 1491         X, Y, C = self._pcolorargs('pcolormesh', *args, allmatch=allmatch)
   1492         Ny, Nx = X.shape
   1493 

ValueError: too many values to unpack (expected 3)

too many computations/reductions are made in the plot function which is against the design of having calculate output numpy arrays ready for the plot part

om4labs installation fails because xwavelet is missing

Running following commands:

git clone https://github.com/raphaeldussin/om4labs.git
cd om4labs/
pip install .

gives following error

ERROR: Could not find a version that satisfies the requirement xwavelet (from om4labs) (from versions: none)
ERROR: No matching distribution found for xwavelet

Using python setup.py install:

Processing dependencies for om4labs==0.0.1
Searching for xwavelet
Reading https://pypi.org/simple/xwavelet/
Couldn't find index page for 'xwavelet' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
No local packages or working download links found for xwavelet
error: Could not find suitable distribution for Requirement.parse('xwavelet')

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.