cosima / cosima-recipes

Example recipes for analyzing model output using the cosima-cookbook infrastructure

cosima-recipes

Example recipes using the cosima-cookbook infrastructure.

To Get Started with the cosima-cookbook, clone this repository locally, probably best in your local space on one of the NCI HPC machines so you can have access to model output. The repository includes a bunch of examples with which you can begin to construct your own analysis code.

Contributing

If you made a notebook for analysing something that is not already included in the recipes, then please consider contributing back to the repository.

To make a contribution follow the steps laid out in the beginner's guide on how to contribute. If they sound intimidating then don't worry! Just raise an issue explaining briefly what the contribution you want to make is and we'll help out with the process!

Repository contents

Tutorials

The notebook COSIMA_CookBook_Tutorial.ipynb outlines the basic philosophy of the cosima-cookbook. This is the best place to start if you are not familiar with the cosima-cookbook. Also included here are some other tutorials, either related to the cookbook (e.g., Make_Your_Own_Database.ipynb) or more general (Making_Maps_with_Cartopy.ipynb).

Don't miss out the tutorial about using cosima-cookbook's explore submodule to find out about available experiments and variables in a database. (The explorer tutorial is better viewed either via nbviewer or by running the jupyter notebook yourself.)

Documented Examples

Νotebooks for simple and not-so-simple diagnostics which are well-documented and explained. If you can find an example that suits your purpose, this is the best place to start.

ACCESS-OM2-GMD-Paper-Figs

Νotebooks to reproduce (as far as possible) the figures from the ACCESS-OM2 model announcement paper (GMD, 2020). These notebooks are mostly uncommented, but they should be functional. They are intended to demonstrate methods to undertake the calculations used in the paper.

Contributed Examples

Α number of examples which are valuable, but not commented sufficiently to make it into DocumentedExamples. An aspirational goal is to eliminate this class of example.

cosima-recipes's Issues

Documented examples: Fix zonal_mean.ipynb

First of all the notebooks' name is non-descriptive. Zonal_mean_Temperature.ipynb is a compelling alternative.

Secondly, the line

memory = Memory(cachedir='/g/data1/v45/cosima-cookbook/', verbose=0)

seems outdated or deprecated.

More problems may arise after the second cell manages to evaluate...

Use OceanSpy to extract "observational data" from ACCESS-OM2

OceanSpy is a python package that can be used to extract observational-like data from model output.

A notebook that uses this to extract, e.g., some virtual mooring data from the model and compare with real data sounds like a need contribution to the cookbook-recipes.=

Meridional heat Transport Plot

This task involves updating ContributedExamples/plot_mht.ipynb to the latest version of the cookbook, and documenting to create a DocumentedExample. It could also benefit from incorporating code from ACCESS-OM2-1-025-010deg-report/figures/meridional_heat_transport/Meridional_Heat_Transport.ipynb.

cf CESM POP diags http://www.cesm.ucar.edu/working_groups/Ocean/metrics.html

AttributeError: module 'cosima_cookbook' has no attribute 'database'

Not sure if my issue is related to #17 but I just started going through COSIMA_CookBook_Tutorial and got the following error:

session = cc.database.create_session()

---

AttributeError Traceback (most recent call last)
in
----> 1 session = cc.database.create_session()

AttributeError: module 'cosima_cookbook' has no attribute 'database'

I am accessing via jupyter_vdi.py and uses conda env:analysis3-19.04.

Zonal mean diagnostics need to account for non uniform grid

Simple mean of xt_ocean is incorrect.

Global Ocean Volumetric Temperature and Salinity Change

Monitoring model stability

The notebook ContributedExamples/plot_w_bt.ipynb was developed to monitor the maximum vertical velocity, w_bt, and plot it to show where the model is most unstable. Needs updating and documenting.

time values for CICE output are shifted by a month using open_mfdataset

This may not be specific to the cosima cookbook, but I have a problem reading the time values correctly for the CICE output using xarray. Basically, the time values are shifted by a month after loading the files with open_mfdataset.

For example, when loading monthly-mean output for 1 year (12 files), the loaded xarray dataset will have time values: Feb. of Year 1 ... Jan. of Year 2, while expecting Jan. of Year 1 ... Dec. of Year 1.

This is problematic for subsequent analysis like "groupby('time.year').mean('time')" because it produces two values (for Year 1 and Year 2) instead of one.

I think a similar problem is reflected in Out[61] of "ContributedExamples/Ice Diagnostics.ipynb", where:

• time (time) datetime64[ns] 2010-02-15 2010-03-16T12:00:00 2010-04-16 ...

while the first time value should be 2010-01-15.

cf Obs4MIPs dataset and documentation https://climatedataguide.ucar.edu/climate-model-evaluation/obs4mips-datasets-and-documentation

Documented Example for WMT diagnostics

Put together an example which illustrates some of the uses of the temperature (or salinity)-space WMT diagnostics.

The lowest hanging fruit would be plotting the global internal heat content budget in temperature space.

Matlab code to do this is contained here - https://github.com/rmholmes/MOM_diatherm_analysis . Here the idea would be to start converting some of this code to python.

T/S diagams

Update tropical Indian Ocean plot.

Update ContributedExamples/u_seasonal_cycle_ind_eq.ipynb to latest version of cookbook.

pcolormesh not working in Model_Resolution_Comparison

Model_Resolution_Comparison used to work but now fails at cell 5 with
ValueError: x and y arguments to pcolormesh cannot have non-finite values or be of type numpy.ma.core.MaskedArray with masked values
which is weird because as far as I can tell the plotted variables are not masked and have no nans or infs.

full error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-507fade5d9d1> in <module>()
      5 ax.add_feature(cft.LAND,color='gray')
      6 plt.pcolormesh(lons1, lats1, uv1, cmap=cm.cm.thermal, 
----> 7                vmax=1, transform=ccrs.PlateCarree())
      8 plt.colorbar(shrink=0.6,label='Speed (m/s)',extend='max')
      9 plt.title('(a) ACCESS-OM2')

/g/data3/hh5/public/apps/miniconda3/envs/analysis3/lib/python3.6/site-packages/matplotlib/pyplot.py in pcolormesh(*args, **kwargs)
   3186                       mplDeprecation)
   3187     try:
-> 3188         ret = ax.pcolormesh(*args, **kwargs)
   3189     finally:
   3190         ax._hold = washold

/g/data3/hh5/public/apps/miniconda3/envs/analysis3/lib/python3.6/site-packages/cartopy/mpl/geoaxes.py in pcolormesh(self, *args, **kwargs)
   1389                              ' consider using PlateCarree/RotatedPole.')
   1390         kwargs.setdefault('transform', t)
-> 1391         result = self._pcolormesh_patched(*args, **kwargs)
   1392         self.autoscale_view()
   1393         return result

/g/data3/hh5/public/apps/miniconda3/envs/analysis3/lib/python3.6/site-packages/cartopy/mpl/geoaxes.py in _pcolormesh_patched(self, *args, **kwargs)
   1433         allmatch = (shading == 'gouraud')
   1434 
-> 1435         X, Y, C = self._pcolorargs('pcolormesh', *args, allmatch=allmatch)
   1436         Ny, Nx = X.shape
   1437 

/g/data3/hh5/public/apps/miniconda3/envs/analysis3/lib/python3.6/site-packages/matplotlib/axes/_axes.py in _pcolorargs(funcname, *args, **kw)
   5183                 if np.ma.is_masked(X) or np.ma.is_masked(Y):
   5184                     raise ValueError(
-> 5185                         'x and y arguments to pcolormesh cannot have '
   5186                         'non-finite values or be of type '
   5187                         'numpy.ma.core.MaskedArray with masked values')

ValueError: x and y arguments to pcolormesh cannot have non-finite values or be of type numpy.ma.core.MaskedArray with masked values

xarray's `.mean()` function is simply an arithmetic mean

xarray's .mean(dim='dimension') is simply an arithmetic mean over that dimension.
It does not take into account that, e.g., dimensions's coordinates could be on a non-uniform grid.

Remark(s) on the use of .mean() should be included in the notebooks.

Probably I'm duplicating old news here? This #12, #40, and #85 are related.

[While this may seem like triviality or common knowledge to pythonistas, newbies in xarray (like myself) may miss on reading the fine print in xarray's docs and work under the assumption that .mean() takes car of non-uniform grids.]

Run notebooks online via binder

Is there a way the repository can be on https://mybinder.org so that jupyter-notebooks can be run online? I guess it's hard (if not impossible) since access to NCI is needed. But if is a possibility it sounds very neat thing to do.

@AndyHoggANU do you think that would be useful?
@aidanheerdegen, @angus-g any ideas whether it's at all feasible?

Update Model Resolution Comparison

The notebook ContributedExamples/Model_Resolution_Comparison.ipynb is just a simple set of files to compare the flow that we obtain from different resolutions of the model.
This could be updated.
See also https://github.com/AndyHoggANU/CC_sandbox/blob/master/ResolutionComparison.ipynb for other ideas on how to plot this.

MOC Routines

I've just added a pull request for some of the MOC density-space calculations that I've been working on. They work but I'm not really happy with them because:

• They don't yet include the submesoscale parameterised component;
• The addition of GM requires a specific flag;
• I think we may also want to add the ability to diff between runs
• I would like to add functionality to specify a date range, rather than just doing the last n files.

Documented example for plotting T & S biases vs. WOA13

Put together a documented example showing how to plot differences between ACCESS-OM2 T & S fields and the WOA13 data set. I don't think there is one yet and I can just update my own scripts.

Compare sea ice with obs

Make a DocumentedExample which merges:

• ContributedExamples/Ice Validation mom025.ipynb
• ACCESS-OM2-1-025-010deg-report/figures/ice_maps/ice_maps.ipynb
  *ACCESS-OM2-1-025-010deg-report/figures/ice_timeseries/ice_timeseries.ipynb
  to compare ice output with observations

Fix kinetic_energy.ipynb from Documented Examples

The kinetic_energy.ipynb has some cells not evaluating properly.

On top of that some further possible enhancements may include:

• Add %config InlineBackend.figure_format='retina' below line %matplotlib inline so that plots are shown in the screen with higher quality; particularly relevant for retina display users.
• MKE is now defined as the sum over vertical levels MKE = KE.mean('time').sum('st_ocean'). This makes no physical sense. Better redefine it to be the vertical integral from top to bottom taking into account that z-levels are not equidistant.
• There exists now a notebook section labeled Functions which is not documented. Add remarks on why those functions need to be defined in that manner and where do they appear useful. (If they don't appear useful, then simply delete perhaps?)
• ... (feel free to do more!)

2nd order conservative interpolation of obs fields

Should we use 2nd order conservative interpolation of obs fields (eg WOA) now that it is available?
COSIMA/access-om2#52 (comment)

Automatic gallery page

If the plan is for all the notebooks in this repository to be evaluated with images, I can take a crack at automatically generating a gallery page with plots, and links to the notebooks.

Documented examples: Kinetic Energy notebook out of date

The kinetic energy notebook @jmunroe wrote uses the old COSIMA Cookbook API, and so is not consistent with the latest version, 0.3.3beta2.

https://github.com/COSIMA/cosima-recipes/blob/master/DocumentedExamples/kinetic_energy.ipynb

I used that as a base to create a notebook for Shweta. I don't think it is suitable to replace James' as it only uses surface velocities, but if someone wanted to use it as a basis to work from it is here:

https://gist.github.com/aidanheerdegen/2a5166ae7cf1dc4d15f623139fcd51c6

Update and document surface water mass transformation plots

Update ContributedExamples/surface_water-mass_transformation.ipynb

Tutorial: Making animations

Write a tutorial on how to make simple animations with the cookbook. Examples could be to show the seasonal cycle of Antarctic sea ice, or a transect that is dragged through a static, or moving, ocean...

relative vorticity

Does someone have a kernel for the relative vorticity:

\zeta = v_x - u_y

to add to this cookbook?

To that, we should add the Rossby number

Ro = \zeta/f

where f=Coriolis parameter.

Allow anyone to add issues to hackathon project

I still cannot add issues to the hackathon project or move them to "claimed". Anyone have a fix for this? @navidcy?

Showcase experiments in default database?

Using the experiments that are available on hh5 and indexed in the "default" cosima cookbook database, what experiments are good to use as example datasets in the recipes?

Currently, there are 157 experiments indexed but I'll assume are some are more useful than others. There are 75 experiments with greater than 1000 netcdf files.

Could we identify some experiments based on ACCESS-OM2 at each of 0.25 / 1 / 0.1 deg that are good to focus on? Could some identify significant spin up / control runs and a few sample variant experiments? Specifically, which are the experiments are reported in the Kiss et al. (2020) paper on ACCESS-OM2?

Applying filtering/coarse-graining techniques on ACCESS-OM2 output

in the spirit of Aluie et al 2017, JPO, doi:10.1175/JPO-D-17-0100.1

module 'cosima_cookbook.querying' has no attribute 'get_variables'

I am trying to run cosima_receipe simple example because of all my scripts in old_version of cosima_receips (with kernel 19.01). I am trying to update with the new version (with kernel 19.07). I can't get_variables and if I gave a path to load the dataset, then the kernel is dying all the time. I am unable to move forward with simple plots too. Please, kindly help me out.

Update SSH Plots

This task involves updating ContributedExamples/plot_ssh.ipynb for the latest version of the cookbook, and documenting. Result would be a plot to compare global SSH and its variance to AVISO obs.
Could also be informed by ACCESS-OM2-1-025-010deg-report/figures/dynamic_topo/DynamicTopography.ipynband ACCESS-OM2-1-025-010deg-report/figures/ssh_variance/SeaLevelRMS.ipynb

ocean annual cycle diagnostics - eg meridional heat transport, cf obs http://www.cgd.ucar.edu/cas/catalog/ocean/ohts/index.html

Documented examples: Fix Transport_Through_Straits.ipynb

The Transport_Through_Straits.ipynb notebook is giving errors. I caught a typo (Straights-->Straits) which might do the job.

Currently, the notebook supports computing transports "only along lines of either constant latitude or longitude". Could we extend this to any general curve/straight line we like?

Also, I noticed division by some 1036 at some point. This smells like "the mean density" of the ocean. Why not define a variable with this value? This will make things clearer...

Furthermore, at the beginning of the notebook the Theory section mentions that the variables to be used will be tx_trans and ty_trans which are in units of Sv. Then why subsequently one divides again with 1e6 and 1036? Something's fishy or unclear here...

Overturning circulation by basin

Take the example in ContributedExamples/atlantic-indopacific-basin-averaged-MOC.ipynb and create a generic method to plot overturning circulation within the Atlantic or Indian-Pacific basin.

Create a tutorial to plot sea ice variables

It would be nice to see a careful tutorial showing how ice variables can be loaded and plotted.
This could replace ContributedExamples/Ice Diagnostics.ipynb.

Update Tropical Pacific plots

This could be a merger of:

• ContributedExamples/plot_TAO_140W_0N.ipynb
• ACCESS-OM2-1-025-010deg-report/figures/equatorial_pacific/Equatorial_Pacific.ipynb
  along with documentation to create a DocumentedExample.

Tutorials/COSIMA_CookBook_Tutorial.ipynb updates

Need to include session = cc.database.create_session() and replace db with session throughout COSIMA_CookBook_Tutorial.ipynb

Put observational datasets in cookbook

Email from @AndyHoggANU from 8 Nov last year:

---

A key aspect of our model evaluation is the comparison of spinup simulations with observations. These evaluations will provide us with objective measures of model sensitivity to forcing, relative to model errors. They will also enable confidence in the model mean state.

Incorporation of observations into the COSIMA cookbook can take several forms:

Simple circulation metrics (e.g. Drake Passage transport, ITF transport, AMOC) can be hardwired into diagnostics scripts. For example, we could plot mean and standard deviation of these values on the timeseries plots. In these cases, the code & figure should include a reference to the relevant work.
Sea ice. We need sea ice climatology (2D) as well as timeseries of sea ice volume, area, etc. In the case of 2D maps we will likely want to compare model means with observed means - but this can be achieved by plotting sea ice boundaries from obs and model on the same map; in other words, regridding is not needed.
Temperature and salinity. This is already available as a monthly climatology from WOA13, and has been regridded onto the 3 existing model resolutions.
Mixed layer depth climatology. Again, comparisons will probably not need regridding.

Directory structure:
Relevant climatologists and datasets should all be saved in a cosima-cookbook friendly way in the cosima/ directory on hh5. A suggested structure involves putting all of the climatology data under a single directory:

…/cosima/
|—access-om2 …
|— access-om2-025 …
|—climatology
|     `—woa13
|     |   `—01 …
|     |   `—025 …
|     |   `—10…
|     `— sea ice
|     `— MLD

Alternatively, we could consider a separate directory for each climatology?

CF Compliance:
We want to ensure these datasets are saved in a consistent way. I’m still not convinced that we currently do this for the woa13 data, but open to advice from James.

Adding to cookbook:
Once data files are in place, let’s put examples on the cookbook regarding how to use each different dataset for comparison purposes.

Documented Example to plot Barotropic Streamfunction

Suggestion is to merge very old code from ContributedExamples/BarotropicStreamFunction.ipynb and ContributedExamples/plot_bsf.ipynb
with code from
https://github.com/COSIMA/ACCESS-OM2-1-025-010deg-report/tree/master/figures/BarotropicStreamfunction
neither of which work with the current Cookbook.

Also, make some nice-looking maps, etc.

Clear up old notebooks

By the end of COSIMA Hackathon 2020, all notebooks that are still not working or they are not documented to the bare minimum will be moved to sandbox.

Documented examples: Fix scalars.ipynb

The scalars.ipynb notebook is failing from the very start --- this needs fixing.

Lawn Mowing Automation

Use the cosima-cookbook to automate the lawn mowing procedure. Task to be undertaken by @amy-gardening.

[this issue is, apparently, a joke that stems from the hackathon promotional video]

Hovmoller - depth plots

This task involves updating ContributedExamples/Hovmoller_Temperature_Depth.ipynb to the latest version of the cookbook.
If possible, add more comments and make it a documented example.

Find out which are the available variables from an experiment

https://github.com/COSIMA/cosima-recipes/blob/master/Tutorials/COSIMA_CookBook_Tutorial.ipynb
I am trying to do

cc.querying.get_variables(session, experiment='025deg_jra55v13_ryf8485_gmredi6', frequency='1 monthly')

to find out the names of the available variables I can get with cc. I learned this from here:
https://github.com/COSIMA/cosima-recipes/blob/master/Tutorials/COSIMA_CookBook_Tutorial.ipynb

However, I'm getting an error:

Object `cc.querying.get_variables` not found.

Was cc.querying.get_variables deprecated? If so, then the tutorial should be updated or deleted. (I can submit a PR for that.)

Plotting MLD

Make an example to plot Mixed Layer Depth from the model. This could be based on ContributedExamples/plot_mld.ipynb.

Documented Example to plot wind stress curl

Would be great to update:
ContributedExamples/CC_WindStress_Curl.ipynb
As well as making it more cookbook-ie, this could also benefit from using xgcm to take derivatives etc.

Documented Example to plot model bathymetry

This task involves a merger of
ContributedExamples/BathymetryDetails.ipynb
with
ContributedExamples/Bathymetry.ipynb
along with some updating of code and additional documentation.

Devise naming structure for scripts in this repo

Computing geostrophic surface velocities from SSH

This will be a useful contribution.