Coder Social home page Coder Social logo

meom-group / cdftools Goto Github PK

View Code? Open in Web Editor NEW
32.0 16.0 30.0 4.53 MB

A fortran package for analysis and diagnostics on NEMO ocean model output.

Smarty 0.13% Makefile 0.74% TeX 5.68% Shell 0.08% Ada 0.03% HTML 6.19% Fortran 87.10% CSS 0.01% Pug 0.04%

cdftools's Introduction

CDFTOOLS

CDFTOOLS is a diagnostic package written in fortran 90 for the analysis of NEMO model output, initialized in the frame of the DRAKKAR project (https://www.drakkar-ocean.eu/). It is now available on GitHub under the CeCILL license (http://www.cecill.info/licences/Licence_CeCILL_V2-en.html).

CDFTOOLS is an open source package and contributions from other group are welcomed. The Git workflow policy is still to be defined.

Actual version of CDFTOOLS is known as version 4. (See changes in paragraph New features in version 4, below).

Using CDFTOOLS

Cloning the git repository

To retrieve a copy of the CDFTOOLS source code and create a working directory, run the following on the command line:

> git clone https://github.com/meom-group/CDFTOOLS

Compiling CDFTOOLS

All the fortran source are in src/ subdirectory. In src/ there is a Makefile for compiling the sources. The compiler/machines related definitions are supposed to be collected in a make.macro file. Some examples of make.macro are given in the Macrolib directory and can be used as template for a new compiler or new machine. Then the good practice is to make a link

> cd src/

> ln -sf ../Macrolib/macro.MACHINE make.macro

In the make.macro file, the PATH for the netcdf library is specified, as well as compiler name and used flags. In order to activate netcdf4/HDF5 chunking and deflation ( available in some cdftools), you need to set:

NC4=-Dkey_netcdf4

in the make.macro file, otherwise set

NC4=

In order to activate CMIP6 variable naming convention (for input files), you need to set:

CMIP6=-Dkey_CMIP6

Then using make (or even make -j n if you can compile on n cores), you will have the cdftools programs executable available in the bin/ sub directory. The executable files are ignore by git.

Running CDFTOOLS

CDFTOOLS is a collection of programs. Every single program performs one or many computation(s) using a set of input files and output the results as a netcdf file, and eventually also gives some results on the standard output.

CDFTOOLS coding rules imply that a usage message is displayed when just running the tool without any arguments ( or with -h ). At the moment it is the only up to date documentation.

As CDFTOOLS is a collection of programs, a full diagnostic of model output can be performed writing a script using a sequence of tools. This is done for example in the Drakkar Monitoring Tools (DMONTOOLS, soon available on GitHub!).

Coding CDFTOOLS

Coding rules

Syntax

The coding rules are the NEMO coding rules, strictly followed. The idea is that people familiar with NEMO are familiar with CDFTOOLS. In DEV_TOOLS/ some template fortran line are available for program, modules, routine or function headers. Also a template for the usage message.

Run time behaviour

Any cdftool, run without argument or with option -h, should display a short documentation (usage message), similar to a unix man page, describing the purpose of the tool, the syntax (arguments, options, etc...) and giving details on the output files. For some tools, mesh or/and mask files are required to be present in the working directory, with respective name mesh_hgr.nc, mesh_zgr.nc or mask.nc (links are OK). The usage message should indicate the required files.

Example:

> cdfcurl

   usage : cdfcurl -u U-file U-var -v V-file V-var -l levlist [-T] [-8]...
        ... [-surf] [-overf] [-nc4] [-o OUT-file ]
   
  PURPOSE :
    Compute the curl of a vector field, at a specified level.
    If level is specified as 0, assume that the input files are
    forcing files, presumably on A-grid. In this latter case, the
    vector field is interpolated on the C-grid. In any case, the
    curl is computed on the F-point (unless -T option is used).
   
  ARGUMENTS :
    -u U-file U-var : file and variable name for zonal component
    -v V-file V-var : file and variable name for meridional component
    -l levlist    : levels to be processed. If set to 0, assume forcing file
             in input. Example of recognized syntax :
               -l "1,10,30"  or -l "1-20" or even -l "1-3,10-20,30-"
               -l  1 . Note that -l "3-" set a levlist from 3 to the bottom
               
  OPTIONS :
    -T : compute curl at T point instead of default F-point
    -8 : save in double precision instead of standard simple precision.
    -surf : work with single level C-grid (not forcing)
    -overf : store the ratio curl/f where f is the coriolis parameter
    -nc4 : use netcdf4 output with chunking and deflation 1
    -o OUT-file : specify output file name instead of curl.nc
   
  REQUIRED FILES :
     mesh_hgr.nc
   
  OUTPUT : 
    netcdf file : curl.nc
      variables : socurl or socurlt (if -T option), units : s^-1
         or socurloverf, no units (if -overf option)
Improving/modifying existing tool

It is possible to improve (of course !) or modify any tools, but one important law to respect is that the modified tool should still be able to be used with previous syntax, in order to avoid breaking of existing scripts using CDFTOOLS. If for some reason, this is not possible, then a discussion must be done to reach to a common decision. Eventually, some old options must be documented as osbolete in the usage message, which means that they may be removed from a future release.

New features in version 4

Modified user interface

  • All arguments are passed with a -key switch. No more free arguments. Example cdfmoy -l fich1.nc fich2.nc

  • Add -o and -nc4 options in all tools (when relevant). With -o the default output name can be changed, allowing easier paralellisation. With -nc4 output file will use NetCdf4/Hdf5 format with chunking and deflation level 1.

  • Use of environment variable CDFT_xxx for overriding the default names of auxiliary files such as mesh_hgr.nc, mask.nc etc...so far there is support for

    CDFT_MESH_HGR CDFT_MESH_ZGR CDFT_MASK CDFT_BASINS CDFT_COORD

Support for vvl simulations

  • When relevant, the switch -vvl indicates that the vertical metrics is time-varying. Therefore, CDFTOOLS assume that the vertical metrics is saved in the same file than the data.

Support for CMIP6 naming convention

  • When the code is compiled with CPP key key_CMIP6 set, the default variable names are taken form modcdfnames_CMIP6.h90 instead of the standard DRAKKAR names.

Simplification

  • The codes have been cleaned for obsolescences. Coding rules were reinforced.
  • Obsolete tools were removed or merged as options in more generic tools.

Improved documentation.

  • Gathering the usage message into man pages still works (make man). Readibility of the man pages is improved by grouping the tools by category. The usage messages have been reviewed in order to give better information.
  • The man pages are automaticaly translated to an html document that can be vizualized from any browser.

Back to release 3:

  • The last version 3's release has been tagged as v3.0.2. Use this tag if you want to stay at version 3.

Introducing TEOS10 in the CDFTOOLS

  • This introduction is made following the NEMO coding of eosbn2, using a polynomial form (Roquet et Al, Ocean Modelling, 2015) for both EOS80 and TEOS10 equation of state, with a side effect of slightly changing the results (even using EOS80). last commit before the introduction of this change corresponds to tag v4.0.0.
    Also note that is you decide to use TEOS10, all relevant CDFTOOLS have now an option (-teos10) that switch the force the used EOS to be TEOS10. Without this option, EOS80 (polynomial form) is used.
    Last but important :
    When using TEOS10, temperatures should be Conservative Temperature (CT, DegC) and salinity should be Absolute Salinity (SA, g/kg)
    When using EOS80, temperatures should be Potential Temperature (PT, DegC) and salinity should be Practical Salinity (SP, PSU)
    As of Oct. 2021, no sanity check is performed for controling this important point.

Interface with GSW library.

  • GSW library provides a collection of functions and routines linked with the TEOS-10 Equation of state for Sea Water, using Conservative Temperature (CT) and Absolute Salinity (SA). In CDFTOOLS, cdf_gsw is an interface with GSW toolbox. For using it, key_GSW must be defined in make.macro and the libgsw.a must be precompiled on your system. Up to now, only a subset of the GSW functions is interfaced, but cdf_gsw provides a usefull framework for interfacing other functions.

cdftools's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cdftools's Issues

Compilation issues with gfortran

I recently installed https://github.com/meom-group/CDFTOOLS on the Jasmin platform here in the UK. When I tried to compile this with key_CMIP6, this error came up:

& (/'uo','vo','wo','zos',(' ', ji=jp_sqdvarmax-5,jp_sqdvarmax) /)
1
Error: Different CHARACTER lengths (2/3) in array constructor at (1)
11:25
& (/'zos','thetao',(' ', ji=3,jp_cubvarmax) /)
1
Error: Different CHARACTER lengths (3/6) in array constructor at (1)

According to stackoverflow, the F90 standard says you are not allowed to have character strings of different lengths in a declaration like this. It seems gfortran insists on this to the letter, while ifort is more tolerant. I padded out the short strings in both cases so that they were all the same length, and this compiled successfully. Is this a known feature of this release?

How to allocate grids when do zonal integration

Hi meom-group,

Recently I am exploring how to do zonal integration in CNRM-CM5 ocean model. What I am going to do is to compute ocean heat transport (using u, v, T) from the model.

Because the grids are curvilinear, It is impossible to do zonal integration without lat_bounds defining where each grid shall be allocated.

One clue I find is using lat_bnds from thetao output. It has two lists with each one specifying the south and north bound. However, I did not succeed to construct ocean heat transport by using lat_bnds.

Below is the OHT I computed from hfy (already rotated). The shape looks good, and it is smooth in the South Pole since the grids are regular, but very noisy afterward. I believe I didn’t use the correct lat bounds as defined in the model.


image

I dug the source file but didn't find obvious clues as the variable name convection is quite different from that in the CMIP5 archive.

Could anyone offer me some help on the latitude bounds?
Thanks a lot!

Incompatible size between mesh_mask / data.nc does not fail

Malcolm reported me that using CMIP data (overlap region removed) with the usual mesh_mask (overlap regions still there) cdfmean (and probably most of the other tools) do not failed. I didn't tried with the cdftools compiled with some debug options as -CB to see if this issue is still valid in this case.

I am not convince it is possible to fix it without touching all the tools.

BTW: I have the feeling that the getdim function is doing much more the reading the dimension out of netcdf. Probably, this function should be split in 2 (getdim and getgloatt (ccase, cconfig, ctime ...).

cdfmoc compatibility of -rapid and -vvl

Hi,

cdfmoc does not work with -rapid AND -vvl as subroutine rapid_amoc tries to read the vertical grid-spacing from mesh_zgr instead of vfile.

Adding the following lines (copied from the main part of cdfmoc) to subroutine rapid_amoc solves the problem:

IF ( lg_vvl ) THEN
    cn_fe3v = cf_vfil
    cn_ve3v = cn_ve3vvvl
ENDIF

Cheers,
Franziska

abnormal value of cdfmoc with rapid option

Dear All,

I tried to use cdfmoc with "rapid" option to do the AMOC decompostion. The abnormal values were obtained, like the transport of GS is "Infinityf". It seems that the "missing values" of vomecrty and sozotaux are not correctly copied with in the code.

max vrapid 9.9692100E+36
max trapid 24.32919
max srapid 37.34721
max dtaux 9.969209968386869E+036
.........

Is there any suggestion to solve this problem?

Best,
Libin Ma

valid max in cdfcofdis

Valid_max in the distance to coast variable is 1 m. This can cause issue for plotting in python (auto masking of data because data are above this value.

I suggest to set this to 10 000 000 m. For high res model 2700 km should be enough (Oceanic pole of inaccessibility) but in practice for eORCA025 where you removed all the small island before computation of the distance, 10 000 000 m is more appropriate.

stypvar(1)%valid_max = 1.

add topics

I suggest adding topics such as oceanography, ocean-model in the About section.

error using cdfmoc - cannot read output file

Hello,

I've recently been introduced to cdftools and I'm having some issues working with the cdfmoc function.

Working with raw nemo output data and with the required files available I get the following error:

Screenshot from 2022-08-29 10-21-49

It creates an output file (moc.nc) but I can't view it with ncview or ncdump as it does not recognise the file format.

The error message is not very helpful. Can anyone help me?

Anna

There is a bug in file cdfbn2.f90

In file cdfbn2.f90, there is no calculation for awk(:,:,idown), but it will be used to calculation the buoyancy frequency at T-grid.

What exactly does cdfmoy_freq do for 5d to 1mo averaging?

If I'm reading the code of cdfmoy_freq correctly, there is a bug in how it associates the input periods to target averaging periods if the boundaries of the input periods do not exactly match those of the desired outputs.

For the case of 5d inputs and 1mo outputs, it will do the following (considering noleap here):

So this will lead to the first 6 bins (Jan-01 to Jan-30) to be counted for the January mean, the following 5 bins (Jan-31 to Feb-24) to go into the February mean, the following 6 bins (Feb-25 to Mar-26) to go into March, ..., and the 65th to 71st bins (Nov-21 to Dec-21) to be counted for December.

Macrolib/macro.irene (compilation on IRENE)

hello,
Just in case it'd be useful to someone else:

To compile on IRENE (CEA) i had to modify the macro for CURIE (Macrolib/macro.curie) and change :

NCDF = -I$(NETCDF_INCDIR) $(NETCDF_LDFLAGS) -lnetcdff 

to:

NCDF = -I$(NETCDFFORTRAN_INCDIR) $(NETCDFFORTRAN_LDFLAGS) -lnetcdff

Generating a cdf namelist gives errors

I am running cdfnamelist -p to generate a template namelist as some of my variable names are different to the prescribed names set. Running this command results in a file PrintCdfNames.namlist with the contents

 ! Tue Sep 12 16:03:34 2023
 ! Namelist automatically generated by PrintCdfNames 
 ! Do not edit without changing its name ... 
 ! ------------------------------------------

and a .err file like

At line 348 of file modcdfnames.F90 (unit = 3, file = 'PrintCdfNames.namlist')
Fortran runtime error: End of record

Error termination. Backtrace:
#0  0x7f6b475c460f in write_character
	at ../../../libgfortran/io/write.c:1450
#1  0x7f6b475c7eab in nml_write_obj
	at ../../../libgfortran/io/write.c:2253
#2  0x7f6b475c1763 in finalize_transfer
	at ../../../libgfortran/io/transfer.c:4283
#3  0x5602ecf72dbe in ???
#4  0x5602ecf72247 in ???
#5  0x5602ecf7227d in ???
#6  0x7f6b46db8554 in ???
#7  0x5602ecf711d8 in ???

Could anyone advise how I should generate a template namelist, and what I may be doing wrong?

use of cdfvT

Hi,

I have a problem when I try to use the command "cdfvT" to do the transport computation. There is climatologically monthly data which calculated from 10-year data and it is named as gridt.nc, gridu.nc, and gridv.nc. How can I use the command "cdfvT" to estimate the transport computation based on these data files?

Thanks a lot.
Libin

Issues with the `cdfbuoyflx` program and documentation on the computation of `alpha` and `beta`

Hello,
I just found another issue with the flags, not using the flag -teos10 in the cdfbuoyflx program does not change how the beta is computed. In other words, despite not having the flag, the default beta is computed with the beta_2d program (teos10) instead of the the beta_2d_80 program.

Additionally, I'm confused with how beta and alpha are computed. I've compared the values of beta and alpha from the CDFTools and python libraries sw and gsw. I've noticed that associated with the issue above, in the CDFtools both beta and alpha when using the using the beta_2d program is multiplied by a factor of 1026 (the reference density). I'm not sure if I'm missing something, but I found it extremely confusing and there is no documentation anywhere on this.

I haven't checked if this issue extends to other routines...

cdfbn2 always uses eosbn2_pol

To calculate N squared, cdfbn2 always calls eosbn2 without the flag for EOS80, even when ll_teos10=.FALSE. resulting in usage of eosbn2_pol.

reorganisation of sources

  • Might be interesting to have all the f90 sources in a sub directory src/ to be more compliant with TOOLS distributed with NEMO
    • if so need to adapt Makefile

how to compile and install CDFTOOLS to deal with ORCA025 data

Dear All,

I tried to use the CDFTOOLS to deal with ORCA025 data, but it is wrong with information "forrtl: severe (174): SIGSEGV, segmentation fault occurred". However, there is no problem when I deal with ORCA1 data. I think there are some flags are missed when I compile it. The following is the flags I used during the compilation:

export HDF5_LIB=/data01/malb/local/lib
export NETCDF_LIB=/data01/malb/local/lib
export NETCDF_INC=/data01/malb/local/include

libs

LIBS =
-L$(HDF5_LIB) -L$(NETCDF_LIB) -I$(NETCDF_INC)
-lhdf5 -lnetcdf -lnetcdff -lhdf5_hl -L/lib64 -limf -lm -lz

F90 = mpiifort
OMP=
#OMP=-openmp
NC4 = -D key_netcdf4
#NC4 =

#CMIP6 = -D key_CMIP6
CMIP6 =

GSW=
#GSW=-D key_GSW
GSWLIB=
#GSWLIB=-lgsw

FFLAGS = -O $(NC4) $(CMIP6) -assume byterecl -convert big_endian -DHAVE_LIBNETCDF64 $(LIBS) $(OMP)

INSTALL = ../bin
INSTALL_MAN = ../man

Does anybody know how to compile and install the CDFTOOLS to deal with the ORCA025 data?

Thank you very much in advance.

Best,
Libin Ma

cdfsigtrp with -vvl option

Hi,

when using cdfsigtrp with the -vvl key, I get an error on the number of input arguments. I guess this is due to the increment of ireq that should not be done in that case :
l236 ; CALL getarg(ijarg, cf_wfil ) ; ijarg=ijarg+1 ; ireq=ireq+1

Do other users don't bother with volumes changes in this type of calculation, or this has never been tested in that case ?

Thomas.

cdfvFWov

cdfvFWov can't find correct mesh_mask section, which it explicitly asks for.
Needs further changes in cdfio.F90 to read e3v properly, see follow up issue.
FIX:

index 53baab5..0a0b9bc 100644
--- a/src/cdfvFWov.f90
+++ b/src/cdfvFWov.f90
@@ -174,7 +174,12 @@ PROGRAM cdfvFWov
   rmaskn(:,:) = getvarxz(cf_mask, cn_tmask, kj=2,   kpi=npiglo, kpz=npk,    kimin=1, kkmin=1)
   rmaskv(:,:) = getvarxz(cf_mask, cn_vmask, kj=1,   kpi=npiglo, kpz=npk,    kimin=1, kkmin=1)
   de1v(:,:)   = getvar  (cf_hgr,  cn_ve1v,  klev=1, kpi=npiglo, kpj=npjglo, kimin=1, kjmin=1)
-  de3v(:,:)   = getvarxz(cn_fe3v, cn_ve3v,  kj=1,   kpi=npiglo, kpz=npk,    kimin=1, kkmin=1)
+
+! cn_fe3v points to "mesh_zgr.nc", but we explicitly ask for mesh_zgr
+  de3v(:,:)   = getvarxz(cf_zgr, cn_ve3v,  kj=1,   kpi=npiglo, kpz=npk,    kimin=1, kkmin=1)```

Missing value in cdfbottomsig

In cdfbottomsig, temperature and salinity fields are masked by using the "MissingValue" from the salinity file (can also be the temperature file if salinity is included there). In my input files, the "MissingValue" is set to 1e20 but does not apply to the salinity field itself - here, "dry" grid cells are masked with zero.

To overcome problems arising from the resulting unmasked fields I changed line 130 from
zsps = getspval(cf_sfil, cn_vosaline)
to
zsps = 0

I'm not sure, whether this only applies to the setting I'm working with or is a rather general behaviour. Therefore I submit this as an issue rather than a merge request.

Transports for a section

Dear everybody,

I am working with the output of a high resolution model and am currently computing transports along a non-zonal/ non-meridional section.

There are the two possibilities to extract the sections either with cdfsections or cdf_xtrac_brokenline.

cdf sections already turned the velocities perpendicular and parallel to the section - here a great number of points on each part of the section is necessary. Here the grid is then
derived via interpolation of the original e3u and e3v onto the section points.

On the other hand cdf_xtrac_brokenline extracts the closest u or v point closest to the point along the chosen section - it also extracts the grids in horizontal and vertical direction.

I did both methods and get very unexpected peaks due to overestimation of the velocities at breaks of the subsections./ overestimation of transports due to too great grid in the vertical - How do you usually tackle this issue?

Could it be that there is some kind of critical angle between two following sections wich lead to this effect?

How do you usually derive transports along a section in a model?
For Show sections wich only consist of one line/ very low deviations from a line there is no problem-
but scaling up and going to more sophisticated structures this effect has a big influence.
Because I do model - observations comparison I need to stick to the given array-section.

I would be very happy about any input concerning this issue since I have already spend a lot of time on this.

Thank you already!!

256 Character limit for input arguments

Currently, there are many 256-character limits. It would be nice to expand this to allow for (much) more than this.

An example: I have section_*.dat-files I use with cdf_xtrac_brokenline to extract data along all of the WOCE sections. Naming the section files section_WOCEA1.dat,section_WOCEA2.dat,... etc. I get a cldum about 1300 characters.

The limit used to be 80 characters and has already been expanded to 256 characters in: 6652eb4

How far could we go with this? Would it be problematic to allow for LEN=2048 or even more?

cdfmeshmask missing arguments in documentation

cdfmeshmask -n namelist -b bathy.nc -c coordinates.nc

running cdfmeshmask prints statement on how to use it, two points are missing:

  • add variable jpk to provide number of levels at beginning of namelist
  • edit 999999.0 in namelist, use 999 instead so that it fits the following code (cdfmeshmask line 258) IF( ppa1 == 999 .AND. &
    or adapt this code.

additional to that, cdf_mesh_hgr.nc is missing without error message - needs a check

Issue with overlap region in cdfmean ?

Hi,

In cdfmean, the default is to use the 3d mask in meshmask. In this case, the overlap region are not masked. So when we are computing a global mean the overlap region is double counted. I suggest to read in addition maskutil and multiply both mask for each level (example available here)

I agree the suggested fix could be improve: the use of maskutil should only be done is no custom mask used.

Pierre

CDFTOOLS port on GitHub

TODO list

  • Add README.md file : need to be improved
  • Licence CeCILL : add a link to web page
  • Need to upgrade obsolete DOC directory with rst files.

Is it a error in file cdfbn2.f90

To whom it may concern,

When I try to use the cdfbn2 to calculate the buoyance frequency, I found it seems that an error when setting the values of ztemp(:,:,idown) and zsal(:,:,idown) as follows:

148 ztemp(:,:,idown) = getvar(cf_tfil, cn_votemper, npk-1, npiglo, npjglo, ktime=jt)
149 zsal( :,:,idown) = getvar(cf_tfil, cn_vosaline, npk-1, npiglo, npjglo, ktime=jt)

Why always use the level about bottom to set them not change as the loop of vertical levels? It probably should be:
do jk = npk-1,2,-1
.....
ztemp(:,:,idown) = getvar(cf_tfil, cn_votemper, jk-1, npiglo, npjglo, ktime=jt)
zsal( :,:,idown) = getvar(cf_tfil, cn_vosaline, jk-1, npiglo, npjglo, ktime=jt)
.....
end do

Best,
Libin Ma

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.