noaa-emc / gsi-utils

GSI related utilities

GSI-Utils

GSI Utility Tools

These are GSI utilities for various functions.

For installation instruction see here

gsi-utils's Issues

Updates to detect_machine.sh needed after the UFSWM updated the return value

What is wrong?

GSI-utils has a copy of detect_machine.sh. This was copied from UFSWM to maintain consistency. The UFSWM has updated and removed the COMPILER add-on to the variable MACHINE_ID for all platforms.

What should have happened?

A consistent return of use of detect_machine.sh should return the name of the machine w/o COMPILER add-on.

What machines are impacted?

All or N/A

Steps to reproduce

None

Additional information

None

Do you have a proposed solution?

Copy detect_machine.sh from ufs-weather-model to GSI-utils.
https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/detect_machine.sh

Update WCOSS2 Modules

The UFS now points to the hpc-stack suite installed in /apps/test/hpc-stack/i-19.1.3.304__m-8.1.12__h-1.14.0__n-4.9.2__p-2.5.10__e-8.4.2. The GSI-utils modules should be updated to point to the same stack.

interp_inc/HDF5

The interp_inc executable currently does not work with hdf5 past version 1.10.8. This is because later versions of hdf5 for certain routines require that the MPI root process be available for work, but the structure of src/netcdf_io/interp_inc.fd/driver.f90 (in the 344-422 line range) is such that it's not available, so the job hangs. I have successfully tested a workaround where the last process (mype=npes-1) is used rather than the root process (mype=0), but am filing this issue because I'm not very familiar with the code and whether there are reasons to avoid that approach.

Use EPIC-maintained hpc-stacks on RDHPCS

The current hpc-stack installs are frozen. Management of hpc-stack installs on the RDHPCS platforms has moved to EPIC. Update GSI-utils to use the EPIC-installed hpc-stacks on Hera, Orion, and Jet.

Matching global-workflow issue NOAA-EMC/global-workflow#1311

Upgrade to Intel LLVM compilers

The 'classic' Intel compilers (ifort, icc, and icpc) are now discontinued and the new LLVM compilers (ifx and icx) are the only option starting with version 2024. All supported machines with working versions of the LLVM compilers should be upgraded to the LLVM suite.

Upgrade compilers to Intel 2021+

The global workflow and subcomponents need to have their compilers upgraded to Intel 2021+ OneAPI compilers. Months ago, when I tested this on a non-NOAA GCP cloud instance, compiling with Intel 2021.2 resulted in some cases where the EnKF recentering utility would freeze at C192 ensemble resolution. I was unable to diagnose the issue at the time, but I have seen similar behavior in the GSI when attempting to allocate enormous amounts of memory (~200GB) for a single array, so if the issue is still present, looking for large allocations may be a place to start (also possible that this was just an issue with GCP alone).

Adding EFSOI analysis and plotting scripts to GSI-utils

@aerorahul With the separation of GSI-utils from GSI, would there be more willingness to have scripts for working with the output of the EFSOI utility included with the EFSOI code? Something dedicated is necessary just for reading the osense binary file, and much of the analysis was specific to EFSOI, and also not very useful for other applications.

Port to Hercules

The GSI Utilities repository should be ported to Hercules which is now possible following the port to spack-stack.

CRTM is not found when building with GSI on supported platforms

When building with GSI and EnKF, CRTM is required.
The CRTM modulefile needs to be loaded.

bugfix for getsfcensmeanp

there is code pertaining to 3d variables that was probably cut and pasted from getsigensmeanp and doesn't work. Since this utilitiy is not used in operations, no one noticed. PR #9 fixes this.

Use EPIC-managed spack-stack installations for non-production machines

The UFS has migrated to spack-stack/1.4.1 and there is work being done to migrate the GSI to the same (NOAA-EMC/GSI#589) on non-production machines (WCOSS2 is still using Intel/19). The GSI-utils should also be migrated to use spack-stack. This will involve a few upgrades at once: Intel 18 to 2021+, hdf5/1.10.6 to 1.14.0, and netcdf/4.7.4 to 4.9.2. It should be noted that during a port to a NESDIS GCP instance, the only compiler available was Intel 2021. The GSI utilities ran fine at coarser resolutions, but the EnKF recentering utility sometimes failed at ensemble resolution C192.

I am going to try this in one step to bypass #13 (since hpc-stack is minimally supported) and address #15. My suggested plan for testing is to run 40 cycles in the global workflow at all available resolutions up to C384/C192 and 4 additional cycles at C768/C384 as it is the only application that makes use of this utility suite.

Add sfc increment ensemble mean calculation

The new soil analysis requires adding the ensemble mean increment to the deterministic member. For the atmosphere the same code is used to calculate the ensemble mean states and ensemble mean increments: gsi_utils.fd/src/EnKF/gfs/src/getsigensmeanp_smooth.fdgetsigensmeanp_smooth

For the sfc files, this code calculates the ensemble mean states:
gsi_utils.fd/src/EnKF/gfs/src/getsfcensmeanp.fd/

We need to

1. update getsfcensmeanp.fd to also handle sfc increment files, following the atmospheric example.
2. getsfcensmeanp.fd assumes the Noah model variables, and needs to be updated for Noah-MP. For netCDF files, this update is very minor.

Upgrade to spack-stack/1.6.0

The GSI should be upgraded to use the newest suite of spack-stack libraries (version 1.6.0). This includes an upgrade of CRTM to 2.4.0.1, netcdf-fortran to 4.6.1, prod_util to 2.1.1, and Python to 3.11.6.

Remove (some) deprecated utilities.

Specifically:
Radiance_utilities/diag2grads
global_angupdate

These 2 (atleast) have dead links to fortran code from the GSI. These two are currently not built (or buildable) and therefore are not used (that is an assumption on my part unless someone has a secret way of building them that is not in this repository).

Suggesting to remove them from develop (They will always be in the repository history).

@RussTreadon-NOAA @ADCollard @dtkleist
Please let me know if there are concerns regarding this by Jan 1, 2024.