g-w jobs fail on Hercules with NetCDF: HDF error about global-workflow

[BrianCurtis-NOAA]

@RussTreadon-NOAA I think we saw something similar in the UFSWM here: ufs-community/ufs-weather-model#2015 and the TL;DR was to add export I_MPI_EXTRA_FILESYSTEM=ON to the job card.

Can you give that a try to see if you still see that issue?

[WalterKolczynski-NOAA]

While this seems like a different issue, workflow is not yet supported on Hercules due to a Lustre issue with ln on Rocky 9. We've had a ticket open for quite a while now.

[RussTreadon-NOAA]

@BrianCurtis-NOAA , thank you for sharing your insight.

env/HERCULES.env currently contains

export I_MPI_EXTRA_FILESYSTEM=1
export I_MPI_EXTRA_FILESYSTEM_LIST=lustre

I replaced the first line above with

export I_MPI_EXTRA_FILESYSTEM=ON

The 202112200 18Z gdasfcst and enkfgdasfcst_mem002 still aborted with

24:  file: module_write_netcdf.F90 line:          761 NetCDF: HDF error
24: Abort(1) on node 24 (rank 24 in comm 496): application called MPI_Abort(comm=0x84000002, 1) - process 24

In contrast, enkfgdasfcst_mem001 successfully ran to completion.

As a follow on test I removed

export I_MPI_EXTRA_FILESYSTEM_LIST=lustre

while retaining

export I_MPI_EXTRA_FILESYSTEM=ON

A rerun of gdasfcst still aborted with the NetCDF: HDF error error. However, this time enkfgdasfcst_mem002 successfully ran to completion.

The seemingly random nature of this behavior is disturbing.

[RussTreadon-NOAA]

Thank you @WalterKolczynski-NOAA for letting me know that g-w does not support Hercules. It is unfortunate that we can not reliably run cycled global parallels on Hercules. Have we elevated the ln and NetCDF: HDF error issues to management?

[WalterKolczynski-NOAA]

The ln issue has been elevated. The HDF error is new to me.

[RussTreadon-NOAA]

OK, we can keep this issue open for awareness. Hercules is not, at present, a viable option for running global parallels.