Re-enable C48mx500_3DVarAOWCDA test about global-workflow

[guillaumevernieres]

I didn't mean to close it @WalterKolczynski-NOAA , sorry! We just merged a fix in the GDASApp.

[JessicaMeixner-NOAA]

FYI @guillaumevernieres @CatherineThomas-NOAA

[WalterKolczynski-NOAA]

This issue isn't resolved until the GDASapp hash has been updated in workflow

[JessicaMeixner-NOAA]

This should be able to be re-enabled after #2514 (updating GSI hash) and #2420 (updating gdas app hash) are merged.

[JessicaMeixner-NOAA]

The GSI hash has been updated. We are now waiting on #2420 to update the GDAS app hash. Hopefully this will be done soon so we can get this test turned back on.

[JessicaMeixner-NOAA]

#2420 is now merged, however there is a new issue #2528 preventing this test from being turned back on.

[JessicaMeixner-NOAA]

The issue #2528 has been resolved enough to get past the first error. Now we have a new error that:
FileNotFoundError: [Errno 2] No such file or directory: '/scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/iau_02/C48mx500_3DVarAOWCDA_t03/COMROOT/c48t03/gdas.20210324/12//model_data/ice/restart/20210324.180000.cice_model.res.nc'
in gdasocnanalprep

[JessicaMeixner-NOAA]

Currently in COM for ice restarts I have:

$ls /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/iau_02/C48mx500_3DVarAOWCDA_t03/COMROOT/c48t03/gdas.20210324/12/model_data/ice/restart/
20210324.150000.cice_model.res.nc

whereas before we had:

$ ls /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/turnonmarine/c48ci01/COMROOT/c48t01/gdas.20210324/12/model_data/ice/restart
20210324.150000.cice_model.res.nc  20210324.180000.cice_model.res.nc  20210324.210000.cice_model.res.nc

I believe this is a change from #2510 @aerorahul should additional ice restart files needed for DA be copied to COM or should the DA jobs be looking for this data in another directory?

[guillaumevernieres]

If this was tested without the "IAU" flag on, then there's a bug in the g-w. The restarts should be for 18z, not 15z. Also, it looks like the atmos restarts are available at 15-18 and 21, but this isn't the case for mom6 and cice6.

It's a bug in the marine DA @JessicaMeixner-NOAA . We're inserting the analysis in the wrong restart when DO_IAU is switched on ... Oops.