noaa-emc / cmeps Goto Github PK

We need to update to the current ESCOMP/master of CMEPS. This is not expected to change any baselines.

1. The Xgrid implementation for calculating A-O fluxes only has been added for CESM at ESCOMP/master (ESCOMP PR #216). This capability should be brought to NOAA-EMC to allow development of the Xgrid for ufs-weather.

2. Accumulation field bundles have been refactored (ESCOMP PR #226). The import and export accumulation FBs are no longer arrays (carried for multiple components). The import accumulation FBs (not used in UFS) are now module specific (runoff and glacial ice). The ocean export accumulation FBs (used for the ocean export in UFS) is now carried in a specific FBExpAccumOcn

3. Clean up of med.F90. When setting up the mediator mesh and related import/export fields from components on both grids and meshes, Med.F90 utilizes an IPDv03p4 phase to modify the the decomp/distribution of the transferred Grid/Mesh. In the refactored code, the process by which an incoming grid or mesh is modified (redistributed) to the mediator PEs is much more clear: a new grid/mesh is created on the mediator PEs and upon leaving the phase, the coordinates and geom base of the old grid or mesh is automatically transferred to the new grid/mesh.

   The transfer of fields from the incoming grid/mesh is completed in the subsequent IPDv03p5 phase. For a transferred grid, the IPDv03p5 creates the mesh and then transfers all fields in the State to the mesh in a NUOPC_realize call. This call acts to kick out the old Grid-based field from the state, and replace it with a Mesh-based one.

   It should be noted that the cleanup of med.F90 resolves an issue where the call to ESMF_FieldEmptyComplete resulted in a non-fatal error. Currently, an error message in the PET logs indicates an attempt to complete a field(s) that is already destroyed:

ESMF_Field.F90:314 ESMF_FieldValidate Invalid object  - Uninitialized or already destroyed Field: fieldstatus not ready

This error did not cause the code to abort because the complete call was also missing a subsequent ChkErr call. The error was produced because for a transferred Grid, the call to NUOPC_realize had already replaced the grid fields with
the mesh fields. Thus the incoming grid field had already been destroyed (replaced by the mesh field).

4. Mediator history refactor. Currently to obtain a coupler history file (e.g. debugging the coupled model), a special med_phase_history_write needs to be added within the run sequence. The coupler history file is also not produced at the right frequency when there are different slow and fast coupling frequency in the run sequence. The refactor of the coupler history writing will allow each component to produce a coupler history file separately; the frequency of the coupler history files will be a configurable option at run time, and can be set differently for each coupled component where a history file is requested.

This update requires ESMF8.2.0bs14 or higher. Additional settings will also need to be added to the nems.configure.

This update will also change baselines because the mediator restart file will not contain the same fields due to the changes noted in item 2 above. Testing has shown that all common fields between the two mediator restarts and all other component model restart files are b4b with existing baselines.

Enable fractional land capability in CMEPS.

During ufs-weather PR 561 (the global RT update), it was found that the cpld_control and cpld_controlfrac tests compiled with GNU did not reproduce. The variables ImpAccumCnt and ExpAccumCnt in the mediator restart file were different between the baseline and the verify run. They also contained valid values only for the ocean component. All other component values were un-realistic (and different between baseline and verify).

The issue is that the AccumCnt is initialized to zero only for active components, but is dimensioned to all components. A fix is required to initialize the AccumCnt for all components. The fix has been tested in ufs-weather compiled with gnu and the mediator restart files are now reproducible for both the cpld_controlfrac and cpld_debugfrac tests.

When CMEPS is used in DATM-MOM6-CICE5, air-sea flux calculations cannot be done in the mediator.

CMEPS version used: f52f263.

Related issue: NOAA-EMC/DATM-MOM6-CICE5#48

In the HAFS-MOM6 system, the regional FV3ATM atmosphere domain and MOM6 ocean domain do not overlap completely. The FV3ATM atmosphere domain and WW3 wave domain do not overlap completely either.

The CMEPS will be updated to add the following features:

1. Capable of coupling both FV3ATM and CDEPS datm to MOM6 at the same time. In the overlapped regions, the MOM6 model receives flux fields from FV3ATM. In the non-overlapped regions, the MOM6 model receives flux fields from GFS forcing through CDEPS datm.
2. Capable of coupling both FV3ATM and CDEPS datm to WW3 at the same time. In the overlapped regions, the WW3 model receives 10m wind fields from FV3ATM. In the non-overlapped regions, the WW3 model receives the 10m wind fields from GFS forcing through CDEPS datm.

Changes made in CMEPS by EMC have been merged at ESCOMP. These changes need to be brought back to emc/develop. Initial testing has shown these changes will change the mediator restart file because the number of components is now 9 due to the addition of another glacier component (in CESM, this allows for both a northern and southern glacier component) but all other fields in the mediator restart file should be B4B.

This update will also remove the ESMF version >8 ifdefs.

uh, does this make sense to anyone?
When ESMF releases 9.0.0 , this is going to blow!

The changes required by HAFS have been merged and escomp/master; we need to bring these changes to noaa-emc for hafs development in the ufs-weather-model.

The esmf_logging option in nuopc.runconfig is not working since the esmf logging is controlled by drv_in using create_esmf_pet_files namelist option (in CMEPS driver code). So, I am not sure but maybe there is no need to expose ESMF_LOGFILE_KIND in the cmeps/cime_config/namelist_definition_drv.xml file. I think CMEPS driver is not looking to it instead of it is looking to the drv_in and this information only meaningful to the driver.

Currently the SWnet calculated for the ocean assumes a constant albedo of 0.06 for both direct and diffuse SW radiation. This albedo does not vary with sun angle. This results in an excess of SW radiation to the ocean. See Issue ufs-community/ufs-weather-model#1768 for more details.

The ocean albedo calculation can be implemented by adding the needed run phase and having UFS call the ocean albedo calculation in med_phases_prep_ocn. Currently this calculation happens only for CESM, but can be generalized for non-cesm use without duplication of code.

PRs merged at ESCOMP are needed to update NOAA-EMC

• updates multiple cime and cesm related files
• adds optional check for NaN in export FBs. This requires additional share modules, which are available in the CDEPS repo. Changes are made in the CMakeList.txt in the CMEPS-interface to access these modules. The check for NaNs in off by default. It can be enabled by adding a configuration variable in the MED attributes check_for_nans = true

CMEPS requires an updates to allow connection of the 4 SW albedo components from CICE6 to the ATM if ATM advertises for these fields:

The original coupling modes were defined at the time of the update from the NEMS mediator. The values of coupling_mode are poorly named for the current uses in UFS. They should be changed to be more representative of the what each mode represents.

nems_orig : this is the original, non-fractional grid with an active ATM.
nems_orig_data : the original, non-fraction grid with DATM
nems_frac : the fractional grid with active ATM
nems_frac_aoflux and nems_frac_aoflux_sbs : the fractional grid using the mediator aoflux calculations.

Better names for these modes are:

nems_orig --> ufs.nfrac
nems_orig_data --> ufs.nfrac.aoflux
nems_frac --> ufs.frac
nems_frac_aoflux and nems_frac_aoflux_sbs --> ufs.frac.aoflux

CMEPS uses packed fields at mapping time, meaning all fields from component n1 to n2 using a given mapping maptype are packed into the unlimited dimension of a single field. As a consequence, all packed fields for n1:n2:maptype must have the same normalization applied. This restriction is noted in code comments but not otherwise enforced in the CMEPS code, so that a user could set a 2nd normalization type without triggering an error.

A trigger needs to be implemented which will cause the application to abort if a second normalization type is specified for a given n1:n2:maptype specification.

The land-sea masks used in CDEPS for cfsr and gefs data sources are different from those used in NEMS_datm. The file "mediator/med_map_mod.F90" needs to be revised.

Currently both the coupler and history files are written using the configuration variable case_name. For consistency w/ ufs-weather and the repo-unification, the coupler history files should be written to the RESTART directory. That can be done using case_name=./RESTART/ufs.s2s. However, this forces the coupler history files to also be written to the RESTART directory instead of the run directory.

We need to add a new optional config variable to specify the restart directory. The history files would then be written by default to the run directory.

Testing in CESM revealed that the mapping and merging of fields in CMEPS was inefficient. Significant performance improvement can be achieved by packing fields into an undistributed dimension of a single field, performing the required mapping and merging on the single field and then unpacking the fields for export to the relevant component. If a field has an undistributed dimension, the packed field accordions out to accommodate the field's undistributed dimension.

Additional performance enhancements can also be achieved through ESMF itself (currently in development). These ESMF related changes are not required to implement the field packing.

The field packing performance improvement is expected to be b4b with previous answers.

A couple of RRFS ensemble members crashed with GFS PBL (TKE-EDMF)/sfclay and clm lake combined. This happened when lake water was frozen to lake ice in clm lake.

The reason for the crash is because clm lake is called between surface layer scheme and PBL scheme. If there is a state change in clm lake, particularly water turning into ice, stability related variables over ice will be missing and cause the crash.

For resolution of issues required for moving the mesh cap from the dev/ufs-weather-model branch of WW3 to the develop branch, testing of the ww3 mesh cap with ice-wave coupling enabled is required. This testing will be carried out in collaboration w/ NCAR.

For wave-ice coupling, the wave model needs to export the wave elevation spectrum and the ice model needs to export both the sea ice thickness and floe diameter. These fields can be added to esmFieldsExchange_nems even when ice-wave coupling is not active.

The CMEPS run failed in s2s when compile with esmf8.1.0bs20. The issue is fixed by updating CMEPS/mediator/makefile.

The update to the lastest ESCOMP master will bring in component level PIO initialization to the HAFS application (or UFS Weather Model) without using shr_pio_mod.F90. The shr_pio_mod.F90 is also removed from the util/ directory.

Included in the update to escomp/master is a bugfix for cart3d in med_map_mod.F90. This mapping method is not currently implemented in ufs-weather.

When attempting to read a mediator restart file, the mediator fails if it cannot find a field in the restart file which matches the first field in the component field bundle. This can occur when a user attempts to use a mediator restart file (start_type=continue) after adding a component when the component was not active when the restart file was written. In fact, the correct setting in this case is to use start_type=startup. However, an error message should be reported by the mediator in order to help the user diagnose what they are doing wrong in this case.

The current implementation of med_fraction_set for nems is setting the ifrac using a two-step procedure where first the ifrac is mapped with nearest neighbor (ifrac_nstod) and then conservative (ifrac). A conditional is used to determine the final ifrac value.

This procedure results in the isolated coastal anomalies seen in the v0.6 milestone runs. See ufs-s2s-model issue 116 for complete details.

The use of ifrac_nstod is unnecessary; the ifrac can be mapped directly with mapnstod_consf.

The following is a section of tile3 from phyf000:07:30.tile3.nc, showing the locations of two of the coastal anomalies (stars). On the left is the sea ice concentration, on the right the land-sea-ice mask. Locations of suspicious coast points are marked with stars.

This is the same comparison when ifrac_nstod is removed and the ifrac is mapped directly with mapnstod_consf in med_fraction_set:

CMEPS has the capability to define generic merges between components. However, these generic merges are not available when the sign of the field needs to change, or the field needs to be converted to one of different units (ie, latent->evap).

Because of this issue, multiple "custom" merges are required in med_phases_prep_ocn. For example, the sensible heat flux from the ATM has the wrong sign, requiring a custom merge:

       customwgt(:) = -ofrac(:)
       call med_merge_field(is_local%wrap%FBExp(compocn),      'Faxa_sen',  &
            FBinA=is_local%wrap%FBImp(compatm,compocn), fnameA='Faxa_sen', wgtA=customwgt, rc=rc)
       if (ChkErr(rc,__LINE__,u_FILE_u)) return

The ATM exports the latent heat flux, but the ocean requires the evaporative flux, which requires a custom merge to change the sign and units of the field:

       customwgt(:) = -ofrac(:) / const_lhvap
       call med_merge_field(is_local%wrap%FBExp(compocn),      'Faxa_evap', &
            FBinA=is_local%wrap%FBImp(compatm,compocn), fnameA='Faxa_lat' , wgtA=customwgt, rc=rc)
       if (ChkErr(rc,__LINE__,u_FILE_u)) return

The momemtum fluxes from the ATM also require a custom merge to change the sign.

This adds needless complexity to maintain these custom merges when a better solution is to have the ATM export the coupling field with the right sign or unit. This also makes it possible to leverage the existing diagnostic capabilities with CMEPS to examine global budgets for UFS applications.

Changes are required in esmFldsExchange_nems and med_map_mod to implement two changes: unity normalization and bilinear mapping of ATM state fields. These are changes from the current implementation in which the criteria used was validating the NEMS mediator functionality.

1. CMEPS has the ability to apply a "unity" normalization when mapping from source to destination grid. This ensures that the mapping weights on the destination side add to one. The normalization is a static field which needs to be calculated only once (in med_map_mod) by mapping a field value 1.0 from each source component to each destination component for each defined mapping type between those two components. The reciprocal of the mapped value 1.0 is added to the mediator internal state and used when specifying that the normalization type for mapping is "one"; e.g.

call addmap(fldListFr(source_component)%flds, 'field_name', destination_component, mapconsf, "one", 'unset')

Note that an equivalent process happens when normalizing using a time-varying ice fraction. In that case, the the normalization in the above call would be set to ifrac. When normalizing by ifrac, the mapped destination field (= mapped source_field*ifrac) is normalized by the factor 1.0/(mapped ifrac).

The ability to apply unity normalization when not normalizing by ifrac was not available in NEMS and so was set to none in the mapping calls used by UFS. Since it is a static field that is calculated only once, there is no penalty to applying the unity normalization and it ensures that the mapping weights always add to one regardless of the mapping type.

2. For CMEPS in UFS, ATM states (T,P,Q,U,V) are being mapped conservatively when mapped to the ICE (when coupled) or to the mediator (when using the mediator aoflux calculation). (The DATM application also maps the 2m and 10m state fields conservatively for use in the bulk flux option).

For the typical case in UFS, this means mapping from a coarse ATM to a fine OCN (relatively). This creates very un-smooth destination fields. For example, the following shows the mapped Tbot (from ATM) exported to the ICE in the C96-mx100 case in the Ross Sea (Antarctic) region. The left shows the conservatively mapped field and the right shows the bilinearly mapped field.

The following shows the value at the x=130 line:

Since ATM T,P,Q,U,V are global, unmasked states and not fluxes, the mapping should be switched to bilinear in order to provide smooth fields to the ICE model (coupled) and for the aoflux calculation (DATM). Changes are required to both esmFieldsExchange_nems (where the mapping is defined) as well as to the srcMaskValue used for a bilinear routehandle creation from ATM->OCN,ICE.

For the datm applications (either cdeps or nemsdatm), the ATM is much finer than the OCN. This results in smooth states being passed to the ICE and aofluxes even though they are being mapped conservatively.

Currently the custom merges in med_phases_prep_ocn_mod.F90 are separate for the modes nems_orig/frac and nems_orig_data. With the change to the bulk formulas (as implemented in the NEMS mediator PR #69), the comparable code in CMEPS can be simplified.

This will result in a single med_phases_prep_ocn_mod.F90 which is applicable for all nems modes (orig,frac and orig_data).

Further testing in CESM revealed that one test case did not pass the restart test due to changes in med_io_mod.F90. Reverting to the previous version passed all CESM tests so the performance changes in this module have been reverted until further testing determines the issue.

The reverted code has been tested in the ufs coupled model. All RTs run to completion and are b4b with previous answers.

The ccpp-physics component is updated to include a new option to consider sea surface current in the computation of air-sea fluxes. The ccpp-physics code used in CMEPS will need to be updated to be consistent with those changes in ccpp-physics.

When the component lat/lon values are written by the mediator to either history files or the restart files, the units are currently set as degrees by default. This is inconsistent with the units sent by fv3atm, which are in radians.

For user clarity, the coordSys used by the component should be used to set the units.

PR #16 broke the compilation of ufs-s2s-model on platform Cheyenne. The ability to compile UFS-S2S-model on cheyenne is critical for continued collaboration with NCAR on CMEPS and UFS-S2S development.

Turning on mediator history files when the field contains an ungridded dimension causes a seg fault. Mediator history files are used only for diagnostic/debugging. For UFS, the only application affected would be when wave-ice coupling is enabled.

For use in the NEMS DATM application, the 10m U,V winds, the 2m temperature and the 2m specific humidity need to be added to the fields passed to the aoflux calculation.

The aoflux calculation itself then needs to be modified to implement Hyun-Chul's preferred bulk formulation.

The added character strings for the variables for the bulk method variables in esmFldsExchange_nems_mod.F90 need to be padded to the same length. This is required for using the gnu compilers.

A fix is required to esmFlds.F90 to allow compiling in ufs-weather with GNU.

When mediator history files are enabled for CMEPS, the ATM component is written on the ATM mesh. This limits the usefulness of the history files because an additional processing step is required to place the fields on the ATM tiled grid for visualization.

Allowing the ATM history files to be optionally written on the cubed sphere tiles would allow developers to make quick checks of coupling fields during development. It could also be useful when implementing a set of graphical outputs that could be generated along with the RTs, as suggested in

• ufs-community/ufs-weather-model#914

.

The hafs coupling mode does not use aoflux calculation but the current code sets up merges and field bundles as though it is used. This adds unneeded and confusing code. The aoflux options under 'hafs' should be removed.

• related to #93

There are currently conditional statements in CMEPS checking the ESMF version:

#if ESMF_VERSION_MAJOR >= 8
#if ESMF_VERSION_MINOR >  0

These should be removed if they are no longer required. @mvertens and @uturuncoglu Is this safe for CESM and HAFS?

Currently a large section of code in med_map_mod.F90 is used to define the src and dst masking for various component couplings.

The field Si_ifrac being exported by CMEPS to the ATM is currently being mapped with maptype mapnstod_consf (non-frac) and mapconsf (frac_mode). This is inconsistent with the value expected by FV3 (which assumes that the ifrac imported is weighted by the whole atm grid cell). The fix is to map by mapnstod_consd for nems_orig and mapconsd for nems_frac in med_fraction_mod.F90. This fix will change all current baselines.

This fix will be implemented as through CMEPS PR #10. This PR implements other changes which are required for CESM budget calculations but do not affect the NEMS implementation.

When FV3 exports the open ocean fraction it used in actually calculating the fluxes being exported, CMEPS needs to use this value to normalize those fluxes when mapping to other components.

Related FV3 issue #230

The CMEPS mediator needs to be updated to bring support for external land component under UFS weather model.

Updates the partitioned stokes drift beween MOM6 and WW3 to use ungridded dimensions.

Includes an update made at ESCOMP/master to make the med_fldList fields a linked list instead of an array. This avoids a lot of small allocations and memory fragmentation (ESCOMP#322). Changes in UFS to modify wave coupling and add optional wave-ice coupling have also been merged at ESCOMP. An update to NOAA-EMC should be made to reconcile these changes.

NOTE: The changes related to CMEPS PR#322 were tested independently at the time of commit to ESCOMP. All baselines were B4B.

The land-sea masks used in CDEPS for cfsr and gefs data sources are different from those used in NEMS_datm. The CFSR/GEFS mesh files will be created with modified land-sea masks. The med_map_mod.F90 needs to be revised.

A dependency listed in ufs/ccpp/data/MED_typedefs.meta, physparam.f, has (PR pending) been removed. It doesn't appear that physparam is used in MED_typedefs.F90, but with physparam.f in the meta file the ccpp_prebuild step is now failing for the cpld_control_noaero_p8_agrid

The last update to CMEPS from ESCOMP was in June. We should update to the latest master commit.

The CMEPS routine which defines the field exchanges for NEMS does not have the same structure as the routines used for HAFS and CESM. Specifically, in those cases the routine is called separately for the advertise and initialize phases. The NEMS routine should be refactored to be consistent with how CESM or HAFS coupling is implemented.

In UFS, fields exchanges are defined primarily by coupling_mode instead of by field. This is less extensible than to consider each field which goes to a component and allowing the coupling mode to be intrinsically set by defining source component.

For example, a given field to compocn comes from the mediator in the case of using the atm-ocean flux calculation (either on a-grid, o-grid or x-grid) or from an ATM (either active or data).

This will make UFS coupling easier to extend and maintain, especially as we move to extend both X-grid and diagnostic capabilities within UFS.

This work is related to #93, since it will depend on minimizing the custom merges that need to be done.

When coupling ATM and WAV, the fields exchanged between the components are state fields, not fluxes. They should be mapped bilinearly to better preserve the gradients in the fields.

The current version of DATM-MOM6-CICE6 is not restart reproducible. The CMEPS code will be updated in a branch to add additional diagnostic outputs.

Related issue:
ufs-community/ufs-weather-model#281