escomp / mizuroute

Reach-based river routing model

mizuRoute

Stand-alone mizuRoute is a post-processor of runoff output from a hydrologic model or Land surface model to produce streamflow estimates in the river network provided by a user. The tool was developed for the large scale, network based river routing (e.g., river network over contiguous United States), but works for gridded river network as well.

Technical documentation is now being built on readthedocs

To get started

1. Obtaining mizuRoute package. Just to use the tool, download the package by clicking “Download Zip” button on right column.

2. Fortran compiler. Since we do not use any compiler-specific extensions, mizuRoute should be complied with any Fortran compilers. We have successfully used the intel Fortran compiler (ifort), the GNU Fortran compiler (gfortran), and PGI fortran compiler (pgf90). If the user does not have a Fortran compiler, gfortran can be installed for free. The easiest way is to use a package manager. Which package manager depends on your machine flavor. We tested with the following compilers:

   • gfortran 8.3.0
   • ifort 18.0.5
   • pgi 19.3

3. NetCDF libraries. NetCDF or the Network Common Data Format, is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. All the mizuRoute I/O (except control file and parameter namelist) use NetCDF. The user needs to ensure that: NetCDF version 4.x is installed in the linux-like machine. NetCDF Fortran library is installed (libnetcdff.*) and not just the C-version. The NetCDF libraries are compiled with the same compiler as you plan to compile mizuRoute The user may use netcdf test code to check if NetCDF libraries are properly installed.

4. Compiling the source code. Once you have all the above, you can compile mizuRoute source codes using the following steps: Navigate to your local copy of the mizuRoute directory and go to the build subdirectory.

   1. Edit F_MASTER (name of path befor build directory) and FC (compiler name: gnu, intel or pgi) and FC_EXE (compiler executable name). You may also need to set NCDF_PATH. You may need to add some extra entries if you are using a different Fortran compiler or your setup is different (if someone wants to contribute an actual configure script that would be great). openMP (shared memory parallel processing) directive is implemented to prallelize the routing process. To activate openMP, set isOpenMP= yes.

   2. Type make under directory where Makefile is located. If all goes well, this will create the executable runoff_route.exe to the bin directory. You may get some warnings (depending on your compiler settings), but you should not get any errors.

   3. Pay attention to the make output. You may need to set some environment variables (LD_LIBRARY_PATH in particular) to support dynamic linking;

   4. Try running the executables:

        ./route_runoff.exe
       	FATAL ERROR: need to supply name of the control file as a command-line argument
      

If you get this far then mizuRoute is built correctly and functional. Now, the user will have to generate input data, runoff data, river network topology and runoff mapping data (depending on input runoff option). Please look at readthedocs to learn more about mizuRoute input data.

The user are encouraged to start with example data to get familiarize the process. testCase are being now developed and posted separately.

For real application, getting river network data and mapping data in netCDF format correctly take work because this most likely requires GIS process and convert shapefile to netCDF).

mizuroute's Issues

Mismatch between hru and hruId

When the dimension name is hru with serial indices and hruId is a variable with HRU indices, routing stops with this error:

 Route all the segments included in network topology
 maximum upstream area =    660573269331.323             4506
 FATAL ERROR: get_qDims/NetCDF: Invalid dimension ID or name; name=hruId

Remove unnecessary variables

Remove some of input variables not used in the model.

hru
integer(i4b) :: xLon = 1 ! longitude
integer(i4b) :: yLat = 2 ! latitude
integer(i4b) :: elev = 3 ! elevation
seg
integer(i4b) :: bLon = 1 ! start longitude of segment
integer(i4b) :: bLat = 2 ! start latitude of segment
integer(i4b) :: eLon = 3 ! end longitude of segment
integer(i4b) :: eLat = 4 ! end latitude of segment

Cleanup 2: Calc_hru_wgtavg_nc.py: Python style (PEP8)

This is a follow up to #14. There are still a number of Python style / PEP8 improvements that remain. Mostly I'd like to see the variable naming convention standardized according to PEP8.

Need to receive Precip and ET over lakes from coupler

mizuRoute needs to receive Precip and ET from the CESM coupler over lakes

Change MODEL to COMP_NAME

In a recent version of cime, components need to change the use of MODEL in buildlib to COMP_NAME. I'm finding this a problem with building and running on izumi.

Get the standalone build working with mpi-serial which could allow the cesm-coupling branch to unite with the main branch

@nmizukami worked on this afternoon. In the latest version the standalone build doesn't work with mpi-serial. The problem we saw was that PIO needs to be linked to mpi-serial and use it (rather than mpi).

Handle irrigation from CTSM in mizuRoute

We need to handle irrigation from CTSM in mizuRoute.

Known limitation, need to use the same number of MPI tasks on restart

@nmizukami let me know that the same number of MPI tasks need to be used on restart as were used to create the restart file. Changing this also would take some effort. Since, this is not something normally needed we are going to leave this a known limitation.

For CESM testing this means that we expect ERP tests to fail. ERT tests should work however as they only change the number of threads.

Handle qgwl like MOSART does

Currently we are ignoring the QGWL Glacier/Wetland/Lake residual term from the coupler. This should be handled similar to how it's handled in MOSART. This likely means there should be several different options on how to handle it.

Integer array write fails in latest version of cime/PIO

We've updated the version of cime being used for mizuRoute for CTSM coupling and found that it has problems when it closes the history files in mizuRoute. Other components work fine.

This update goes from: cime5.8.26 to branch_tags/cime5.8.42
I can't easily see what PIO tags this corresponds to, but the hashes in cime seem to be as follows...
git diff 875191844ae6e390592872f60959bfdb86c32f56...44df5e702410e6f17efbbade29ea4e677e666b3c

After working through the problem we find that it's writing an integer array from master-proc for our largest grid. It works for smaller grids, but fails for the MERIT-Hydro grid.

In looking at the updates that happened in other components, we think similar changes need to come into mizuRoute.

I see the following change in MOSART that may need to go into mizuRoute for updating to the latest CTSM.

-    integer :: start(2), count(2)      ! output bounds
     integer :: status                  ! error code
     logical :: varpresent              ! if true, variable is on tape
     character(len=32) :: vname         ! variable error checking
@@ -1280,19 +1279,7 @@ subroutine ncd_io_char_var1_nf(varname, data, flag, ncid, readvar, nt )
        call ncd_inqvid  (ncid, varname, varid, vardesc)
 
        if (present(nt))      then
-          count(1) = len_trim(data)
-          count(2) = 1
-          do m = 1,count(1)
-             tmpString(m:m) = data(m:m)
-          end do
-          if ( count(1) > size(tmpString) )then
-             write(iulog,*) subname//' ERROR: input string size is too large:'//trim(data)
-             call shr_sys_abort( )
-          end if
-          start(1) = 1
-          start(2) = nt
-          status = pio_put_var(ncid, varid, start=start, count=count, &
-               ival=tmpString(1:count(1)) )
+          status = pio_put_var(ncid, varid, (/1,nt/), ival=data)

dummy issue

dummy issue

On normal continue run add to last history file (if not full) rather than opening a new one

We need an option for a normal continue run so that when history files are not full, that the restart file will finish filling the file, before opening a new one. The current mechanism of opening a new file on restart is good for the CESM "branch" option, so we likely want both options.

In order to do this on restart the history file name name needs to be written, so it can open that same file to add the last time samples to it. Since, files can either be monthly , or yearly (or daily, but then each time a new file will be opened) there is no need to write the number of time samples on the file, it'll just open a new one at the beginning of the new month or new year.

Add an option to build with ESMF to provide the mpiuni library for MPI

We'd like to try to build with the ESMF library as a way to provide a MPI interface in place of the mpi-serial MCT library. We'll add it as an option you can use in place of the mpi-serial option.

timing information for v1.2

The time step level subprocess time reporting be made an option for a --v (verbose) flag, and add a summary of timing at the end appearing by default.

SourceMods functionality issue

As written buildlib for mizuRoute does not find a file added to SourceMods/src.mizuroute unless

1. The code has not been built prior to adding the file or
2. The file bld/rof/obj/Filepath is removed.

Connect continue_run in control file in buildnml

continue_run is now a logical variable in the control file. This needs to be connected to the CESM XML variable CONTINUE_RUN in the buildnml. We should also check to make sure the user doesn't try to override the setting from the CESM XML variable.

Send back amount of water removed from lakes to CESM

As a longer term goal, mizuRoute will need to send back to CESM the amount of water taken out of lakes. Normally this will just be Evaporation-Precip that was sent into it, but when lakes are low it could be less than that or even zero. This is needed to get the model water balance to be correct.

This will likely be similar to how irrigation is handled in CTSM/MOSART, with the exception that it likely won't occur nearly as often.

runoff remapping option

Issue: Now remapping netcdf is alway read in and is processed in getAncillary.f90 even if runoff need not be remapped.

Potential resolution: Make this data process optional since there is variable is_remap which is input by a user (from control file) to tell whether or not runoff provided is remapped.

Exact restart tests failing because the list of time constant fields is different

Exact restart tests are failing because the list of time constant fields is different. For example the test ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default fails as follows...

PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default CREATE_NEWCASE
PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default XML
PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default SETUP
PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default SHAREDLIB_BUILD time=10
PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default MODEL_BUILD time=42
PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default SUBMIT
PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default RUN time=269
FAIL ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default COMPARE_base_rest
PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default GENERATE ctsm5.1.dev069.mizu
PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default MEMLEAK insuffiencient data for memleak test
PASS ERS.5x5_amazon_rHDMA.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default SHORT_TERM_ARCHIVER

with the TestStatus.log file showing the following issue...

SUMMARY of cprnc:
 A total number of    493 fields were compared
          of which      0 had non-zero differences
               and      0 had differences in fill patterns
               and      0 had different dimension sizes
 A total number of      4 fields could not be analyzed
 A total number of      0 time-varying fields on file 1 were not found on file 2.
 A total number of      0 time-constant fields on file 1 were not found on file 2.
 A total number of      0 time-varying fields on file 2 were not found on file 1.
 A total number of     10 time-constant fields on file 2 were not found on file 1.
  diff_test: the two files seem to be IDENTICAL
      (But note that there were differences in field lists just for time-constant fields.)

ERP threaded test is failing I think because of a glitch with mizuRoute history files

The following test is failing in the compare base to restart test:

ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default

PASS ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default CREATE_NEWCASE
PASS ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default XML
PASS ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default SETUP
PASS ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default SHAREDLIB_BUILD time=303
PASS ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default MODEL_BUILD time=120
PASS ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default SUBMIT
PASS ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default RUN time=264
FAIL ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default COMPARE_base_rest
PASS ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default MEMLEAK insuffiencient data for memleak test
PASS ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default SHORT_TERM_ARCHIVER

I think the problem is that it can't find history base and restart files for mizuRoute. See the following error in the TestStatus.log file:

2022-01-15 21:17:55: Comparing hists for case 'ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6' dir1='/glade/scratch/erik/ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6/run', suffix1='base',  dir2='/glade/scratch/erik/ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6/run' suffix2='rest'
  comparing model 'datm'
    no hist files found for model datm
  comparing model 'clm'
    ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6.clm2.h0.2000-01-12-00000.nc.base matched ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6.clm2.h0.2000-01-12-00000.nc.rest
  comparing model 'sice'
    no hist files found for model sice
  comparing model 'socn'
    no hist files found for model socn
  comparing model 'mizuroute'
    File 'ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6.mizuroute.h.2000-01-01-00000.nc.base' had no compare counterpart in '/glade/scratch/erik/ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6/run' with suffix 'rest'
    File 'ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6.mizuroute.h.2000-01-07-00000.nc.rest' had no original counterpart in '/glade/scratch/erik/ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6/run' with suffix 'base'
  comparing model 'sglc'
    no hist files found for model sglc
  comparing model 'swav'
    no hist files found for model swav
  comparing model 'sesp'
    no hist files found for model sesp
  comparing model 'cpl'
    ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6.cpl.hi.2000-01-12-00000.nc.base matched ERP_P1x25_D.5x5_amazon_r05.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default.20220115_191342_49rbw6.cpl.hi.2000-01-12-00000.nc.rest
FAIL

The clm and cpl files that it compared were identical, so I think the problem is just that it couldn't find suitable files to compare for mizuRoute.

Standalone build should send DEBUG option to both the PIO and mpi-serial libraries

Right now the DEBUG level for the PIO and mpi-serial libraries for the standalone build don't tell them about the DEBUG level. As such I think they assume it's on. We want to give the user control over this so the applicable flags should be set so that when $(MODE)=="debug" it flags DEBUG mode and otherwise doesn't.

This will both be important for debugging potentially, as well as important for performance (if DEBUG==TRUE is truly the default like I think it might be).

add fill value in output

issue: There is no fill value attribute in output netcdf

resolution:
Define in write_simout.f90
https://www.unidata.ucar.edu/software/netcdf/netcdf-4/newdocs/netcdf-f90/NF90_005fDEF_005fVAR_005fFILL.html#NF90_005fDEF_005fVAR_005fFILL

caution:

1. Be careful that missing values are propagated through routing process (runof read, remapping, delayed runoff, and then channel routing processes).

2. If there is missing runoff in hru at one time, possibly hru does not have any hydrologic model hru overlapped, meaning there is no valid runoff through the simulation period.

3. How would you handle for effect of missing hrus on downstream segment routed runoff.

new feature - tracer

Goal:
to enable to trace DOC/DON (or other constituents) from land models

Urgency:
months to a year

Background:
Use advection-dispersion equation
dC/dt = -UdC/dx + Dd2C/dx2
where C = concentration [M/L3], U = velocity [L/T], D=dispersion coefficient [L2/T]
1st term in LHS = advection
2nd term in LHS = dispersion

Tasks:

• Receive/process DOC/DNC variables from NUOPC coupler

• add subroutine to solve mass balance equation per reach
  start with advection only equation.

Relevant issues:
#153
ESCOMP/CTSM#1216

Remove <varname_pfafCode> in the control file for parallel mizuRoute

Parallel run of mizuRoute does not necessarily need <varname_pfafCode>, while the code still requires that <varname_pfafCode> is included in the control file and the corresponding variable is included in the river network netcdf file. I need to make a fake pfafCode variable in the netcdf file to run parallel mizuRoute. A better solution is to remove <varname_pfafCode> or to loose the variable check.

Cleanup 1: Calc_hru_wgtavg_nc.py: Stop using a class

Follow up to #14. The use of a Class object in Calc_hru_wgtavg_nc.py is an unnecessary abstraction.

Add internal checks & corrections for topology

I ran into an issue where some river segments in my domain shapefile have length 0 and this leads to a mizuRoute error:

 FATAL ERROR: init_data/init_ntopo/check_river_properties/reach length for reach id 74030207 is negative:   0.000000

I briefly discussed this with @ShervanGharari and he suggested to set the reach length in such cases to 1 m, so that mizuRoute can continue. I will be implementing this as a pre-processing step for my own work, but we thought it may be helpful to suggest this as a possible enhancement for future mizuRoute versions.

This could take the form of a warning + correction, so that the user is aware that their input topology has been (slightly) changed but the run can still complete.

history and restart file name control for stand-alone run

history and restart file use CESM name convention, e.g., history file is <case_name>.mizuroute.h.yyyy-mm-dd-sssss.nc.

For stand-alone use, use may want a shorter name. so at least add capability for user to control "mizuroute" part (remove or use mizuRoute)

<param_nml> IRF parameters resulting in no output

In trying to calibrate flows at the TDA and KOOTE (1950s-1980s), increasing velo above 6.0 and diff below 400 will result in no outputs for TDA despite outputs for KOOTE.

spatially varying hru uh parameters

Implement some transfer function to compute gamma distribution parameter time_scale parameters, which will be consistent with TF in MPR

Add in area correction to mizuRoute

With the latest version of CTSM we have area correction added into CTSM, MOSART and RTM. This is multiplying fields by the ratio of what the model things the area is divided by the area as defined by the mesh file.

Here's an example section added to the end of realize fields for RTM:

+    ! Determine areas for regridding
+    call ESMF_MeshGet(Emesh, numOwnedElements=numOwnedElements, rc=rc)
+    if (chkerr(rc,__LINE__,u_FILE_u)) return
+    call ESMF_StateGet(exportState, itemName=trim(fldsFrRof(2)%stdname), field=lfield, rc=rc)
+    if (ChkErr(rc,__LINE__,u_FILE_u)) return
+    call ESMF_FieldRegridGetArea(lfield, rc=rc)
+    if (chkerr(rc,__LINE__,u_FILE_u)) return
+    call ESMF_FieldGet(lfield, farrayPtr=dataptr, rc=rc)
+    if (chkerr(rc,__LINE__,u_FILE_u)) return
+    allocate(mesh_areas(numOwnedElements))
+    mesh_areas(:) = dataptr(:)
+
+    ! Determine flux correction factors (module variables)
+    allocate(model_areas(numOwnedElements))
+    allocate (mod2med_areacor(numOwnedElements))
+    allocate (med2mod_areacor(numOwnedElements))
+    n = 0
+    do g = runoff%begr,runoff%endr
+       n = n + 1
+       model_areas(n) = runoff%area(g)*1.0e-6_r8/(re*re)
+       mod2med_areacor(n) = model_areas(n) / mesh_areas(n)
+       med2mod_areacor(n) = mesh_areas(n) / mod2med_areacor(n)
+    end do
+    deallocate(model_areas)
+    deallocate(mesh_areas)
+
+    min_mod2med_areacor = minval(mod2med_areacor)
+    max_mod2med_areacor = maxval(mod2med_areacor)
+    min_med2mod_areacor = minval(med2mod_areacor)
+    max_med2mod_areacor = maxval(med2mod_areacor)
+    call shr_mpi_max(max_mod2med_areacor, max_mod2med_areacor_glob, mpicom_rof)
+    call shr_mpi_min(min_mod2med_areacor, min_mod2med_areacor_glob, mpicom_rof)
+    call shr_mpi_max(max_med2mod_areacor, max_med2mod_areacor_glob, mpicom_rof)
+    call shr_mpi_min(min_med2mod_areacor, min_med2mod_areacor_glob, mpicom_rof)
+
+    if (masterproc) then
+       write(iulog,'(2A,2g23.15,A )') trim(subname),' :  min_mod2med_areacor, max_mod2med_areacor ',&
+            min_mod2med_areacor_glob, max_mod2med_areacor_glob, 'RTM'
+       write(iulog,'(2A,2g23.15,A )') trim(subname),' :  min_med2mod_areacor, max_med2mod_areacor ',&
+            min_med2mod_areacor_glob, max_med2mod_areacor_glob, 'RTM'
+    end if
+

Then import_fields adds an extra argument for each field...

     allocate(temp(begr:endr,3))
 
-    call state_getimport(importState, 'Flrl_rofsur', begr, endr, output=temp(:,1), rc=rc)
+    call state_getimport(importState, 'Flrl_rofsur', begr, endr, output=temp(:,1), do_area_correction=.true., rc=rc)
     if (ChkErr(rc,__LINE__,u_FILE_u)) return

Which then does this...

+    ! determine output array
+    if (do_area_correction) then
+       fldptr(:) = fldptr(:) * med2mod_areacor(:)
     end if
+    do g = begr,endr
+       output(g) = fldptr(g-begr+1)
+    end do
+
+    ! check for nans
+    call check_for_nans(fldptr, trim(fldname), begr)

Similar stuff for export state...

slowness in writing restart file

data structure changes?

Possible bottol neck in making uh

It is known that subroutine mk_uh in irf_route takes hours when running large domain river network (e.g., CONUS GF routing). make_uh in irf_route.f90

-Need to do profiling.
-Need to do better handling memory allocations in this routine?

Need basin IRF routing option

issue: Some of runoff input is already routed over the hru, and now there is no option to turn off the basin routing.

A blank line in the user_ctl_mizuroute file gives a hard to decipher error

A blank line in the user_ctl_mizuroute file gives the following hard to dechiper error:

ERROR: Error in reading in line:

The fix is just to remove the blank line. But, we should make it more obvious what is going wrong and how the user might fix it.

Update mizuRoute to handle frozen stream

Update mizuRoute to handle the frozen stream for coupling to CESM.

mizuRoute contents

1. Move settings (configuration files) and ancillary data (netCDF attributes files) outside of the gitHub repo.

2. Move scripts directory out side of the gitHUB repo, and create new repo for this

Some cases are failing when running with CTSM for the gnu compiler

We have some cases that fail when running coupled with CTSM/CESM for the gnu compiler.

This is with

CTSM:
ctsm5.1.dev061-60-gda18e6577
mizuRoute (on cesm-coupling branch)
f564c19

For example:

SMS.f09_f09_mg17_rMERIT.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default

This happens just after initialization.

The error give is just shepherd terminated on cheyenne...

cesm.log

251:
1367:
1259:
MPT: shepherd terminated: r2i0n3.ib0.cheyenne.ucar.edu - job aborting

Various types of reach ID and HRU ID in network data

Issue: currently allow only 32bit integer for river network ID.

ideal: allow 32bit, 64bit integer and string for IDs.

Possible solution: Create module to read IDs based on netCDF type and then generate dictionary (hash table) to map internal ID (which use 32bit integer) to these external IDs. use internal ID for network topology augmentation.

Plan to work on this dev1.2_hru-id branch

Threaded test coupled with CTSM at f19_f19 MERIT-Hydro resolution with 40 tasks and 36 threads fails on cheyenne

Running the test SMS_P40x36.f19_f19_mg17_rMERIT.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default failed on cheyenneat initialization of mizuRoute. I think it ran out of memory because

SMS_P80x18.f19_f19_mg17_rMERIT.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default

works fine. This probably just shows that there needs to be a balance of OpenMP processors and memory. So you can't use too many OpenMP processors or you'll run out of memory.

user_ctl_mizuRoute file doesn't work for some settings

The user_ctl_mizuRoute override for a specific case isn't working to control some settings (like changing filenames). It works for other settings (like route_opt). user_nl_mizuRoute seems to work fine.

Get build to work with either gmake or regular make

We'd like to get the build to work with either GNU-make or regular make. There are some things I took from the CESM example build that require GNU make.

Add list of files needed to be read in to mizuroute.input_data_list

In the buildnml we need to add the list of files that mizuRoute is going to read to the file: for.input_data_list. Currently we've been relying on those files already being in place for mizuRoute to work. Adding this will allow cime to download them before the run is done. I noticed this starting to work on izumi, since it wasn't automatically downling the needed files.

USGS_GFmz grid fails to run

The USGS_GFmz grid fails to run because of a known issue with there being a different number of river segments to HRU's. @nmizukami tried a simple fix for this that allowed it to work, but had impact on other grids.

Three tests that fail are:

ERS.nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default RUN
ERS.nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default RUN
SMS_D.nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default RUN

The intel and nag ERS tests are also expected to fail on izumi.

ERS.nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.izumi_intel.mizuroute-default RUN
ERS.nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.izumi_inag.mizuroute-default RUN

On cheyenne the SMS_D test seems to fail in datm with the following backtrace in the cesm.log file..

5:MPT: #1  0x00002b67a1552306 in mpi_sgi_system (
5:MPT: #2  MPI_SGI_stacktraceback (
5:MPT:     header=header@entry=0x7ffe0f65f3d0 "MPT ERROR: Rank 5(g:5) received signal SIGFPE(8).\n\tProcess ID: 63742, Host: r13i0n2, Program: /glade/scratch/erik/tests_ctsm51d86mizum/SMS_D.nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.cheyenne_intel.mizur"...) at sig.c:340
5:MPT: #3  0x00002b67a15524ff in first_arriver_handler (signo=signo@entry=8, 
5:MPT:     stack_trace_sem=stack_trace_sem@entry=0x2b67b0e60080) at sig.c:489
5:MPT: #4  0x00002b67a1552793 in slave_sig_handler (signo=8, siginfo=<optimized out>, 
5:MPT:     extra=<optimized out>) at sig.c:565
5:MPT: #5  <signal handler called>
5:MPT: #6  0x00000000009b00ec in datm_datamode_clmncep_mod::datm_datamode_clmncep_advance (mainproc=.FALSE., logunit=6, mpicom=20, rc=0)
5:MPT:     at /glade/work/erik/ctsm_mizuRoute/components/cdeps/datm/datm_datamode_clmncep_mod.F90:422
5:MPT: #7  0x00000000009a2bd6 in atm_comp_nuopc::datm_comp_run (importstate=..., 
5:MPT:     exportstate=..., target_ymd=20000101, target_tod=0, target_mon=1, 
5:MPT:     orbeccen=0.016703660392765603, orbmvelpp=4.9374577904881578, 
5:MPT:     orblambm0=-0.032472495661529328, orbobliqr=0.40910112257977893, 
5:MPT:     restart_write=.FALSE., rc=0)
5:MPT:     at /glade/work/erik/ctsm_mizuRoute/components/cdeps/datm/atm_comp_nuopc.F90:630
5:MPT: #8  0x00000000009a07ee in atm_comp_nuopc::initializerealize (gcomp=..., 
5:MPT:     importstate=..., exportstate=..., clock=..., rc=0)
5:MPT:     at /glade/work/erik/ctsm_mizuRoute/components/cdeps/datm/atm_comp_nuopc.F90:411
5:MPT: #9  0x00002b679aead489 in ESMCI::FTable::callVFuncPtr (this=0xc1f5e00, 
5:MPT:     name=0xc19a540 "InitializeIC07P", vm_pointer=0xc1f5c80, 
5:MPT:     userrc=0x7ffe0f663088)
5:MPT:     at /glade/p/cesmdata/cseg/PROGS/build/28560/esmf-8.2.0b23/src/Superstructure/Component/src/ESMCI_FTable.C:2167
5:MPT: #10 0x00002b679aea8c31 in ESMCI_FTableCallEntryPointVMHop (vm=0xc1f5c80, 
5:MPT:     cargoCast=0xc19a540)

The gnu test shows a more sensible error inside of mizuRoute

36:MPT: 	add-auto-load-safe-path /glade/u/apps/ch/opt/gnu/10.1.0/lib64/libstdc++.so.6.0.28-gdb.py
36:MPT: line to your configuration file "/glade/u/home/erik/.gdbinit".
36:MPT: To completely disable this security protection add
36:MPT: 	set auto-load safe-path /
36:MPT: line to your configuration file "/glade/u/home/erik/.gdbinit".
36:MPT: For more information about this security protection see the
36:MPT: "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
36:MPT: 	info "(gdb)Auto-loading safe path"
36:MPT: Missing separate debuginfos, use: zypper install glibc-debuginfo-2.22-49.16.x86_64
36:MPT: (gdb) #0  0x00002ae3b05fb6da in waitpid ()
36:MPT:    from /glade/u/apps/ch/os/lib64/libpthread.so.0
36:MPT: #1  0x00002ae3b0941c66 in mpi_sgi_system (
36:MPT: #2  MPI_SGI_stacktraceback (
36:MPT:     header=header@entry=0x7ffced91f9d0 "MPT ERROR: Rank 36(g:36) received signal SIGABRT/SIGIOT(6).\n\tProcess ID: 7973, Host: r13i3n11, Program: /glade/scratch/erik/tests_ctsm51d86mizum/ERS.nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.cheyenne_gnu"...) at sig.c:340
36:MPT: #3  0x00002ae3b0941e66 in first_arriver_handler (signo=signo@entry=6, 
36:MPT:     stack_trace_sem=stack_trace_sem@entry=0x2ae3bdca0080) at sig.c:489
36:MPT: #4  0x00002ae3b09420f3 in slave_sig_handler (signo=6, siginfo=<optimized out>, 
36:MPT:     extra=<optimized out>) at sig.c:565
36:MPT: #5  <signal handler called>
36:MPT: #6  0x00002ae3b10d28d7 in raise () from /glade/u/apps/ch/os/lib64/libc.so.6
36:MPT: #7  0x00002ae3b10d3caa in abort () from /glade/u/apps/ch/os/lib64/libc.so.6
36:MPT: #8  0x00002ae3b11101b4 in __libc_message ()
36:MPT:    from /glade/u/apps/ch/os/lib64/libc.so.6
36:MPT: #9  0x00002ae3b11159d6 in malloc_printerr ()
36:MPT:    from /glade/u/apps/ch/os/lib64/libc.so.6
36:MPT: #10 0x0000000000aa8c15 in __mpi_process_MOD_comm_ntopo_data ()
36:MPT: #11 0x0000000000a7c95a in __init_model_data_MOD_init_ntopo_data ()
36:MPT: #12 0x0000000000a74a34 in __rtmmod_MOD_route_ini ()
36:MPT: #13 0x0000000000a65d6b in __rof_comp_nuopc_MOD_initializerealize ()

Similarly the ERS intel test shows an error inside of mizuRoute for MPI

36:MPT: #20 0x00000000014c3ac0 in for_alloc_allocatable ()
36:MPT: #21 0x0000000000eebdb1 in mpi_utils_mp_shr_mpi_allgatherlogicalv_ ()
36:MPT:     at /glade/work/erik/ctsm_mizuRoute/components/mizuRoute/route/build/src/mpi_utils.f90:521
36:MPT: #22 0x0000000000ec9f8e in mpi_process_mp_comm_ntopo_data_ ()
36:MPT:     at /glade/work/erik/ctsm_mizuRoute/components/mizuRoute/route/build/src/mpi_process.f90:410
36:MPT: #23 0x0000000000ec07ab in init_model_data_mp_init_ntopo_data_ ()
36:MPT:     at /glade/work/erik/ctsm_mizuRoute/components/mizuRoute/route/build/src/init_model_data.f90:228
36:MPT: #24 0x0000000000eb1e04 in rtmmod_mp_route_ini_ ()
36:MPT:     at /glade/work/erik/ctsm_mizuRoute/components/mizuRoute/route/build/cpl/RtmMod.F90:177
36:MPT: #25 0x0000000000ea0975 in rof_comp_nuopc_mp_initializerealize_ ()
36:MPT:     at /glade/work/erik/ctsm_mizuRoute/components/mizuRoute/route/build/cpl/nuopc/rof_comp_nuopc.F90:521
36:MPT: #26 0x00002ac1d95b3dee in ESMCI::FTable::callVFuncPtr(char const*, ESMCI::VM*, int*) ()

I/O frequency/timing flexibility

Now:
only one runoff netcdf is read in and output the results at the same time step for the same period as the input.

Potential improvement:
Add the capability to start and end a user specified time
Add the capability to read multiple netcdfs
Add the capability to generate and write output netcdf(s) with frequency specified by user (e.g., monthly, annual etc)

flux and state data structure organization

It is proposed that one routing scheme is used for one simulation (i.e., remove route_opt = 0, route with all the schemes), because as new routing schemes are added, using all the schemes at the same run is not practical.

With this in mind, there might be need for re-organization of flux data structure (STRFLX) and state data structures (KREACH for kinematic wave).

Suggestion

For Flux, the same data structure is shared with different routing schemes
e.g.,

TYPE, public :: STRFLX
   real(dp)           :: BASIN_QI      ! instantaneous runoff volume from the local basin (m3/s)
   real(dp)           :: BASIN_QR(0:1) ! routed runoff volume from the local basin (m3/s)
   real(dp)           :: REACH_Q       ! time-step average streamflow (m3/s)
   real(dp)           :: UPSTREAM_QI   ! sum of upstream streamflow (m3/s)
   real(dp)           :: TAKE          ! average take
   logical(lgt)       :: CHECK_ROUTE   ! .true. if the reach is routed
END TYPE STRFLX

For states, each scheme has different state variable. so separate, unique data structure needs to be defined.

ERI tests fail because of trouble doing a hybrid startup

ERI tests are failing on the "ref2" step trying to do the hybrid startup piece. The problem is that the restart file is set in mizuroute.control as follows...

<fname_state_in> STATE_IN_NC

And then mizuroute fails trying to read that file:

rof.log

  Reading Slope into structure SEG
  Reading seg_id into structure NTOPO
  Reading Tosegment into structure NTOPO
 ERROR: (route_ini)init_state_data/read_state_nc/get_nc_dim_len/[No such file or directory; file=./STATE_IN_NC]

I'm guessing likely the same problem will happen on the "branch" step. And I assume this means there is something wrong in the buildnml code for mizuRoute for hybrid or branch run types.

Connect PIO xml settings

Currently the cesm-coupling branch has the PIO settings hardcoded in, and doesn't respond to the XML ROF PIO settings. This needs to change so that it responds to the XML settings. @jedwards4b pointed this out in issue #184

tag vs release

can you tag the commit but no release?
if so, what is the rule of thumb for only tag (no release) and both?

Two cesm-coupling half-degree tests that fail

There are two tests that are failing for me with latest add_mizuroute CTSM branch and latest cesm-coupling branch for mizuRoute.

SMS.hcru_hcru_mt13.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-hcru
SMS.hcru_hcru_mt13_rHDMA.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-hcru

It just seems to stop after initialization, and I can't really see what it going wrong or why it's aborting. So it ends with status PEND since MPI doesn't shut down properly.

Limitation: Different number of tasks change answers

I'm not sure if this is expected or not. But, I did find that mizuRoute changes answers if a different number of tasks is used. The test I did for this is:

PEM_Ld10.nldas2_rMERIT_mnldas2.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default

The test has no answer changes for CTSM or CPL, but mizuRoute fields change as follows...

 RMS basRunoff                        3.0671E-05            NORMALIZED  2.6883E+00
 RMS instRunoff                       1.8783E+00            NORMALIZED  3.6466E+00
 RMS dlayRunoff                       1.2893E+00            NORMALIZED  3.4618E+00
 RMS sumUpstreamRunoff                6.4705E+02            NORMALIZED  1.4161E+01
 RMS KWTroutedRunoff                  6.5100E+02            NORMALIZED  1.5828E+01
 RMS IRFroutedRunoff                  2.6097E+02            NORMALIZED  9.0326E+00
 RMS volume                           2.1904E+06            NORMALIZED  1.1718E+01