uw-hydro / rvic Goto Github PK
View Code? Open in Web Editor NEWRVIC Streamflow Routing Model
Home Page: http://rvic.readthedocs.org
License: GNU General Public License v3.0
RVIC Streamflow Routing Model
Home Page: http://rvic.readthedocs.org
License: GNU General Public License v3.0
Todo:
uwhydro
binstar channel: UW-Hydro/conda-recipes-uwhydro#1It would be cool to have an option to output data with the coordinates (time, basin_id) instead of (time, lat, lon). Unless an application requires routing to virtually every grid cell, most users only want the time series at a series of points. The output files should also be smaller if an empty time series isn't saved for every other grid cell. I am willing to work on this in a couple of weeks from now if it seems reasonable.
I've been using the RVIC parameters generation tool to generate parameters on a 10km grid in the Arctic. The temporary remap files are pretty large (16GB each) and are not being cleaned fast enough to save enough disk space for the next file to be able to fit on the disk (200 GB in total).
For large domains with many outlets, we may need to rethink the workflow when using REMAP=TRUE
and multiprocessing
. For now, the work around seems that we need to use less processors so the total temporary file load remains less than the disk space.
@yixinmao found a bug in finish_param()
that is leaving some outlets with less source grid cells than it should. In the test case below, the RVIC parameter file was missing about 15% of the source grid cells leading to mass balance error:
Formatting changes are needed to support python 3.
Cdo.py will also need to support python 3 before this will work.
from @tcchiao:
To reproduce the 31 day months, try setting RVICHIST_MFILT : month and RVICHIST_NHTFRQ : 1. It gives me output files on Jan 31st, March 3rd, April 3rd, May 4th, and so on.
In RVIC/docs/user-guide/parameters.md, the section titled "RVIC Parameter Configuration File" actually uses the convolution configuration file as the example. Swapping in the parameter config file template would be useful.
Restarts, drystarts, startups, etc.
Currently, there is a bit of a convoluted coordinate mapping routine in parameters.py
. This could be simplified, in part, using the logic shown here: http://earthpy.org/interpolation_between_grids_with_ckdtree.html
RVIC produces diagnostic plots when creating its parameter files. However, these figures are not closed once they have been written to disk and this can lead to a build up of open figure handles when running RVIC in a interactive session (e.g. ipython notebook).
Following on issue #92 The area-weighting uses grid_area
as a multiplier then divides by self._outlet_upstream_area
; however, self._outlet_upstream_area
is the multiple of the grid cell area and the grid cell fraction link. grid_area
appears to just be the grid cell area link, not the grid cell area within the basin. grid_area
should probably be grid_area * dom_data[dom_frac][outlet.y_source, outlet.x_source]
When developing parameters with > 1 processor, the multiprocessing step fails silently. All tests pass on a single processor.
This is true for both python 2.7 and 3.4.
Develop branch.
In history.py, units are meant to be converted from kg m^-2 s^-1 to mm/day. However, (kg m^-2 s^-1) * (m^2) * (s d^-1) * (m^3 kg^-1) * (m^-2) yields m/day. The conversion to mm is still needed.
From Cindy...
I was looking at my routed flow results and realized that they are 1 year short of my forcings (netcdf vic output). I think this is because in read_forcing.py for "yearly files" inputs, a for loop using "for year in xrange(start, end)" was used to read in the files. As a result, it actually only go up to (end-1) year (see line 57-62).
This can be quickly fixed but since I would want the last year of my data today, I was wondering if I can just re-route the last 1.5 year and use the last year? I remember you saying that the routing model doesn't need a lot of time for spin up. Let me know if that's a valid option, thanks a lot!
I get some error messages when executing rvic parameters with the RASM sample dataset, but the code is able to generate a final parameter file in netcdf format. Although, when using this parameter file for executing rvic convolution I get more errors. Here are the lines with the error messages when generating RVIC parameters:
ERROR:write>> /usr/local/lib/python2.7/dist-packages/rvic-1.1.1-py2.7-linux-x86_64.egg/rvic/core/param_file.py:331: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
ERROR:write>> dtype=np.float64)
ERROR:write>> /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3114: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
ERROR:write>> dout = self.data[indx]
ERROR:write>> /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3169: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
ERROR:write>> dout._mask = _mask[indx]
INFO:subset>> Done subsetting
DEBUG:finish_params>> plotting unit hydrograph timeseries now for before / after subseting
INFO:finish_params>> UHS before subset Plot: ../cases/sample_rasm_parameters/plots/uhs_before_subset_sample_rasm_parameters_20170511.png
INFO:finish_params>> UHS after subset Plot: ../cases/sample_rasm_parameters/plots/uhs_after_subset_sample_rasm_parameters_20170511.png
INFO:finish_params>> Adjusting Fractions to be less than or equal to domain fractions
INFO:adjust_fractions>> Adjusting fractions now
INFO:adjust_fractions>> Adjust fractions for 0 grid cells
DEBUG:group>> n_outlets: 5
DEBUG:group>> n_sources: 10
DEBUG:group>> subset_length: 10.0
ERROR:write>> /usr/local/lib/python2.7/dist-packages/rvic-1.1.1-py2.7-linux-x86_64.egg/rvic/core/param_file.py:405: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
ERROR:write>> dtype=np.float64)
I attach the file with the whole log:
run_rvic_parameters.txt
On the other hand, the error messages that appear when running rvic convolution are shown below:
ERROR:write>> Traceback (most recent call last):
ERROR:write>> File "/usr/local/bin/rvic", line 4, in <module>
ERROR:write>> __import__('pkg_resources').run_script('rvic==1.1.1', 'rvic')
ERROR:write>> File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 719, in run_script
ERROR:write>> self.require(requires)[0].run_script(script_name, ns)
ERROR:write>> File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1504, in run_script
ERROR:write>> exec(code, namespace, namespace)
ERROR:write>> File "/usr/local/lib/python2.7/dist-packages/rvic-1.1.1-py2.7-linux-x86_64.egg/EGG-INFO/scripts/rvic", line 61, in <module>
ERROR:write>> main()
ERROR:write>> File "/usr/local/lib/python2.7/dist-packages/rvic-1.1.1-py2.7-linux-x86_64.egg/EGG-INFO/scripts/rvic", line 50, in main
ERROR:write>> convolution(args.config_file)
ERROR:write>> File "/usr/local/lib/python2.7/dist-packages/rvic-1.1.1-py2.7-linux-x86_64.egg/rvic/convolution.py", line 54, in convolution
ERROR:write>> time_handle, directories)
ERROR:write>> File "/usr/local/lib/python2.7/dist-packages/rvic-1.1.1-py2.7-linux-x86_64.egg/rvic/convolution.py", line 350, in convolution_run
ERROR:write>> history_restart_files)
ERROR:write>> File "/usr/local/lib/python2.7/dist-packages/rvic-1.1.1-py2.7-linux-x86_64.egg/rvic/core/variables.py", line 427, in write_restart
ERROR:write>> setattr(timemgr_rst_step_sec, key, val)
ERROR:write>> File "netCDF4/_netCDF4.pyx", line 3702, in netCDF4._netCDF4.Variable.__setattr__ (netCDF4/_netCDF4.c:37094)
ERROR:write>> ValueError
ERROR:write>> :
ERROR:write>> invalid literal for float(): 0, 86400
And this is the log file for the convolution:
run_rvic_convolution.txt
I am trying to find the solution on internet, but it seems that there is nobody with the same problem. I will really appreciate it if you give me some help. I also attach the paramater and convolution configuration files:
Thanks!!
The netcdf history file produced by rvic convolution
contains a dummy time step at the end. For example, in my file it is `"0000-01-01":
data: time = "1980-01-01", "1980-01-02", "1980-01-03", "1980-01-04", "1980-01-05", ... "1980-12-23 12", "1980-12-24 12", "1980-12-25 12", "1980-12-26 12", "1980-12-27 12", "1980-12-28 12", "1980-12-29 12", "1980-12-30 12", "1980-12-31 12", "0001-01-01" ; }
Hey, I'm running the convolution and getting the error:
INFO:_check_domain_file>> domain_file: domain.arcardia.nc
INFO:_check_domain_file>> Parameter RvicDomainFile: Routing.nc
[...]
ERROR:write>> raise ValueError('dom files do not match in parameter and '
ERROR:write>> ValueError
ERROR:write>> :
ERROR:write>> dom files do not match in parameter and domain file
This error is pretty general and doesn't really tell me what the problem is.
The wierd thing is, that the convolution doesn't need/have Routing.nc
as an input or dies that just mean the routing information in the parameter file?
I didnt have problems running the sample files, but now I'm running my own data, I get this error.
I checked the data and they are consistent.
Philipp
subset_length
needs to be cast as an int in either line 52 or 330 of param_file.py
since it is used on line 330 to initialize out_uh
array.
When running fraction2domain.bash, we encounter the following error:
+ ncap2 -O -v -s mask=frac temp_frac.nc temp_mask1.nc
Command-line script:1:5: expecting (, found '='
It is possible that 'mask' is a reserved word in ncap2 expression.
lines 333, 334 of param_file.py
in subset
function define d_left
and d_right
, which need to be integers, to be +/- subset_length / 2
, which will be float if 'subset_length' is not a multiple of 2. Since d_left
and d_right
are used as array indices, they cannot have float values, yet there is no logic in rvic to ensure that subset_length
is an even number.
test raises:
local variable 'full_time_length' referenced before assignment
when remap=False
.
Needs further investigation. May just be a error in the test module.
Current, RVIC is using an ad hoc input/output units scheme, checking for a few known combinations of unit strings (e.g. 'mm', 'kg m-2 s-1', etc). There are a few Python projects that have wrapped the UDUNITS library that support conversion of a range of units.
Two packages that come to mind:
One of these should be included in read_forcings.py
and history.py
to handle the units conversion that is currently done in a more sophisticated way.
CC: @bartnijssen, @mrstu
related: #55
RVIC currently has a lot of homegrown datastores that are trying to replicate the netCDF datamodel. Adopting the xray data structures would dramatically reduce the amount of source code in RVIC. I think a internal refactor would be pretty straightforward and would reduce the overall brittleness of the model.
RVIC convolution is failing on the sample dataset. The error seems to occur while writing a restart file, in particular in lines 425, 441, 459 and 475 of rvic/core/variables.py. The look like the next ones:
for key, val in iteritems(share.timemgr_rst_curr_tod):
if val:
setattr(timemgr_rst_curr_tod, key, val)
The error that shows up is the one below
ERROR:write>> setattr(timemgr_rst_step_sec, key, val)
ERROR:write>> File "netCDF4/_netCDF4.pyx", line 3679, in netCDF4._netCDF4.Variable.__setattr__ (netCDF4/_netCDF4.c:38607)
ERROR:write>> ValueError
ERROR:write>> :
ERROR:write>> could not convert string to float: '0, 86400'
Specifically it occurs while writing valid_range. If I comment the lines above, the program finishes without any error. I am pretty sure that those lines are there for a reason, however, I do not know how to solve the issue.
I have installed RVIC through anaconda in OS X Sierra, although I was getting this very same error on Ubuntu. I installed the cdo libraries from Homebrew because Anaconda's version did not seem to be working.
When running RVIC across multiple sites, the output history file has a time index that ends with this (taken from output of ncdump -c -t
on the file):
...734490.5, 734491.5, 734492.5, 734493.5, 734494.5, 734495.5, 734496.5,
734497.5, 734498.5, 734499.5, 734500.5, 734501.5, 734502.5, 734503.5,
-Infinityf ;
The last few timestamps of the forcing file look like this:
"2011-12-17", "2011-12-18", "2011-12-19", "2011-12-20", "2011-12-21",
"2011-12-22", "2011-12-23", "2011-12-24", "2011-12-25", "2011-12-26",
"2011-12-27", "2011-12-28", "2011-12-29", "2011-12-30", "2011-12-31"
Relevant parts of the convolution config file:
[HISTORY]
#--Number of History tapes.
RVICHIST_NTAPES : 1
#--Per tape series maximum number of time samples.
RVICHIST_MFILT : 100000
#--Per tape series history file density (i.e. output precision)
# 1=double precision
# 2=single precision
RVICHIST_NDENS : 1
#--Per tape series history write frequency.
# positive means in time steps
# 0 = monthly
# negative means hours
RVICHIST_NHTFRQ : 1
#--Per tape series history write frequency.
# A - Average, over the output interval.
# I - Instantaneous, output the value at the output interval.
# X - Maximum, over the output interval.
# M - Minimum, over the output interval.
RVICHIST_AVGFLAG : A
#-- Per tape series history file format (char)
# Valid Values: grid, array
RVICHIST_OUTTYPE: array
RVICHIST_NCFORM: NETCDF3_CLASSIC
#-- Per tape series output units (char)
# Valid Values: kg m-2 s-1, m3/s
RVICHIST_UNITS: m3/s
@orianac brought up this question in our chat about the model yesterday. Currently, RVIC
does not use numpy
's masked arrays, choosing the speed of pure numpy
arrays rather that the automatic conversion from fill value
to masked arrays that netCDF4
offers. As a result, if there is a fill value in your forcing data where you are routing flows, that fill value will be routed downstream.
Relevant code is here. Note the lines with
self.current_fhdl.variables[fld].set_auto_maskandscale(False)
where the masked array is declined.
Things I do not want to see:
What we probably want to do:
read_forcing.py
. If a fill value will be routed, I think we should raise an error.@orianac, thoughts? I assigned this to you since you had already attempted one fix.
It shouldn't be too difficult to apply the convolution (rvic_convolve
) in parallel.
It is currently possible for RVIC to rout fill values if the routing mask and forcing mask do not match.
To fix, implement a check in convolution.init()
insuring that all routing source points have real data in them.
Given the difficulties with ensuring the correct grid cell is chosen during the parameter generation, it would be fairly simple to add a check to make sure the model determined basin area was within a certain margin of error of a value provided by the user. In cases where it is not, a simple warning message should be printed.
Dear sir,
I read this paper "Effects of irrigation on the water and energy balances of the Colorado and Mekong river basins" and found that irrigation module and reservoir module have been added into the routing routine. Therefore, I want to ask, does RVIC include irrigation module and reservoir module?
Thanks!
I'm finding negative integers in the source_decomp_ind
and outlet_decomp_ind
variables in the generated parameters netCDF. This is definitely a bug that needs to be fixed right away (RVIC 1.1.1).
I'm getting this error running RVIC in hourly timesteps.
INFO:get_time_mode>> Coupling Timestep is (seconds): 3599.99999665
INFO:get_time_mode>> RVIC Timestep is (seconds): 86400.0
This problem seems to be (another) floating-point problem since the internal calculation does compute 1/24*8640
which ends up being an double instead of an int:
RVIC/rvic/core/read_forcing.py
Line 168 in 622f45d
I worked around it by rounding the result whoch gives me 3600 seconds :
self.secs_per_step = round((t1 - t0) * SECSPERDAY)
See thecommit in my fork:
PhilippBuehler@8f333ce
I also created a pull request:
#104
I have to do additional testing of any of the other errors are related to my tweak (right now I'm getting ERROR:write>> local variable 'end_timestamp' referenced before assignment
)
Philipp
In parameters.py
Characters such as (){}[]!@#$%^&*'"?><`~+= should be stripped from station names before they are assigned to filenames.
When I install rvic using the conda package manager (
conda install --channel https://conda.anaconda.org/UW-Hydro rvic) and run the following command:
rvic parameters samples/configs/rvic.parameters.rasm.cfg
I get the following errors:
When REMAP = False in config file:
ImportError: dlopen(/Users/user/miniconda3/envs/test4/lib/python3.4/site-packages/netCDF4/_netCDF4.so, 2): Library not loaded: @rpath/libjpeg.8.dylib
Referenced from: /Users/user/miniconda3/envs/test4/lib/libnetcdf.11.dylib
When REMAP = True in config file:
ImportError: dlopen(/Users/user/miniconda3/envs/test8/lib/python3.4/site-packages/netCDF4/_netCDF4.so, 2): Library not loaded: @rpath/libjpeg.8.dylib
Referenced from: /Users/user/miniconda3/envs/test8/lib/libnetcdf.11.dylib
Reason: image not found
I am using OSX 10.12.5, but got similar issue on Ubuntu 16.04
UPDATE: I have reinstalled CDO with NetCDF, now I am getting the error:
ERROR:write>> f = Dataset(nc_file, 'r')
ERROR:write>> File "netCDF4/_netCDF4.pyx", line 1848, in netCDF4._netCDF4.Dataset.init (netCDF4/_netCDF4.c:13992)
ERROR:write>> OSError
ERROR:write>> :
ERROR:write>> NetCDF: Unknown file format
This is my CDO configuration:
Climate Data Operators version 1.7.2 (http://mpimet.mpg.de/cdo)
Compiled: by davidedelerma on Davide.local (x86_64-apple-darwin16.6.0) Jun 25 2017 08:43:48
Compiler: gcc -fPIC -D_THREAD_SAFE -pthread
version: Apple LLVM version 7.0.2 (clang-700.1.81)
Features: DATA PTHREADS HDF5 NC4/HDF5 OPeNDAP SSE3
Libraries: HDF5/1.8.19
Filetypes: srv ext ieg grb grb2 nc nc2 nc4 nc4c
CDI library version : 1.7.2 of Jun 25 2017 08:43:25
CGRIBEX library version : 1.7.5 of Jun 3 2016 14:44:00
GRIB_API library version : 1.14.4
NetCDF library version : 4.4.1.1 of Jun 24 2017 22:04:42 $
HDF5 library version : 1.8.19
SERVICE library version : 1.4.0 of Jun 25 2017 08:43:21
EXTRA library version : 1.4.0 of Jun 25 2017 08:43:19
IEG library version : 1.4.0 of Jun 25 2017 08:43:20
FILE library version : 1.8.2 of Jun 25 2017 08:43:20
Thanks,
Davide
There are a bunch of sloppy numpy appends in the finish params functions. These can all be removed by counting the total number of source and outlet points before subseting, or combining arrays.
Currently, the last history file will not be written if the end of the simulation does not occur at the same time that the history file is full. This will require adding a new method to the History
class to write the current tape values.
This should also be done in such a way as to support the addition of history file restarts.
Rather than having a set number of time samples per output file, provide an option to set RVICHIST_MFILT
as year
, month
, etc.
When I run the command
conda install --channel https://conda.anaconda.org/UW-Hydro rvic
in a new conda environment, I get an error
UnsatisfiableError: The following specifications were found to be in conflict:
I am guessing there is an issue with the dependencies specifications in the conda package, but I am fairly inexperienced; it might just have to do with my system.
RVIC has been getting more bug report of weird rounding behavior. This is a common issue that arrises when supporting Python 2 and 3 simultaneously. Good news is, core dependencies like numpy have identified a timeline for sunsetting Python 2 (ref: http://www.python3statement.org/). I suggest removing Python 2 support in the next major release. This will help eliminate some of these pesky little bugs.
cc @bartnijssen
The use of the term "unit hydrograph" for both the overland flow to the channel, and again for flow within the channel, appears to be confusing for new users. I think it would be possible to clear up some of the confusion by changing the name of the overland unit hydrograph from UH_BOX
(which may not connote overland flow to a new user) to UH_land
or something similar. Along the same lines, perhaps call the other unit hydrographs UH_channel
or something similar.
Also, perhaps the presence of 2 steps, each involving different unit hydrographs, could be made clearer in the documentation by changing both the text on the readthedocs page to clearly label the 2 stages and also by numbering the stages explicitly in the left-hand menu bar.
These are just suggestions, but might help.
When I call rvic.parameters.parameters
with multiple cores in Python 2.7.11, I get the following error:
INFO:parameters>> On Outlet 1 of 1
ERROR:write>> Traceback (most recent call last):
ERROR:write>> File "run_rvic.py", line 62, in
ERROR:write>> main()
ERROR:write>> File "run_rvic.py", line 56, in main
ERROR:write>> rvic.parameters.parameters(param_cfg_name, numofproc=cores)
ERROR:write>> File "/d6/eclark/miniconda2/envs/otl_env/lib/python2.7/site-packages/rvic-1.1.0-py2.7-linux-x86_64.egg/rvic/parameters.py", line 73, in parameters
ERROR:write>> error_callback=error)
ERROR:write>> TypeError
ERROR:write>> :
ERROR:write>> apply_async() got an unexpected keyword argument 'error_callback'
Apparently error_callback
is not a keyword option for apply_async
in python 2 (ref). In this case, it looks like nothing happens if there is an error; the callback just isn't called. I can think of 3 options for how to handle this:
error_callback
if user is using python 3. Set up an error check in the stat
return value of apply_async
to show if the job completed or not. The problem, of course, being that you don't get the error code from gen_uh_run
.gen_uh_run
.Monthly history files are only created for January in the first year. In subsequent years there is no file for January. Instead, December for the previous year is averaged over 2 months (62 days). E.g.:
gunnison_001.rvic.ha.2003-01.nc
gunnison_001.rvic.ha.2003-02.nc
gunnison_001.rvic.ha.2003-03.nc
gunnison_001.rvic.ha.2003-04.nc
gunnison_001.rvic.ha.2003-05.nc
gunnison_001.rvic.ha.2003-06.nc
gunnison_001.rvic.ha.2003-07.nc
gunnison_001.rvic.ha.2003-08.nc
gunnison_001.rvic.ha.2003-09.nc
gunnison_001.rvic.ha.2003-10.nc
gunnison_001.rvic.ha.2003-11.nc
gunnison_001.rvic.ha.2003-12.nc
MISSING
gunnison_001.rvic.ha.2004-02.nc
gunnison_001.rvic.ha.2004-03.nc
gunnison_001.rvic.ha.2004-04.nc
gunnison_001.rvic.ha.2004-05.nc
gunnison_001.rvic.ha.2004-06.nc
gunnison_001.rvic.ha.2004-07.nc
time_bnds for gunnison_001.rvic.ha.2003-12.nc are:
time_bnds =
731564, 731626 ;
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.