uw-hydro / tonic Goto Github PK

There appears to be a mismatched between the configuration file (example) and the code. It gives a key error ("Options" not found).

Email from Shervan:

I am trying to move to VIC image and I would like to transfer all my files which I prepared for the classical version (version 5) to NetCDF format. I am using tonic toolkit.

I have encounter couple of issues which I managed to solve but there is an issue which persists.

The issue is in grid_params function when the code try to add bare soil parameter in case it is not specify in the veg library.

My code together with the error messages is as follow; I have tried many combination but the issue still persists (such as removing the bare soil None changing it or adding or removing version_in and…). I tied to find the issue in the code it is around line 1010 to 1020 with new[:-1,yi,xi] = … The example is from stehekin basin. I just simplify the vegetation param to include only one vegetation type.

I would appreciate if you can give me some insight into why this problem remains as it remains. I hope the issue is not very stupid…

BTW, I see that apparently there is no version_in 5. I am working with version 5 and basically I want to have the classical files to be in a netcdf. I am not sure if this make lot of difference in this case.

from tonic.models.vic.grid_params import soil, snow, veg, veg_class, Cols, Desc, calc_grid, grid_params

n_veg_classes = 11
root_zones = 3

# Read the soil parameters
soil_dict = soil('D:/VIC_modeling/stehekin/params/vic/soil_param.txt', c=Cols(nlayers=3))

#%%
# Read the snow parameters
snow_dict = snow('D:/VIC_modeling/stehekin/params/vic/snowbands.txt',
                 soil_dict, c=Cols(snow_bands=5))

#%%
# Read the veg parameter file
veg_dict = veg('D:/VIC_modeling/stehekin/params/vic/veg_param_1.txt',
               soil_dict,
               vegparam_lai=True,
               lai_src = 'FROM_VEGPARAM',
               veg_classes=n_veg_classes)

#%%
# Read the veg library file
veg_lib = veg_class('D:/VIC_modeling/stehekin/params/vic/veglib.txt')
#%%
# Determine the grid shape
target_grid, target_attrs = calc_grid(soil_dict['lats'], soil_dict['lons'])
 
#%%
# Grid all the parameters
grid_dict = grid_params(soil_dict, target_grid, version_in='4.2',
                        veg_dict=veg_dict, veglib_dict=veg_lib, snow_dict=snow_dict,
                        vegparam_lai=True, lai_src = 'FROM_VEGPARAM',
                        lib_bare_idx=None)​

yields

  File "C:\Users\SHERVAN\Anaconda3\lib\site-packages\tonic-0.0.0.dev_67cc1a1-py3.6.egg\tonic\models\vic\grid_params.py"

    new[:-1, yi, xi] = veglib_dict[lib_var][:, np.newaxis]

 TypeError: tuple indices must be integers or slices, not str

snowbands.txt
soil_param.txt
veg_param_1.txt
veglib.txt

I am getting this error while using the Tonic package:

 File "~\grid_params.py", line 1013, in grid_params
    new[:-1, yi, xi] = veglib_dict[lib_var][:, np.newaxis]

KeyError: 'lib_overstory'

• overstory
• Ctype
• Nscale
• fs_active
• lake_idx
• numnod
• Nveg

When running vic2netcdf, an error occurs when outputting 4-d variables (e.g. soil moisture) using the standard memory mode.

I used the "standard" option and got an error. If I tried to include soil moisture in the configuration file I got the following error when the code tries to write to the netcdf files after it loads all the files in the current chunk:

Traceback (most recent call last):
File "./bin/vic_utils", line 5, in
pkg_resources.run_script('tonic==0.0.0.dev-2bf5167', 'vic_utils')
File "/glade/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib/python2.7/site-packages/pkg_resources.py", line 534, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/glade/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib/python2.7/site-packages/pkg_resources.py", line 1441, in run_script
exec(script_code, namespace, namespace)
File "./lib/python2.7/site-packages/tonic-0.0.0.dev_2bf5167-py2.7.egg/EGG-INFO/scripts/vic_utils", line 221, in
File "./lib/python2.7/site-packages/tonic-0.0.0.dev_2bf5167-py2.7.egg/EGG-INFO/scripts/vic_utils", line 197, in main
File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 546, in _run
File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 896, in vic2nc
File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 459, in nc_add_data_standard
File "netCDF4.pyx", line 3267, in netCDF4.Variable.setitem (netCDF4.c:39658)
ValueError: total size of new array must be unchanged

I traced it back to line ~448:

self.f.variables[name][:, i, ys, xs]

is looking for something that is 2-dimensional while

p.df[sn].values[self.slice]

is only 1 dimensional with a length set at the number of time steps going into the current netcdf file. If I removed soil moisture in the configuration file, I got this option to output properly, so it is an issue with 4-d variables.

Insures that missing coords are included.

Here are a few ideas for tomorrow's sprint:

• standardized read function/class for main models (VIC, SUMMA, PRMS, ULM)
• unit testing
• setup conda/binstar repository to support: conda install tonic
• documentation (maybe read-the-docs?)
• add plotting or analysis tools

feel free to add more ideas below if you have them

cc: @orianac

Apparently uncrustify hasn't been run in a while.

Hi, the Tonic setup page still refers to the Python package, xarray, as xray.

In 4.2 and 5.0 classic, vegcover optionally can be in the veg_lib and both vegcover and albedo can be in the veg_param file. This needs to be handled in tonic.

This also involves changing the logic for options - currently there is just a flag for global_lai (or lai_index in some cases) but this needs to be replaced by the set {veglib_vegcover, vegparam_lai, vegparam_vegcover, vegparam_albedo, lai_src, vegcover_src, and albedo_src}. global_lai would correspond to vegparam_lai = true and lai_src = from_vegparam.

Add suppot for veg_hist forcings.

Add support for the following options:
organic_fract
spatial_snow
spatial_frost
july_tavg_supplied

Add support for carbon variables

Getting an error when attempting to run snow17:

            elif ((qw >= deficit) and
                  ait((qw + w_q) <= ((deficit * (1 + plwhc)) + w_qx))):
                # THEN the snow is NOT yet ripe, but ice is being melted

TypeError: 'int' object is not callable

It looks like it's trying to call ait(bool). ait is a scalar value in this context.
Should ait simply be omitted here?

            elif ((qw >= deficit) and
                  ((qw + w_q) <= ((deficit * (1 + plwhc)) + w_qx))):

Make sure the latitude order convention matches between vic2netcdf.py and rvic

I'm using vic2netcf to convert the ascii data files into a netcdf file. However, there is an error when using standard memory mode, where the data cannot be inserted into the NetCDF variable.

Traceback (most recent call last):
File "/home/scott/venv/tonic/bin/vic_utils", line 4, in
import('pkg_resources').run_script('tonic===0.0.0.dev-190aca2', 'vic_utils')
File "/home/scott/venv/tonic/lib/python3.5/site-packages/pkg_resources/init.py", line 743, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/scott/venv/tonic/lib/python3.5/site-packages/pkg_resources/init.py", line 1505, in run_script
exec(script_code, namespace, namespace)
File "/home/scott/venv/tonic/lib/python3.5/site-packages/tonic-0.0.0.dev_190aca2-py3.5.egg/EGG-INFO/scripts/vic_utils", line 221, in
File "/home/scott/venv/tonic/lib/python3.5/site-packages/tonic-0.0.0.dev_190aca2-py3.5.egg/EGG-INFO/scripts/vic_utils", line 197, in main
File "/home/scott/venv/tonic/lib/python3.5/site-packages/tonic-0.0.0.dev_190aca2-py3.5.egg/tonic/models/vic/vic2netcdf.py", line 512, in _run
File "/home/scott/venv/tonic/lib/python3.5/site-packages/tonic-0.0.0.dev_190aca2-py3.5.egg/tonic/models/vic/vic2netcdf.py", line 861, in vic2nc
File "/home/scott/venv/tonic/lib/python3.5/site-packages/tonic-0.0.0.dev_190aca2-py3.5.egg/tonic/models/vic/vic2netcdf.py", line 462, in nc_add_data_standard
File "netCDF4/_netCDF4.pyx", line 4074, in netCDF4._netCDF4.Variable.setitem (netCDF4/_netCDF4.c:46416)
ValueError: cannot reshape array of size 70858 into shape (35429,2,2)

The problem is coming from Segment.nc_add_data_standard() when assinging the 2d array to the variable on line 462 as it's iterating over all combinations of (ys,xs) which is not the same size as data.

self.f.variables[name][:, ys, xs] = data

Tonic needs to be able to convert state files, for users to be able to switch from older versions to 5.0 or from classic to image mode.

In addition, it would be helpful to add the capability of generating an initial state file. This could allow us to remove that function from VIC itself. Once tonic has the ability to convert state files between ascii and netcdf, we would just need to add a function to define state variables, which would then be written to netcdf (or ascii, I suppose). The other parameter files would need to be read in for tonic to know the set of states that exist in each grid cell. For the default values, we could either port the logic currently in VIC or we could allow the user to specify the (a) fraction of maximum storage to initialize to (constant across all moisture storages) and (b) the initial temperature (constant across all temperatures).

Tonic could/should convert global parameter files between different versions.

In addition, the global parameter file could/should serve as the configuration file for tonic to set options. After all, that's the purpose of the global parameter file in VIC.

So, this kills 2 birds with one stone!

At present, tonic doesn't read the veg class descriptions ("comment" field, which contains a varying number of space-separated strings) from the veg library file. This is because it reads the veg_lib file with numpy.loadtxt(), which (a) can't handle inhomogeneous numbers of columns per row and (b) interprets all fields as the same type. So, it just reads all columns up to but not including the "comment" field and reads them as floats, which is fine for getting the bulk of the information out of the library file.

Perhaps to read the comments we should do a second read of the veg_lib file - at the moment I don't know what the options are for doing this, but presumably there's another way to parse text files that would allow this (albeit not as efficient as loadtxt()).

It's important to get the class names/descriptions into the netcdf file and it would be ideal for tonic to do it.

When I run vic_utils grid_params --help, I get the usage, like I would expect, but if I run vic_utils grid_params soil_file.txt, I get the same usage that I would get from vic_utils --help. According to the grid_params help, the soil file is the only required input. the others are not optional? grid_params() doesn't seem to need these parameters. Either way, I would think that the vic_utils grid_params usage would print to the screen instead of the vic_utils usage.

Your snow17.py script (line 263) has the following conditional expression that generates a syntax error:

ait((qw + w_q) <= ((deficit * (1 + plwhc)) + w_qx))):

Removing 'ait' fixes the syntax error. However, it seems this line should also be checked for correctness (even once 'ait' is removed).

I am trying to convert the livenh VIC parameter files to netcdf format.

I have installed tonic to a python 3.9 environment with these packages:

conda create -n tonic python=3.9
conda activate tonic
conda install -c conda-forge pandas numpy scipy xarray jupyter netCDF4 configobj

I am getting the following trace back from grid_params

TUPLE INDICIES MUST BE INTEGERS OR SLICES NOT STR

Which points to Line 1013 of grid_params.py and read as:

if extra_class: # <--- Line 1012
    new[:-1, yi, xi] = veglib_dict[lib_var][:, np.newaxis] # <--- Line 1013

Here is my script:

import pandas as pd
from tonic.models.vic.grid_params import soil, snow, veg, veg_class, Cols, Desc, write_netcdf, grid_params, calc_grid
# Read the soil parameters
soil_dict = soil(r'livenh\vic.nldas.mexico.soil.txt', c=Cols(nlayers=3))


# Read the snow parameters
snow_dict = snow(r'livenh\vic.nldas.mexico.snow.txt.L13',
                 soil_dict, c=Cols(snow_bands=5))


# Read the veg parameter file
veg_dict = veg(r'livenh\vic.nldas.mexico.veg.txt',
               soil_dict,
               vegparam_lai=True,
               veg_classes=11)


# Read the veg library file
veg_lib = veg_class(r'livenh\LDAS_veg_lib')


# Determine the grid shape
target_grid, target_attrs = calc_grid(soil_dict['lats'], soil_dict['lons'])


# Grid all the parameters
grid_dict = grid_params(soil_dict, target_grid, version_in = '4.1.2',
                        veg_dict=veg_dict, veglib_dict=veg_lib, snow_dict=snow_dict,
                        )


# Write a netCDF file with all the parameters
write_netcdf(r'livenh\livenh.params.vic5.nc', target_attrs,
             target_grid=target_grid,
             soil_grid=grid_dict['soil_dict'],
             snow_grid=grid_dict['snow_dict'],
             veglib_dict=veg_lib,
             veg_grid=grid_dict['veg_dict'],
             version_in='4.1.2')

Could I be missing something in veg?

I would like to use tonic to convert some VIC outputs from ASCII to NetCDF format, but when I run ./vic_utils vic2netcdf config_file.cfg, I get the following key error:

Traceback (most recent call last):
  File "./scripts/vic_utils", line 221, in <module>
    main()
  File "./scripts/vic_utils", line 197, in main
    args.func(args)
  File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 512, in _run
  File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 885, in vic2nc
  File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 448, in nc_add_data_to_array
  File "/home/jschap/miniconda3/envs/py2/lib/python2.7/site-packages/pandas/core/frame.py", line 2139, in __getitem__
    return self._getitem_column(key)
  File "/home/jschap/miniconda3/envs/py2/lib/python2.7/site-packages/pandas/core/frame.py", line 2146, in _getitem_column
    return self._get_item_cache(key)
  File "/home/jschap/miniconda3/envs/py2/lib/python2.7/site-packages/pandas/core/generic.py", line 1842, in _get_item_cache
    values = self._data.get(item)
  File "/home/jschap/miniconda3/envs/py2/lib/python2.7/site-packages/pandas/core/internals.py", line 3843, in get
    loc = self.items.get_loc(item)
  File "/home/jschap/miniconda3/envs/py2/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 2527, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 117, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 139, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1265, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1273, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Precipitation'

I am confused because it looks like "Precipitation" is in fact in the "fields" dictionary. Any ideas why this error is occurring? tonic reports the configuration options like this:

-------------------------------
Configuration File Options
-------------OPTIONS-------------
input_files: /home/jschap/Documents/ISR/data/Tuolumne/FromAtlantic/Tuo6_VIC_Outputs_nohead/fluxes*
input_file_format: ascii
regular_grid: True
out_directory: /home/jschap/Documents/ISR/data/Tuolumne/
memory_mode: original
chunksize: 100
out_file_prefix: tuo_fluxes
out_file_format: NETCDF4
precision: single
start_date: 2006-01-01-00
end_date: 2011-12-31-00
calendar: proleptic_gregorian
time_segment: year
snow_bands: False
veg_tiles: False
soil_layers: 3
Fields: Precipitation, Evaporation, Runoff, Baseflow, Wdew, Soil_liquid, Swnet, Netrad, Evap_canopy, Transpiration, Bare_soil_evap, Sublimation_canop, Sublimation_snow, Aero_resist, Surface_temp, Albedo, Relhumid, In_long, Air_temp, Wind
--------GLOBAL_ATTRIBUTES--------
title: Tuolumne basin simulation using Livneh forcings and parameters
grid: WGS84
--------RUN MODE--------
Memory Mode: original
---------------------------------

The tool needs to be able to understand differences among pre-5.0 VIC versions. As per my email exchange with @jhamman, we can assume that the netcdf files will all be 5.0 image mode (if different image drivers require different formats, we can handle that later).

I expect that pre-5.0 VIC formatting will cluster around 4.0.x, 4.1.2, and 4.2/5.0. I'll work my way backwards starting from 4.2/5.0 and try to accommodate as many formatting options as I can.

I am trying to convert a netcdf file representing the livenh dataset back to ascii to ensure the conversion from #76 worked.

My Script is:

from tonic.models.vic.ncparam2ascii import subset

param_file =   r'livenh\livenh.params.vic5.nc'
veg_export  =  r"livenh\output\veg.txt"
snow_export =  r"livenh\output\snow.txt"
soil_export =  r"livenh\output\soil.txt"


subset(param_file, soil_file = soil_export,
       snow_file = snow_export, veg_file = veg_export,
       )

It appeared to complete the veg file export, but failed on the snow export.

Exception has occurred: AttributeError
module 'tonic.models.vic.grid_params' has no attribute 'cols'
  File "C:\workspace\git_clones\tonic\tonic\models\vic\ncparam2ascii.py", line 328, in snow
    c = grid_params.cols(snow_bands=snow_bands)
  File "C:\workspace\git_clones\tonic\tonic\models\vic\ncparam2ascii.py", line 59, in subset
    snow(data, xinds, yinds, snow_file)
  File "C:\workspace\git_clones\tonic\tests\levenh_to_ascii.py", line 9, in <module>
    subset(param_file, soil_file = soil_export,

grid_params appears to have the attribute Cols as apposed to cols.

After inspecting the output veg file, I see this in the output:

450000 2
6 0.828125 0.1 0.1 1.0 0.65 1.0 0.25 
1.68 1.52 1.68 2.9 4.9 5.0 5.0 4.6 3.44 3.04 2.16 2.0 
7 0.171875 0.1 0.1 1.0 0.65 1.0 0.25 
2.0 2.25 2.95 3.85 3.75 3.5 3.55 3.2 3.3 2.85 2.6 2.2 

And this is from the Livenh input file I started with:

450000 2
6 0.828125 0.10 0.10 1.00 0.65 1.00 0.25 
0.175 0.238 0.275 0.275 0.425 0.825 1.175 1.300 0.837 0.350 0.200 0.175 
7 0.171875 0.10 0.10 1.00 0.65 1.00 0.25 
0.487 0.500 0.213 0.213 0.400 0.875 1.525 1.562 0.775 0.363 0.225 0.350 

I believe the mistaken lines represent monthly LAI numbers. Why are they different?

Add conversion of netcdf parameter and state files to ascii (classic).

This separates the netcdf-ascii conversion from the ascii-netcdf conversion in issues #29, #31, #34, #35, and #37, and collects it all into a single issue.

So, issues #29, #31, and #34 can be closed now.

snow17.py generate a non-zero outflow even when there is no precip or snowmelt. In the screenshot below, cumulative outflow (after all snow has melted) exceeds cumulative precip:

I noticed that when I comment out the w_q term in line 234 (see screenshot below), I no longer see the discrepancy between cumulative outflow and cumulative precip.

Here's a plot of the results after commenting out w_q in line 234:

Commenting out w_q in line 234 appears to have resolved the issue in this particular case, but I don't know if it's a sound solution. Somewhere in the code, it seems there must be an implementation error, because the unedited snow17.py code generates a non-zero outflow in the absence of precip and snowmelt.

Add support for lake parameters.

The VIC2nc converter doesn't behave in a very friendly way if we don't have the regular_grid and DOMAIN sections of the config file set up properly.

The two pertinent sections of the config file are below:

# input grid type (if False, Read from domain section below)
# valid values: True, False
regular_grid: True

and

# Grid File (domain file): uncomment if grid is irregular or you want to use a domain file
[DOMAIN]
filename: /Users/jhamman/Dropbox/RASM_Joe/routing_inputs/pour_points/domain.lnd.wr50a_ar9v4.100920.nc
longitude_var: xc
latitude_var: yc
y_x_dims: nj, ni
# Non-coordinate variables to include in each netcdf
copy_vars: frac, mask, area, xv, yv

If you set regular_grid: True and comment out the DOMAIN section the script runs fine. If you set regular_grid: False and leave in the DOMAIN section the script also runs fine. But if you set regular_grid: True and leave in the DOMAIN section the script spits out this error:

Traceback (most recent call last):
 File "/home/mbapt/anaconda/envs/tonic/bin/vic_utils", line 4, in <module>
   __import__('pkg_resources').run_script('tonic===0.0.0.dev-c3c957e', 'vic_utils')
 File "/home/mbapt/anaconda/envs/tonic/lib/python2.7/site-packages/setuptools-19.1.1-py2.7.egg/pkg_resources/__init__.py", line 745, in run_script
 File "/home/mbapt/anaconda/envs/tonic/lib/python2.7/site-packages/setuptools-19.1.1-py2.7.egg/pkg_resources/__init__.py", line 1677, in run_script
 File "/home/mbapt/anaconda/envs/tonic/lib/python2.7/site-packages/tonic-0.0.0.dev_c3c957e-py2.7.egg/EGG-INFO/scripts/vic_utils", line 221, in <module>

 File "/home/mbapt/anaconda/envs/tonic/lib/python2.7/site-packages/tonic-0.0.0.dev_c3c957e-py2.7.egg/EGG-INFO/scripts/vic_utils", line 197, in main

 File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 505, in _run
 File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 722, in vic2nc
 File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 428, in nc_fields
ValueError: Field DOMAIN missing units                                      attribute

Can we include logic in the code to accommodate the instance in which regular_grid: True and the DOMAIN section is present? Or at least have a more informative error message?

Hello,
I am trying VIC 4.2.d parameter files to NetCDF using Tonic, but I am getting a data-type/formatting error while reading the vegetation parameter file. How can I please fix this issue?

File "~\tonic\models\vic\grid_params.py", line 1503, in veg
    gridcel[cell], nveg[cell] = np.array(line).astype(int)

ValueError: invalid literal for int() with base 10: ''

266566 3
    1 0.524590 0.10 0.05 1.00 0.45 5.00 0.0
      5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175
    6 0.311475 0.10 0.10 1.00 0.65 1.00 0.25
      0.087 0.112 0.125 0.263 0.812 2.338 3.425 2.963 2.550 1.600 0.375 0.125
    7 0.163934 0.10 0.10 1.00 0.65 1.00 0.25
      0.275 0.300 0.350 0.650 1.112 1.900 2.050 1.775 1.262 0.625 0.388 0.312
266568 3
    1 0.625000 0.10 0.05 1.00 0.45 5.00 0.50
      5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175 5.175
    6 0.296875 0.10 0.10 1.00 0.65 1.00 0.25
      0.087 0.112 0.125 0.263 0.812 2.338 3.425 2.963 2.550 1.600 0.375 0.125
    7 0.078125 0.10 0.10 1.00 0.65 1.00 0.25
      0.275 0.300 0.350 0.650 1.112 1.900 2.050 1.775 1.262 0.625 0.388 0.312

subset_vegetation.params.txt

I used the "standard" option and ran into an issue. If I tried to include soil moisture (4-d variable) in the configuration file I got the following error when the code tries to write to the netcdf files after it loads all the files in the current chunk:

Traceback (most recent call last):
  File "/glade/u/home/anewman/bin/vic_utils", line 5, in <module>
    pkg_resources.run_script('tonic==0.0.0.dev-2bf5167', 'vic_utils')
  File "/glade/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib/python2.7/site-packages/pkg_resources.py", line 534, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/glade/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib/python2.7/site-packages/pkg_resources.py", line 1441, in run_script
    exec(script_code, namespace, namespace)
  File "/glade/u/home/anewman/lib/python2.7/site-packages/tonic-0.0.0.dev_2bf5167-py2.7.egg/EGG-INFO/scripts/vic_utils", line 221, in <module>

  File "/glade/u/home/anewman/lib/python2.7/site-packages/tonic-0.0.0.dev_2bf5167-py2.7.egg/EGG-INFO/scripts/vic_utils", line 197, in main

  File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 546, in _run
  File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 896, in vic2nc
  File "build/bdist.linux-x86_64/egg/tonic/models/vic/vic2netcdf.py", line 459, in nc_add_data_standard
  File "netCDF4.pyx", line 3267, in netCDF4.Variable.__setitem__ (netCDF4.c:39658)
ValueError: total size of new array must be unchanged

I traced it back to line ~448:

self.f.variables[name][:, i, ys, xs] 

is looking for something that is 2-dimensional while

p.df[sn].values[self.slice]

is only 1 dimensional with a length set at the number of time steps going into the current netcdf file. If I removed soil moisture in the configuration file, I got this option to output properly, so it was an issue with 4-d variables.

I then made some modifications to the code and got it to work for 4-d variables. This is really my first halfway serious go with python, so my syntactical understanding is limited, lots of potential for me to have messed the fix up in some fashion.

I ran the code a bunch and it worked fine. It seemed a little slow, but there is lots of I/O both in and out so I didn't think much of it. Then I got an email from our supercomputer system administration folks stating that my code was performing an excessive amount of disk writes to the same location. They reported that the read rates were fine, but the output was many times the input. That makes me think I fixed the code in an improper fashion so the netcdf writes are occurring an excessive number of times...

The changes are in the function: nc_add_data_standard. What is the best way for me to post my "fixed" code?

Cheers,
Andy