jgcri / tethys

Spatiotemporal downscaling model for global water use

Tethys

Demand for water is often modeled as part of integrated human-Earth systems models, such as GCAM, which operate at coarse spatiotemporal scales in order to model long-term global economic linkages. tethys was designed to downscale projections from such models, producing high-resolution water demand data consistent with global dynamics. This allows coupling with gridded hydrology models, which represent physical processes on much finer scales.

Documentation

Check out Getting Started for installation and basic usage. Details on methodology and advanced usage are available on the User Guide.

Installation

Install tethys using pip:

pip install tethys-downscale

Contributing to Tethys

Whether you find a typo in the documentation, find a bug, or want to develop functionality that you think will make Tethys more robust, you are welcome to contribute! See the full contribution guidelines in our online documentation.

tethys's Issues

netCDF outputs are badly formatted

Here is the header for one of the netCDF output files:

WE27755% ncdump -h twddom.nc 
netcdf twddom {
dimensions:
	index = 67420 ;
	month = 72 ;
variables:
	float data(index, month) ;
		data :units = "km3/month" ;
		data :description = "Downscaled Results: 67420 rows, 72 columns (months)" ;
	int ID(index) ;
	float lon(index) ;
	float lat(index) ;
	int ilon(index) ;
	int ilat(index) ;
	int mapindex(index) ;
	int months(month) ;
}

I suppose that this is technically correct, but it's not all that useful. What we should be writing out is a georeferenced array with dimensions of lat, lon, time (with NaN values in the ocean grid cells). I don't particularly object to including the flattened array, but most people using netCDF are going to want to use those netCDF files with tools that are expecting results in geographical coordinates.

Regarding temporal downscaling of irrigation water use

I'm trying to run the package to get monthly irrigation water use. I used Demeter output for spatial downscaling so right now I have annual water withdrawals/consumption for each crop included on a grid scale. I understand that the temporal downscaling uses a proxy that determines the share of irrigation for each month on a grid scale but I can't find this input file in the example online. My questions are:

1. are these files included with the example file package?

2. Does the current structure of Tethys temporally downscale irrigation water for each crop separately since that is the output from the spatial downscaling?

Thanks!

JOSS Review: Missing item under documentation section for how to get support

Hi there,

I'm one of the reviewers for the JOSS paper on this package - openjournals/joss-reviews#5855. I've encountered one item that appears incomplete:

Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

I don't see in the README or documentation any notes for where someone would seek support. If this already exists, can you point me to it, and otherwise, I'd consider this a minor modification that needs addressing before I can check that item off.

Thank you!

Population maps stop at 2030

Hi .. I noticed that the files attached with the example online had gridded population .tiff maps for 2010,2020,2030. Wondering if you can provide maps for 2040 and 2050 as well ?

The model calls sys.exit() on error

In case of an error, we should be throwing an exception, preferably one that we define. Calling sys.exit() causes the entire python session to end, possibly forcing the user to lose unsaved work. Don't ever do this!

How to modify the configuration file to run global water withdrawals by sector?

Hello, I would like to run GCAM for global water withdrawals by sector data. How can I modify the configuration information? I have made modifications to config_example.yml, but I have encountered an issue with how to write the proxy_files section. There are 6 Water Demand Sectors in this data. How should I specify the tif files for these sectors? Below is my configuration file.
Thank you very much!!!

---

Basic example config file for Tethys

Project level settings
years: [1990,2005,2010,2015,2020,2025,2030,2035,2040,2045,2050,2055,2060,2065,2070,2075,2080,2085,2090,2095,2100]
resolution: 0.125
demand_type: withdrawals

output settings
output_file: output/example_output.nc

csv input
csv: data/example_data.csv

dictionary of proxy files and promised variables and years
proxy_files:
data/population/ssp2_{year}.tif:
variables: Population
years: [2010, 2020, 2030]

mapping of inputs sectors to proxies
downscaling_rules:
water_td_elec_W: Population
water_td_an_W: Population
water_td_irr: Population
water_td_ind_W: Population
water_td_muni_W: Population
water_td_pri_W: Population

list of map files
map_files:

• data/maps/regions.tif

Setup on PyPi

Make installable via pip under the name tethys-downscale

regrid (lines 151-160) in gridded.py may have issues

@crvernon @hassaniazi

Please note that Mengqi & I were trying to use the regrid part of this code for the Building Energy Demand (BED) model and were running into issues when testing line 151-160 in gridded.py. We were performing a manual check on how this chunk was re-gridding examples co-ordinates and we thought it was not working as expected. Someone, should just double check this code again.

tethys/tethys/datareader/gridded.py

Line 151 in 933d117

if method == 'label':

if method == 'label':
    ds = ds.isel(lon=np.arange(ds.lon.size).repeat(r)).coarsen(lon=s).max()
    ds = ds.isel(lat=np.arange(ds.lat.size).repeat(r)).coarsen(lat=s).max()
else:
    ds = ds.isel(lon=np.arange(ds.lon.size).repeat(r)).coarsen(lon=s).sum()
    ds = ds.isel(lat=np.arange(ds.lat.size).repeat(r)).coarsen(lat=s).sum()
    if method == 'extensive':
        ds = ds / (r * r)  # preserve original sum
    elif method == 'intensive':
        ds = ds / (s * s)  # take average

Running Tethys causes output not to show in jupyter notebooks

Prior to running Tethys, output goes to the notebook. The "Starting Tethys..." output also goes to the notebook, but subsequent output goes to the console, including anything you execute after Tethys has completed.

I suspect the problem is related to this item on Stack Overflow:
https://stackoverflow.com/questions/25494182/print-not-showing-in-ipython-notebook-python

Eliminate use of exec in code base.

Nothing we're doing should require using exec. Most of these seem to be replaceable with getattr.

Example:

tethys/tethys/DataReader/IniReader.py

Line 129 in a5ff0d0

exec("ifn = IsFile(settings." + s + ")")

Remove unnecessary execute permissions on files.

Remove unnecessary execute permissions on files. No change to contents. Apply fix to all files.

Remove unnecessary overhead files.

Remove GCAM database example files for AEZ related databases and other non-essential data to lower the installation footprint.

Update all functions to remove ambiguous arguments

Scan through the entire code and replace ambiguous arguments which pass an entire dictionary such as "settings" to different functions.

• Replace settings.x through entire code with explicit arguments.

Installation issues for v1.3

Dear developers,

During installation process and trying to follow the example.py, I found several issues which I think is useful to share:

1. In the version 1.3 of the code source:

• The setup.py reference to v1.2 instead of v1.3;

• The package gcam_reader is required (specifically version 1.0.0), but not specified in the requirements;

• Java is required to use gcam_reader (conda install openjdk)

In the end, I wasn't able to have a working installation.
I stopped debugging at the following error:
File "/home/saba/anaconda3/envs/tethys/lib/python3.8/site-packages/tethys-1.3.0-py3.8.egg/tethys/DataReader/GCAMOutputs.py", line 448, in land_to_array df = df.groupby(['region', 'subreg', 'crop', 'use', 'Year']).sum(axis=1) TypeError: sum() got an unexpected keyword argument 'axis

I believe it comes from some package version issues.

At the same time, I installed the packages according to the requirements (attached conda list output file, tethys_envs.txt), so I don't really know how to advance here.

Please let me know if you have some ideas how to solve this issue.

Best regards,
Sabin

JOSS Review: Unable to Download Supplement Data per Instructions

Hi there,

I found another issue as part of my JOSS review (openjournals/joss-reviews#5855) related to getting the example data (Python 3.10, Windows 11 - I can print out my environment contents if you'd like, but I don't think it's related):

(base) (tethys) PS C:\Users\nickrsan\CodeLocal> python
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tethys
>>> tethys.get_example_data()
Downloading example data for Tethys version 2.0.0. This might take a few minutes.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "environments\tethys\lib\site-packages\tethys-2.0.0-py3.10.egg\tethys\utils\install_supplement.py", line 72, in get_example_data
    zen.fetch_zenodo()
  File "environments\tethys\lib\site-packages\tethys-2.0.0-py3.10.egg\tethys\utils\install_supplement.py", line 47, in fetch_zenodo
    with zipfile.ZipFile(BytesIO(r.content)) as zipped:
  File "\Python\Python310\lib\zipfile.py", line 1267, in __init__
    self._RealGetContents()
  File "Python\Python310\lib\zipfile.py", line 1334, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

It takes quite a bit, even on a good connection before the error shows up, and I see the RAM consumption and network activity indicating the download is proceeding. I spent a bit of time debugging it, but the size of the response object in this case prevents debugger introspection on my machine -I suspect that simply writing the file to disk first would work around that, and maybe the root cause.

Happy to provide any additional useful information. Thanks!

Should the output from running the example scenario match the output committed in the repo?

Most of the output files from running the example scenario match, but a few don't.

WE27755% for file in *.nc; do; echo $file; diff $file ../Test001; done    
twddom.nc
twdelec.nc
twdirr.nc
twdliv.nc
Binary files twdliv.nc and ../Test001/twdliv.nc differ
twdmfg.nc
twdmin.nc
wddom.nc
wdelec.nc
wdirr.nc
wdliv.nc
Binary files wdliv.nc and ../Test001/wdliv.nc differ
wdmfg.nc
Binary files wdmfg.nc and ../Test001/wdmfg.nc differ
wdmin.nc
wdnonag.nc
wdtotal.nc
Binary files wdtotal.nc and ../Test001/wdtotal.nc differ

Is this expected?

随机森林降尺度问题

最近在用随机森林算法对地下水降尺度，应用结果在空间表现上不太好，作出的图显示有条带化的现象，请问有人知道是什么原因吗？

DataParser suppresses IOError exceptions

Here is one example of many:

tethys/tethys/Utils/DataParser.py

Lines 24 to 25 in 823b3be

	except IOError:
	pass

Are we sure this is ok? Will we be able to complete the calculation if one of these operations fails? At the very least we should have some comments that explain why we can continue on as if nothing has happened when we fail to read the data we were looking for. In this case the exception handler was the last thing in the function, so if there is an exception it returns None instead of the data that was expected, which seems like it is bound to cause problems.

Not clear how to install "example" for version 1.3

Dear developer,

After installation of version 1.3 the examples are not provided, nor a reference .ini file.
It seems that the example file should be uploaded from zenodo, but it is not clear how.
Should we use the "example" from the v1.2? Some guiding will be much appreciated.

Cheers,
Sabin

Bug - GMIA Cropland file units

The units for the 'tethys/example/Input/harmonized_inputs/GMIA_cropland.csv' file are in hectares NOT km2 as expected. This does not effect outputs since these values are converted to grid cell fraction. Bug found by @Xinya-Li .

Expected outcome: create new GMIA_cropland file in the correct units of square kilometers.

Quickstarter Notebook

Setup a quickstarter notebook

Allow Tethys to run with either python 2 or python 3.

Allow Tethys to run with either python 2 or python 3. Add warning for Python 2 users that new development will only be in Python 3.

Use Demeter irrigated land fraction by default

Use Demeter’s outputs for irrigated and rainfed cropland for years 2010-2100 (5-yr intervals) for each of the 67,420 0.5 degree grid cells. This will replace having to apply the static fractions for 2005 that are currently in Tethys to all GCAM years.

We need to:

• Add this feature as the default option; keep existing method as an option
• Add functionality to process Demeter output into the format expected by Tethys
• Test against original

test kanban issue

test issue. Complete the following tasks:

• task 1
• task 2