climada-project / climada_python Goto Github PK

Change from concurrent.futures to pathos framework to deal with parallelisation of no pickable functions.

When running under Windows there seems to be a problem between rasterio and the demo file used in the test:

======================================================================
ERROR: test_load_EU_all (climada.hazard.test.test_crop_potential.TestCropPotential)
Test defining crop potential hazard from complete demo file (Central Europe)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "rasterio\_base.pyx", line 213, in rasterio._base.DatasetBase.__init__
  File "rasterio\_shim.pyx", line 64, in rasterio._shim.open_dataset
  File "rasterio\_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: 'data\demo\lpjml_ipsl-cm5a-lr_ewembi_historical_2005soc_co2_yield-whe-noirr_annual_FR_DE_DEMO_1861_2005.nc' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "climada\hazard\test\test_crop_potential.py", line 38, in test_load_EU_all
    soc='2005soc', co2='co2', crop='whe', irr='noirr', fn_str_var=FN_STR_DEMO)
  File "climada\hazard\crop_potential.py", line 185, in set_from_single_run
    geometry=list([shapely.geometry.box(bbox[0], bbox[1], bbox[2], bbox[3])]))
  File "climada\hazard\base.py", line 253, in set_raster
    geometry, dst_crs, transform, width, height, resampling))
  File "climada\hazard\centroids\centr.py", line 218, in set_raster_file
    height, resampling)
  File "climada\util\coordinates.py", line 456, in read_raster
    with rasterio.open(file_name, 'r') as src:
  File "C:\...\Anaconda3\envs\env_climada\lib\site-packages\rasterio\env.py", line 423, in wrapper
    return f(*args, **kwds)
  File "C:\...\Anaconda3\envs\env_climada\lib\site-packages\rasterio\__init__.py", line 216, in open
    s = DatasetReader(path, driver=driver, **kwargs)
  File "rasterio\_base.pyx", line 215, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: 'data\demo\lpjml_ipsl-cm5a-lr_ewembi_historical_2005soc_co2_yield-whe-noirr_annual_FR_DE_DEMO_1861_2005.nc' not recognized as a supported file format.

unit test fails in develop
climada.entity.exposures.test.test_black_marble.TestCountryIso.test_bolivia_pass
climada.entity.exposures.test.test_black_marble.TestCountryIso.test_che_kos_pass
climada.entity.exposures.test.test_black_marble.TestCountryIso.test_haiti_pass
climada.entity.exposures.test.test_black_marble.TestCountryIso.test_korea_pass
climada.entity.exposures.test.test_black_marble.TestCountryIso.test_wrong_fail
climada.entity.exposures.test.test_black_marble.TestProvinces.test_country_iso_geom_pass

This is extracted from the discussion in #48 (sorry for retagging everyone). The lengthy part about large files became off topic, so I decided to move it here.

The repository size is currently around ~580 MiB (using git count-objects -vH). Cloning this repository therefore takes considerable amounts of time. However, an analysis of large binary blobs in the git history suggests that we can easily save more than 80% of the space by rewriting git history (using https://github.com/newren/git-filter-repo).

List of largest files in repository's history

For reference, the list of the 100 largest files in the repository's history (command line from https://stackoverflow.com/questions/10622179/how-to-find-identify-large-commits-in-git-history):

Object hash   Size   Path
------------------------------------------------------------------------------------
afdf9d250244  1,5MiB data/supplychain/USA_services_XXX
7b0bde40f2d8  1,5MiB doc/tutorial/climada_entity_Exposures.ipynb
7b5cadab5cfa  1,6MiB doc/tutorial/climada_entity_Exposures.ipynb
c49fc7111e0d  1,6MiB doc/tutorial/climada_entity_Exposures.ipynb
71a04b0332ed  1,6MiB doc/tutorial/climada_entity_Exposures.ipynb
7771f3263d57  1,6MiB doc/tutorial/climada_entity_Exposures.ipynb
2f7e0ca73836  1,6MiB script/tutorial/1_main_climada.ipynb
607362854358  1,6MiB doc/tutorial/climada_entity_LitPop.ipynb
c90a0f3284d9  1,6MiB data/supplychain/FRA_agriculture_XXX
7a2a1295d6b6  1,6MiB doc/tutorial/climada_entity_LitPop.ipynb
358001e6d875  1,6MiB doc/tutorial/climada_entity_LitPop.ipynb
805f29c38ebc  1,6MiB doc/tutorial/climada_entity_LitPop.ipynb
7a9e5af40035  1,6MiB doc/tutorial/climada_entity_LitPop.ipynb
ba2fb96b6ee3  1,6MiB doc/tutorial/climada_entity_LitPop.ipynb
9b738e384042  1,6MiB doc/tutorial/climada_entity_LitPop.ipynb
04647bd8b895  1,6MiB doc/tutorial/climada_entity_LitPop.ipynb
bdef60960e28  1,6MiB doc/tutorial/climada_entity_LitPop.ipynb
9a3fa22242eb  1,7MiB doc/tutorial/climada_entity_LitPop.ipynb
dc26c1560790  1,7MiB data/supplychain/ROW_agriculture_XXX
b78a959c7569  1,7MiB doc/tutorial/climada_entity_LitPop.ipynb
56008533bd4a  1,7MiB data/demo/California_firms_2016_viirs.csv
74c6286c0d8f  1,7MiB data/supplychain/AUS_agriculture_XXX
9b16d9f4c6f8  1,7MiB script/workshop_I/Hands-on_I.ipynb
1ccabe803c86  1,8MiB data/supplychain/TUR_agriculture_XXX
bc03576dc17f  1,8MiB script/tutorial/climada_entity_Exposures.ipynb
c95ded79ebab  1,8MiB doc/tutorial/climada_entity_Exposures.ipynb
944049d01093  1,8MiB data/supplychain/CAN_agriculture_XXX
2807d06acd33  1,9MiB data/system/NatID_grid_0150as.nc
05b8b073fa47  2,0MiB doc/tutorial/climada_entity_BlackMarble.ipynb
3e361005ea56  2,0MiB doc/tutorial/climada_entity_BlackMarble.ipynb
4048178c77d3  2,0MiB doc/tutorial/climada_entity_BlackMarble.ipynb
d3ab436b04f1  2,0MiB data/supplychain/MEX_agriculture_XXX
7a6c555d6760  2,1MiB script/tutorial/climada_entity_LitPop.ipynb
2f9cdd203bd9  2,1MiB script/tutorial/climada_entity_LitPop.ipynb
ae226ed99cff  2,1MiB climada/test/data/buildings_47_8.dbf
52a3b730ef46  2,1MiB script/tutorial/climada_entity_LitPop.ipynb
c9dd435c0d1e  2,1MiB doc/tutorial/climada_entity_LitPop.ipynb
ab7e168310ef  2,1MiB script/tutorial/climada_entity_LitPop.ipynb
82ba67d2d905  2,1MiB script/tutorial/climada_entity_LitPop.ipynb
c1d248a585bd  2,2MiB data/demo/h08_gfdl-esm2m_ewembi_historical_histsoc_co2_dis_global_daily_DEMO_FR_2001_2003.nc
1953d6fe0d8c  2,3MiB data/supplychain/CAN_services_XXX
b40a3cf5ccd9  2,4MiB data/supplychain/IDN_agriculture_XXX
b7ea0f4a34c5  2,6MiB data/system/GDP2Asset_converter_2.5arcmin.nc
febc69be140c  2,7MiB script/tutorial/5_blackmarble.ipynb
f881dc1edee9  2,7MiB script/tutorial/climada_entity_LitPop.ipynb
1170d76eb438  2,7MiB script/tutorial/climada_entity_LitPop.ipynb
acd637415a02  2,7MiB script/tutorial/climada_entity_LitPop.ipynb
dd1190329814  2,7MiB script/tutorial/climada_entity_LitPop.ipynb
b920c0a578ef  2,7MiB script/tutorial/climada_entity_LitPop.ipynb
d29e8ea548a2  2,8MiB doc/tutorial/climada_entity_BlackMarble.ipynb
aaefe1ae6818  2,8MiB script/tutorial/climada_entity_BlackMarble.ipynb
23756ddc625a  2,8MiB script/tutorial/climada_entity_BlackMarble.ipynb
32b12fe30ba5  2,8MiB script/tutorial/climada_entity_BlackMarble.ipynb
cc64db7d153d  3,1MiB climada/engine/test/data/supplychain/test_hazard
b6ac52e0df52  3,1MiB script/tutorial/5_blackmarble.ipynb
2ca5f50c405e  3,4MiB data/demo/earth_engine/landcover.tif
cfc1110ac778  3,4MiB data/supplychain/IND_agriculture_XXX
3389da6ed038  3,7MiB data/demo/h08_gfdl-esm2m_ewembi_historical_histsoc_co2_dis_global_daily_DEMO_FR_2001_2005.nc
68c0e2ec19be  3,8MiB script/applications/eca_san_salvador/San_Salvador_Risk.ipynb
8416b7843ecf  3,9MiB data/demo/tc_fl_1975_2011.h5
9c72e4b14e59  4,0MiB data/supplychain/RUS_services_XXX
4a4f2ae421b8  4,3MiB data/supplychain/RUS_agriculture_XXX
3758d01539d9  4,5MiB data/demo/WS_ERA40.mat
03ecae28d707  4,8MiB data/supplychain/BRA_agriculture_XXX
e528b62cb0bc  4,8MiB script/applications/eca_san_salvador/San_Salvador_Risk.ipynb
ab730b2e6070  5,3MiB script/applications/eca_san_salvador/San_Salvador_Risk.ipynb
2304b65ad0ba  5,3MiB script/applications/eca_san_salvador/San_Salvador_Risk.ipynb
1a693d23e8df  5,4MiB data/supplychain/CHN_agriculture_XXX
799b5f35455a  5,5MiB script/applications/eca_san_salvador/San_Salvador_Risk.ipynb
835ed0d1e415  5,5MiB script/applications/eca_san_salvador/San_Salvador_Risk.ipynb
0efdda0ef036  5,7MiB data/supplychain/USA_agriculture_XXX
4fe57575bcd9  5,8MiB script/applications/eca_san_salvador/San_Salvador_Risk.ipynb
9ce0afd9b4f4  5,9MiB data/demo/flddph_WaterGAP2_miroc5_historical_flopros_gev_picontrol_2000_0.1.nc
1e54fca74b5f  6,2MiB data/demo/fldfrc_WaterGAP2_miroc5_historical_flopros_gev_picontrol_2000_0.1.nc
f3aed133427c  6,9MiB script/applications/eca_san_salvador/San_Salvador_Risk-Copy1.ipynb
6719ca15aee3  7,1MiB data/demo/atl_prob.mat
7f603961260f  7,6MiB climada/hazard/test/data/test_global_landslide_nowcast_20190508.tif
589b0ef33943  7,7MiB climada/hazard/test/data/test_global_landslide_nowcast_20190501.tif
2675811213d0  7,8MiB climada/hazard/test/data/test_global_landslide_nowcast_20190509.tif
b0c1cd004b06  8,1MiB climada/hazard/test/data/cropping_test_LS.tif
cee69928a590  9,0MiB dist/climada-0.0.1.tar.gz
ca4c7071c104   10MiB climada/hazard/test/data/Victoria_firms.csv
a64e0f3e5c24   13MiB data/F101992.v4b_web.stable_lights.avg_vis.tif.gz
b9e8e61a2eec   14MiB climada/test/data/system/admin0.mat
7390384f5085   15MiB data/system/ls_pr_NGI_UNEP/ls_pr.tif
1678be6b232f   15MiB climada/test/data/GLB_NatID_grid_0360as_adv_1.mat
0fa15d7d60ef   15MiB climada/test/data/GLB_NatID_grid_0360as_adv_2.mat
2a1a77be0dc8   15MiB data/system/GLB_NatID_grid_0360as_adv_2.mat
7bc480bd96d2   16MiB data/F152007.v4b_web.stable_lights.avg_vis.tif.gz
93c8b4eb9131   16MiB data/supplychain/ROW_services_XXX
3109d2e4bf3b   17MiB data/F162007.v4b_web.stable_lights.avg_vis.tif.gz
2a5a60d34a27   17MiB data/demo/earth_engine/rgb_zurich.tif
8b8bc642dfa6   18MiB data/F182012.v4c_web.stable_lights.avg_vis.tif.gz
439e36b6bb9a   24MiB data/system/IBTrACS.ALL.v04r00.nc
0bd3d429871b   25MiB data/system/ECLIPSE_base_CLE_V5a_NOx.nc
5e8e2e083e95   31MiB climada/hazard/test/data/nasa_global_landslide_catalog_point.dbf
864876f19e62   41MiB data/demo/gdp2asset_demo_exposure.nc
3026099e8d97   51MiB data/supplychain/GLB_agriculture_XXX
090d7bb7444a   82MiB data/supplychain/WIOT2014_Nov16_ROW.xlsx
7f0d4a40ca5d   91MiB data/supplychain/sup_ib_hazard_default

Obsolete files and folders

There are whole folders that can be safely removed recursively using git filter-repo --invert-paths --path <path>:

script/tutorial/
data/supplychain/
script/workshop_I/

Furthermore, there is a list of single blobs that are obsolete. They and all files with the same base names can be removed from the git history using git filter-repo --invert-paths --path-glob */<basename>.

data/system/ECLIPSE_base_CLE_V5a_NOx.nc
doc/tutorial/climada_hazard_RelativeCropyield_entity_CropProduction.ipynb
script/applications/eca_san_salvador/San_Salvador_Risk-Copy1.ipynb
climada/hazard/test/data/cropping_test_LS.tif
dist/climada-0.0.1.tar.gz
data/F101992.v4b_web.stable_lights.avg_vis.tif.gz
climada/test/data/system/admin0.mat
climada/test/data/GLB_NatID_grid_0360as_adv_1.mat
data/F152007.v4b_web.stable_lights.avg_vis.tif.gz
data/F162007.v4b_web.stable_lights.avg_vis.tif.gz
data/F182012.v4c_web.stable_lights.avg_vis.tif.gz
data/demo/gdp2asset_demo_exposure.nc
data/demo/tc_fl_1975_2011.h5
data/demo/atl_prob.mat
data/demo/WS_ERA40.mat
data/demo/h08_gfdl-esm2m_ewembi_historical_histsoc_co2_dis_global_daily_DEMO_FR_2001_2005.nc
climada/hazard/test/data/test_global_landslide_nowcast_20190508.tif
climada/hazard/test/data/test_global_landslide_nowcast_20190501.tif
climada/hazard/test/data/test_global_landslide_nowcast_20190509.tif
data/system/ls_pr_NGI_UNEP/ls_pr.tif
data/system/IBTrACS.ALL.v04r00.nc
data/system/WIOT2014_Nov16_ROW.xlsx
climada/hazard/test/data/nasa_global_landslide_catalog_point.dbf
climada/hazard/test/data/01_ClimatologyMonthly_032818_9600x5400_test.tif
climada/hazard/test/data/02_ClimatologyMonthly_032818_9600x5400_test.tif
climada/hazard/test/data/03_ClimatologyMonthly_032818_9600x5400_test.tif
data/demo/earth_engine/landcover.tif
data/demo/earth_engine/rgb_zurich.tif
climada/hazard/test/data/Victoria_firms.csv
data/demo/flddph_WaterGAP2_miroc5_historical_flopros_gev_picontrol_2000_0.1.nc
data/demo/fldfrc_WaterGAP2_miroc5_historical_flopros_gev_picontrol_2000_0.1.nc

Finally, there are files where we will only keep the latest revision because previous versions were too large. The BGF Repo-Cleaner can do exactly this using the --delete-files option. It won't remove the files in the current HEAD.

data/system/GLB_NatID_grid_0360as_adv_2.mat
script/applications/eca_san_salvador/San_Salvador_Risk.ipynb
doc/tutorial/climada_hazard_entity_Crop.ipynb
doc/tutorial/climada_entity_BlackMarble.ipynb
doc/tutorial/climada_entity_LitPop.ipynb
doc/tutorial/climada_entity_Exposures.ipynb
doc/tutorial/climada_entity_TropCyclone.ipynb

Large files in active branches

All files larger than 3 MiB have been removed from all currently active branches!

Conclusion

Addressing all of the above files with git-filter-repo will bring the repository size down to ~96 MiB which seems acceptable to me. However, permanently removing files from a git history changes a lot of commit hashes. That's why I think we should make one single change that removes large files once and for all (using https://github.com/newren/git-filter-repo). After that, we can implement a much stricter policy for large files and we should be fine in the future.

Since this issue requires changes to be made to all branches including the main branch, fixing it is expected to take at least until the next release in early 2021.

The plotting function util.u_plot.geo_im_from_array(array_sub) returns a numpy array of mpl.geoaxes.GeoAxesSubplot if array_sub is an iterable with two or more elements, otherwise returns a single instance of mpl.geoaxes.GeoAxesSubplot.

This inconsistent behavious leads to a failure of for instance hazard.plot_rp_intensity(return_periods=(25, 50, 100, 250)) if return_period is an iterable with one value, e.g. return_period=[100].

The failure is because the code tries to unpack the return value of util.u_plot.geo_im_from_array(array_sub) in the line _, axis = u_plot.geo_im_from_array(inten_stats, self.centroids.coord,\ colbar_name, title, smooth=smooth, axes=axis, **kwargs).

Three tests fail in develop:
climada.entity.impact_funcs.test.test_imp_fun_set.TestExtend.test_extend_different_extend
climada.entity.impact_funcs.test.test_imp_fun_set.TestExtend.test_extend_equal_same
climada.entity.impact_funcs.test.test_imp_fun_set.TestExtend.test_extend_to_empty_same

Hi all,
I encountered an issue when using the read_raster() function in util/coordinates.py when you want to re-project and set a geometry at the same time. The function then returns the reprojected array without masking the geometry. I programmed a fix but haven't had time to check and push the changes. Just to let you know that there is a fix already if you encounter the same issue.

The module climada.util.earth_engine requires the package ee (Google Earth engine or earthengine-api) which is not in the requirements. Should we add it to the requirements or tell pylint to ignore this particular package?

It was the last "red" pylint error to be resolved in my feature/pylint_warnings branch where I try to remove pylint errors and some whitespace and indentation warnings that can be automated with tools like autopep8. See here: https://ied-wcr-jenkins.ethz.ch/job/climada_branches/job/feature%252Fpylint_warnings/ In the branch, I decided to have pylint ignore the import. Since the package can be used with a Google account only, we shouldn't require it by default I think.

Apparently, the "Build: Passing" badge in README.md is still referring to the old continuous integration builds. I guess @emanuel-schmid may be the one to have a look at this? Maybe we can also have a badge for the develop branch...

running:

tr_ibtracs = TCTracks()
tr_ibtracs.read_ibtracs_netcdf(provider='usa', correct_pres=True, year_range=(2002, 2007))

I get:

AttributeError: 'DataArray' object has no attribute 'str'

at line 232 of tc_tracks.py:

232 years = ds.sid.str.slice(0, 4).astype(int)

it seems this bit of code was changed here

Please join the discussion about CLIMADA modules that need packages/libraries that are not listed in the conda requirements files. Under which circumstances can we accept this? How do we deal with such a situation?

Examples:

• The module climada.util.earth_engine imports ee (provided by earthengine-api), which is not in the requirements files. Only very few users would be interested in using climada.util.earth_engine and it's only useful if you have a Google account. Additionally, installing earthengine-api brings in a lot more requirements (especially a lot of google-* packages). So it seems restrictive to force all users of CLIMADA to install this requirement. On the other hand, when importing climada.util.earth_engine, an ImportError is raised without instructions on how to deal with it. A more user-friendly output is desirable.
• The module climada.util.alpha_shape imports from descartes which is not in the requirements files (it's in setup.py, but not in the conda requirements files). I don't know anything about climada.util.alpha_shape. It's used nowhere else in the CLIMADA code. Anyone?
• The module climada.hazard.tc_surge_geoclaw uses the Fortran library geoclaw and will compile Fortran code at runtime:

  climada_python/climada/hazard/tc_surge_geoclaw.py

  Line 322 in e235135

  with subprocess.Popen(["make", ".output"],

  
  Fortran compilers are only available as OS-dependent packages in Conda, so we can't easily add one to the requirements files. I will make sure that users who import climada.hazard.tc_surge_geoclaw will be informed about the additional requirement.
• The module climada.hazard.tc_surge imports the elevation package that doesn't work with Windows: #9

Questions:

• Do you know more examples? Are you working on a new feature that introduces similar additional requirements?
• Do we want to generally avoid such situations under all circumstances? We could strictly remove such modules from the CLIMADA repository and force them to be placed in an external "add-on" repository. This is how the old MATLAB version of CLIMADA was organized and, from my understanding, this is not desirable for the new Python-based CLIMADA which should instead provide all features in one central place.
• If we tolerate such additional requirements, do we want to formulate a policy under which circumstances this is acceptable?
• If we tolerate such additional requirements, what are conventions when and how to instruct users to install the additional requirements?

Hi Jan,

very nice job! I was having a look to your merge and I have two comments:

• In StormEurope.read_footprints():
  if you have the same centroids for all events, you can consider using the Hazard method _append_all() after having generated the list of new hazards. This will improve performance considerably. As example, this has been implemented in TropCyclone.set_from_tracks().

• Centroids base:

  • I like the new properties and methods! Could you add some tests to its corresponding test file?
  • the resolution setter could set the _resolution parameter without inputs, just computing the distance between the first two consecutive coordinates (for regular grids this does the job). This way it does not look like you are modifying the coordinates to resample them to the provided resolution.

Best,
Gabriela

Plotting hazard intensity returns a TypeError and a ValueError and no figure is displayed.

The two errors are:

TypeError: _pcolorargs() got an unexpected keyword argument 'allmatch'
ValueError: Expected 2-dimensional array, got 1

Below two examples with different hazard types.

Are there any recent changes in climada/util/plot.py (last changed by @tovogt), the requirements (last changed by @emanuel-schmid, @tovogt, and @mmyrte), or anywhere else that could have broken plot_intensity?

PS: plotting functions are unfortunately not thoroughly tested on jenkins currently.
PPS: I tagged it as "blocker" because the plotting functions are quite crucial for students, users, and developers; especially important to run and understand tutorials and check freshly initiated hazard sets.

1. plotting crop hazard (relative_cropyield):

in[0]:

hazard.plot_intensity(event=0)

out[0]:

~/Documents/Projects/climada_python/climada/util/plot.py:312: UserWarning: Tight layout not applied. The left and right margins cannot be made large enough to accommodate all axes decorations. 
  fig.tight_layout()
Traceback (most recent call last):

  File "<ipython-input-7-5c7ab9e6fd57>", line 1, in <module>
    haz1.plot_intensity(event=0)

  File "~/Documents/Projects/climada_python/climada/hazard/base.py", line 719, in plot_intensity
    smooth, axis, **kwargs)

  File "~/Documents/Projects/climada_python/climada/hazard/base.py", line 1134, in _event_plot
    l_title, smooth=smooth, axes=axis, **kwargs)

  File "~/Documents/Projects/climada_python/climada/util/plot.py", line 272, in geo_im_from_array
    axis.pcolormesh(grid_x, grid_y, np.squeeze(grid_im), transform=proj, **kwargs),

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py", line 1459, in pcolormesh
    result = self._pcolormesh_patched(*args, **kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py", line 1491, in _pcolormesh_patched
    X, Y, C = self._pcolorargs('pcolormesh', *args, allmatch=allmatch)

TypeError: _pcolorargs() got an unexpected keyword argument 'allmatch'

/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py:388: MatplotlibDeprecationWarning: 
The 'inframe' parameter of draw() was deprecated in Matplotlib 3.3 and will be removed two minor releases later. Use Axes.redraw_in_frame() instead. If any parameter follows 'inframe', they should be passed as keyword, not positionally.
  inframe=inframe)
/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/feature_artist.py:225: MatplotlibDeprecationWarning: Using a string of single character colors as a color sequence is deprecated since 3.2 and will be removed two minor releases later. Use an explicit list instead.
  **dict(style))
Traceback (most recent call last):

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/IPython/core/formatters.py", line 341, in __call__
    return printer(obj)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/IPython/core/pylabtools.py", line 248, in <lambda>
    png_formatter.for_type(Figure, lambda fig: print_figure(fig, 'png', **kwargs))

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/IPython/core/pylabtools.py", line 132, in print_figure
    fig.canvas.print_figure(bytes_io, **kw)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 2217, in print_figure
    **kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 1639, in wrapper
    return func(*args, **kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 509, in print_png
    FigureCanvasAgg.draw(self)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 407, in draw
    self.figure.draw(self.renderer)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/figure.py", line 1864, in draw
    renderer, self, artists, self.suppressComposite)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
    a.draw(renderer)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py", line 388, in draw
    inframe=inframe)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py", line 411, in wrapper
    return func(*inner_args, **inner_kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/axes/_base.py", line 2748, in draw
    mimage._draw_list_compositing_images(renderer, self, artists)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
    a.draw(renderer)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/feature_artist.py", line 228, in draw
    c.draw(renderer)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/collections.py", line 931, in draw
    Collection.draw(self, renderer)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/collections.py", line 412, in draw
    self._offset_position)

  File "/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 175, in draw_path_collection
    offset_position)

ValueError: Expected 2-dimensional array, got 1

<Figure size 648x936 with 2 Axes>

2. TC hazard (witnessed by @simonameiler ):

in[0]:

hazard_dict[‘Kerry_AL’].plot_intensity(event=i)

out[0]:

/Users/user/Documents/WCR/CLIMADA_develop/climada_python/climada/util/plot.py:314: UserWarning: Tight layout not applied. The left and right margins cannot be made large enough to accommodate all axes decorations.
  fig.tight_layout()
Traceback (most recent call last):

  File “<ipython-input-7-f64a17d65b5d>“, line 1, in <module>
    hazard_dict[‘Kerry_AL’].plot_intensity(event=hazard_dict[‘Kerry_AL’].event_id[impact_dict[‘Kerry_AL’].at_event == impact_dict[‘Kerry_AL’].at_event[idx]][0])

  File “/Users/user/Documents/WCR/CLIMADA_develop/climada_python/climada/hazard/base.py”, line 720, in plot_intensity
    smooth, axis, **kwargs)

  File “/Users/user/Documents/WCR/CLIMADA_develop/climada_python/climada/hazard/base.py”, line 1136, in _event_plot
    l_title, smooth=smooth, axes=axis, **kwargs)

  File “/Users/user/Documents/WCR/CLIMADA_develop/climada_python/climada/util/plot.py”, line 274, in geo_im_from_array
    transform=proj, **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py”, line 1459, in pcolormesh
    result = self._pcolormesh_patched(*args, **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py”, line 1491, in _pcolormesh_patched
    X, Y, C = self._pcolorargs(‘pcolormesh’, *args, allmatch=allmatch)

TypeError: _pcolorargs() got an unexpected keyword argument ‘allmatch’

/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py:388: MatplotlibDeprecationWarning:
The ‘inframe’ parameter of draw() was deprecated in Matplotlib 3.3 and will be removed two minor releases later. Use Axes.redraw_in_frame() instead. If any parameter follows ‘inframe’, they should be passed as keyword, not positionally.
  inframe=inframe)
/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/feature_artist.py:225: MatplotlibDeprecationWarning: Using a string of single character colors as a color sequence is deprecated since 3.2 and will be removed two minor releases later. Use an explicit list instead.
  **dict(style))
Traceback (most recent call last):

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/IPython/core/formatters.py”, line 341, in __call__
    return printer(obj)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/IPython/core/pylabtools.py”, line 248, in <lambda>
    png_formatter.for_type(Figure, lambda fig: print_figure(fig, ‘png’, **kwargs))

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/IPython/core/pylabtools.py”, line 132, in print_figure
    fig.canvas.print_figure(bytes_io, **kw)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backend_bases.py”, line 2217, in print_figure
    **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backend_bases.py”, line 1639, in wrapper
    return func(*args, **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py”, line 509, in print_png
    FigureCanvasAgg.draw(self)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py”, line 407, in draw
    self.figure.draw(self.renderer)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py”, line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/figure.py”, line 1864, in draw
    renderer, self, artists, self.suppressComposite)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/image.py”, line 131, in _draw_list_compositing_images
    a.draw(renderer)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py”, line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py”, line 388, in draw
    inframe=inframe)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py”, line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py”, line 411, in wrapper
    return func(*inner_args, **inner_kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/axes/_base.py”, line 2748, in draw
    mimage._draw_list_compositing_images(renderer, self, artists)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/image.py”, line 131, in _draw_list_compositing_images
    a.draw(renderer)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py”, line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/cartopy/mpl/feature_artist.py”, line 228, in draw
    c.draw(renderer)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py”, line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/collections.py”, line 931, in draw
    Collection.draw(self, renderer)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/artist.py”, line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/collections.py”, line 412, in draw
    self._offset_position)

  File “/Users/user/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py”, line 175, in draw_path_collection
    offset_position)

ValueError: Expected 2-dimensional array, got 1

<Figure size 648x936 with 2 Axes>

Hi everyone

with the Impact class, the EAI (expected annual impact) values at each exposure point is computed saved per default as "eai_exp".

Question:
Do you know of a method to get something like "impact exceeded every N years" per exposure coordinate?
I guess, I need to calculate the full impact matrix by putting save_mat=True(?)
From there on, how would you proceed?

(To clarify, I want to plot a map showing the impact that is expected to be exceeded every 10 or 50 years per exposure coordinate)

This file https://github.com/CLIMADA-project/climada_python/blob/master/LICENSE is a different license than the what the Readme.md states.

Currently in read_ibtracs_netcdf (tc_tracks.py) latitude and longitude are always the values created by IBTrACS (netCDF variables 'lat' and 'lon') by combining values from various providers. I suggest that, when 'provider' is given as an input, those values are instead read from the desired provider (ibtracs_ds[f'{provider}_lat'] and ibtracs_ds[f'{provider}_lon']). On the other hand, if provider is not given then I think it is fine to read from the combined IBTrACS data.

On the develop branch I found the following issue:

Calling read_ibtracs_netcdf with estimate_missing set to True return tracks with misleading data when both pressure and wind speed data are missing.

For instance:

all_tracks = TCTracks()
all_tracks.read_ibtracs_netcdf(provider='usa', basin='SI', year_range=(1850, 1851))

correctly issues a warning:

climada.hazard.tc_tracks - WARNING - No valid wind/pressure values found for 1851080S15062, 1851080S15063, 1851080S21060, 1851080S21420, 1851305S08089.

However:

all_tracks = TCTracks()
all_tracks.read_ibtracs_netcdf(provider='usa', basin='SI', year_range=(1850, 1851), estimate_missing=True)

does not issue any warning, but the tracks that are loaded contain data that don't look correct:

• max_sustained_wind and central_pressure are all -1
• radius_max_wind and radius_oci are all 0.

I think the issue is that _estimate_pressure and _estimate_vmax convert all NaNs to -1 and then the line

ibtracs_ds['valid_t'] &= ibtracs_ds.wind.notnull() & ibtracs_ds.pres.notnull()

does not filter those out.

Is this intended and if so what will be the impact on the wind field?
I would suggest that even if estimate_missing is True, tracks without any pressure or wind information MUST be skipped, but I might be wrong.

Hello!
I have installed shapely module in my computer like this

(climada_env) C:\Users***>pip install shapely
Requirement already satisfied: shapely in e:\anaconda\envs\climada_env\lib\site-packages (1.6.4.post1)

but when I run the tests_install.py file to test installing, the console gave me like this

runfile('D:/climada_python-1.3.1/climada_python-1.3.1/tests_install.py', wdir='D:/climada_python-1.3.1/climada_python-1.3.1')
Reloaded modules: climada.util.constants, climada.util.config
2019-12-09 21:18:23,181 - climada - DEBUG - Loading default config file: D:\climada_python-1.3.1\climada_python-1.3.1\climada\conf\defaults.conf
Traceback (most recent call last):

File "", line 1, in
runfile('D:/climada_python-1.3.1/climada_python-1.3.1/tests_install.py', wdir='D:/climada_python-1.3.1/climada_python-1.3.1')

File "E:\anaconda\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)

File "E:\anaconda\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "D:/climada_python-1.3.1/climada_python-1.3.1/tests_install.py", line 9, in
from climada.util.constants import SOURCE_DIR

File "D:\climada_python-1.3.1\climada_python-1.3.1\climada_init_.py", line 22, in
from .util.config import CONFIG, setup_conf_user, setup_logging, setup_environ

File "D:\climada_python-1.3.1\climada_python-1.3.1\climada\util_init_.py", line 23, in
from .coordinates import *

File "D:\climada_python-1.3.1\climada_python-1.3.1\climada\util\coordinates.py", line 28, in
import shapely.vectorized

File "E:\anaconda\lib\site-packages\shapely\vectorized_init_.py", line 3, in
from ._vectorized import (contains, touches)

ModuleNotFoundError: No module named 'shapely.vectorized._vectorized'

Would it be possible for you to tell me how to deal with this?
Thank you for your attention!

A user just made me aware of the following issue:

In the TCTrack Module:

SAFFIR_SIM_CAT = [34, 64, 83, 96, 113, 135, 1000]

""" Saffir-Simpson Hurricane Wind Scale in kn"""

Should the value 135 be 137? This would make a category 5 storm start at 137 kn instead of 135 kn. This would allign better with https://www.nhc.noaa.gov/aboutsshws.php

Can somebody working with TC check and change if appropriate?

There exists a function for selecting a region of interest by country in the hazard module climada.hazard.flood as a static method of RiverFlood:

As it returns a Centroids object and because its functionality is very general, I suggest moving it to the climada.hazard.centroids module, e.g. as a method of the Centroids class.

Running TCTracks.calc_random_walk the package Numba issues a series of warnings. Can the author (I think it is @tovogt who rewrote the calc_random_walk) please have a look and comment on these warnings?

If the warnings actually mean that Numba is not able to use this code, then either the code must be adapted of Numba removed.

using the log-functionality in plot_hexbin, the colorbar is not labelled correctly.
In the screenshot, the values per pixel actually range from 0 to 16926138554 USD

We started work to update the CLIMADA tutorials and had our first meeting on 29 Sept 2020. This issue summarises the discussion and gives us a place to continue it.

Summary of the Tutorial update session: (thanks @chahank)
A few general points were addressed and summarised in https://etherpad.wikimedia.org/p/CLIMADA-Tutorials . You are all welcome to give your opinion. Furthermore, we distributed the task to update different tutorial notebooks to different team members. We still need more persons that would help.

Not assigned are the notebooks BlackMarble, Exposures_polygons_lines, LitPop, openstreemap, RiverFlood, StormEurope, TropCyclone, drought, emulator, entity_crop, util_earth_engine, MeasureSet. Maybe not all need an update, but if you can take up one (you do NOT need to be an expert) and update it according to the points mentioned in the link above, please feel free to do so.

Please let @chahank know if you want to work on a given tutorial, or comment here. We would like to avoid two person working on the same document.

We agreed to make the updates until Monday 12.10.2020 and meet again on this date at 14:30. Contact @chahank for a meeting invite.
Thanks to all of you!

We're collecting work on the tutorial_update branch. Since most people are only working on one document there won't be many conflicts, so you can ususally push directly to the branch (make sure you're up to date and there aren't conflicts by pulling first).

Currently assigned:

• Main outline tutorial @ChrisFairless
• Exposure @Evelyn-M
• Hazard @simonameiler
• Impact, impact data @chahank
• Impact functions/IFS @manniepmkam
• Discount rates @samluethi

This many does not really qualifies as an issue, as it is related to how python works. There is an easy work-around, but I mention it so that it stays on the radar.

When an object from a class (e.g. Exposures) which inherits from GeoPandas is manipulated with functions from Pandas, the objects losses its CLIMADA class and reverts to a GeoPandas object.

Here is an example with litpop. First we create an exposures from LitPop using the country USA and then setting the administration level 1.

exp = LitPop()
countries_list = ['USA']
exp.set_country(countries_list, admin1_calc = True, res_arcsec=300, reference_year=2014)
exposure_set_admin1(exp, res_arcsec=300)

exp is an Exposure object here.

Second we select only those entries from Florida as

exp1 = exp[exp['admin1'] == 'Florida']

exp1 is now a GeoPandas dataframe.

It still contains all the relevant information, and thus can easily be reconverted into an Exposures by doing

Exposures(exp1)

Hi everyone

Is there a any reasonable way to filter or combine CLIMADA Impact instances at the moment?

My guess is that the Impact would require methods comparable to Hazard.select() and Hazard.append(), right?

Here is an example what I would like to do:

impact = Impact()
impact.calc(...)
impact_subset1 = impact.select(years=[2005,2006,2008,2020,2021])
impact_subset2 = impact.select(event_names=['Petra', 'Antoinette', 'Hugo', 'Sergej'])

and
impact_combined = impact_subset1.append(impact_subset2)

When filtering or combining two impact sets, we would need in any case to recompute the values of aai_agg, tot_value, and frequency, right? Maybe filtering/combining the hazard sets and re-running impact.calc() in a loop is a valid alternative?

For my application, however, I want to filter and re-combine my impact instances several times during analysis without rerunning impact.calc() all the time. Therefore, filtering hazard set beforehand is not my favourite option.

Are there any suggestions other than writing such methods in impact.py?
Would it make sense to create a new module for "impact_manipulation"?

Thanks in advance for your inputs

Hello again,

The plot_hexbin function per default clips the map of an entity tightly to the grid points with data.
For low resolutions, parts of the country are sometimes cut off, as can be seen in the plot below:
Parts of Schaffhausen and eastern Switzerland are cut off.
Of course this can be set manually, but I would suggest a default with a certain buffer (or clipping according to borders instead of grid points for country plots).
What do you think?
(Code below)

from climada.entity.exposures.litpop import LitPop
ent = LitPop()
ent.set_country('CHE', fin_mode='pc', res_arcsec=300, reference_year=2017)
ent.plot_hexbin(pop_name=True, linewidth=7)

When using the 'read_ibtracs_netcdf()' function in the TCTracks CLIMADA module an AttributeError arises. It seems to be connected to the year_range parameter. However, if I drop that, I still get the same error.

Here's my code to read in TC tracks from IBTrACS:

    tracks_IB = TCTracks()
    tracks_IB.read_ibtracs_netcdf(year_range=(1980, 2017), basin=basin, correct_pres=False)
    tracks_IB.equal_timestep(time_step_h=.5)

I get the following Error message and to me, it looks like there’s an issue in the read_ibtracs_netcdf() function.

  File "/Users/simonameiler/Documents/WCR/Scripts/TC_IB_K_G_his_1.py", line 167, in init_tc_tracks_IBTrACS
    tracks_IB.read_ibtracs_netcdf(year_range=(1980, 2017), basin=basin, correct_pres=False)

  File "/Users/simonameiler/Documents/WCR/CLIMADA_develop/climada_python/climada/hazard/tc_tracks.py", line 219, in read_ibtracs_netcdf
    years = ds.sid.str.slice(0, 4).astype(int)

  File "/Users/simonameiler/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/xarray/core/common.py", line 179, in __getattr__
    (type(self).__name__, name))

AttributeError: 'DataArray' object has no attribute 'str'

Any ideas on how to solve this?

Two tests fail in develop:
climada.entity.exposures.test.test_black_marble.TestEconIndices.test_fill_econ_indicators_pass
climada.entity.exposures.test.test_black_marble.TestEconIndices.test_fill_econ_indicators_na_pass

Both issue #46 (util functions) and #37 (config & constants) have referred to the topic of "where do we keep what". If we (or probably just @emanuel-schmid 😀) are going to package climada for distribution, we're going to need to reorganise the repository. According to the official docs, this is the structure a package should follow:

packaging_tutorial
├── LICENSE
├── README.md
├── example_pkg
│   └── __init__.py
├── setup.py
└── tests

I'll use climada_python as the root / from here on. Also, this is just a proposal, forgive my absolute language.

/data, /script, and /doc/tutorial need to move into other github repos.

• I find the climada_papers ill suited for tutorials, so we'd need a new climada_tuts. /script and /data/demo can be moved there as well.
• The /data/system, as discussed in #37, should be created on the user's machine, like e.g. $HOME/.climada; we should probably check the licenses and host it outside a repository.
• The data necessary for testing needs to be radically minimised and put into /tests, along with all the unit tests now living in /climada subdirs.

I think this topic is quite involved and needs a lot of reading up on the formats of packaging formats for pypi/wheel/setuptools and conda/conda-forge/conda-build. I only stuck my toes into it and would be glad to let @emanuel-schmid decide for all of us, because I imagine that none of us has the knowledge or the time to find best practices and implement them.

As noted by @sameberenz (see 1d61fe2), Natural Earth has some Admin 0 regions without numeric identifiers. Among them is Norway, which is easy to solve since it actually has an ISO code. But there are others that are more complicated to deal with like Kosovo (Kosovo is not in ISO 3166).

Any ideas, what to do about these regions? We could define our own codes above 900, since those are officially omitted in ISO 3166.

These are the Natural Earth NAME attributes of the affected regions together with the numeric codes that have been used by me so far:

{
    "Dhekelia": 826,  # UK
    "Somaliland": 706,  # Somalia
    "Norway": 578,  # Norway
    "Kosovo": 983,  # used in iso3166 package
    "USNB Guantanamo Bay": 840,  # USA
    "N. Cyprus": 196,  # Cyprus
    "Cyprus U.N. Buffer Zone": 196,  # Cyprus
    "Siachen Glacier": 356,  # India
    "Baikonur": 398,  # Kazakhstan
    "Akrotiri": 826,  # UK
    "Indian Ocean Ter.": 826,  # UK
    "Coral Sea Is.": 36,  # Australia
    "Spratly Is.": 912,  # ?
    "Clipperton I.": 250,  # France
    "Bajo Nuevo Bank": 170,  # Colombia
    "Serranilla Bank": 170,  # Colombia
    "Scarborough Reef": 156,  # PR China
}

There appears to be a mismatch between the hurricane category setter and the names; my IDE tells me it's only relevant for logging and plotting, but we may be mislabeling Cat1 as mere Tropical Depressions.

Can someone verify this before I fix the indices?

Hi All,

I am having some issues trying to export plots of TropCyclone Class -

I have tried to export write_Hdf5 command but it seems the file is empty.

Is there any option to write a geotiff or hdf5, correctly?

Thank you,
Jose

There is a function called exposure_set_admin1 in litpop.py which allows us to use state-level information to calculate exposures (if available; only for selected countries). I didn't write the function but I'm using it and noticed that the function should be made available for all exposures. I also saw that it is not correctly initiated (self is missing).

What needs to be done with the function:

• place it in the exposures/base.py
• correctly initiate it
• write a unit test

original post from @chahank: "[...] you might have trouble installing spyder in your new environment. This can be solvex either by downgrading the version of spyder, or by installing spyder in another clean environment and specify to spyder to use climada_env as an environment for the ipython console."

Originally posted by @chahank in #68 (comment)

Hi all,
I am struggling to find the best CLIMADA-thonic way of implementing following situation:

I have a hazard on a regular raster and exposure available on NUTS3-level. Instead of remapping my exposure to the hazard, I want to statistically aggregate (e.g. max, mean value) the hazard across corresponding NUTS3 regions and then match with the exposure to determine the impact.

I understand that CLIMADA usually works the opposite way, as everything is adjusted to the hazard. However, I find this not very practical in my case as the impact needs to be defined at NUTS3-level in the end. Any suggestions of how to deal with that?

The optimal way in my eyes would look like that:

1. create a geodataframe from the hazard and use the dissolve function to aggregate the hazard to NUTS3 region. Then calculate the impact based on the NUTS3 regions only.

My (not optimal) ideas for a workaround are these:

1. preprocess hazard data (aggregate to NUTS3 regions) and continue with CLIMADA from that point.
2. create downscaled raster exposure by just populating grid points within a NUTS3 region with identical exposure value. Once the impact is then calculated I have to postprocess this data to aggregate the result to NUTS3 level.

Any suggestions on how to deal with this within CLIMADA greatly appreciated. If nothing like this is presently implemented, happy to discuss pros and cons of making it possible.

Thanks, Tobias

With geopandas 0.6.1 these tests fail:
climada.test.test_osm.TestOpenStreetMapModule.test_get_osmstencil_litpop
climada.test.test_osm.TestOpenStreetMapModule.test_make_osmexposure

both here:

...
    '_' + str(int(min(High_Value_Area_gdf.bounds.minx))) + '.h5')
  File "/var/lib/jenkins/jobs/climada_ci_night/workspace/climada/entity/exposures/base.py", line 457, in write_hdf5
    store.put('exposures', pd.DataFrame(self))
  File "/var/lib/jenkins/.conda/envs/climada_env/lib/python3.7/site-packages/pandas/io/pytables.py", line 889, in put
    self._write_to_group(key, value, append=append, **kwargs)
  File "/var/lib/jenkins/.conda/envs/climada_env/lib/python3.7/site-packages/pandas/io/pytables.py", line 1415, in _write_to_group
    s.write(obj=value, append=append, complib=complib, **kwargs)
  File "/var/lib/jenkins/.conda/envs/climada_env/lib/python3.7/site-packages/pandas/io/pytables.py", line 3022, in write
    blk.values, items=blk_items)
  File "/var/lib/jenkins/.conda/envs/climada_env/lib/python3.7/site-packages/pandas/io/pytables.py", line 2812, in write_array
    self._handle.create_array(self.group, key, value)
  File "/var/lib/jenkins/.conda/envs/climada_env/lib/python3.7/site-packages/tables/file.py", line 1152, in create_array
    flavor = flavor_of(obj)
  File "/var/lib/jenkins/.conda/envs/climada_env/lib/python3.7/site-packages/tables/flavor.py", line 198, in flavor_of
    "supported objects are: %s" % (type_name, supported_descs))
TypeError: objects of type ``GeometryArray`` are not supported in this context, sorry; supported objects are: NumPy array, record or scalar; homogeneous list or tuple, integer, float, complex or bytes

Hi! I've followed the installation documentation, tried a few approaches, but I can't get the Spyder application to launch. I've run the installation test scripts, which indicates the error is related to matplotlib, as it can't seem to find this library. However, checks for the library turn out fine. Also, everything works fine outside the Climada environment.

I've searched around looking for people who've had similar issues. I've found some, but they're a bit dated, and the solutions (reinstalling matplotlib) haven't worked.

I've not yet tried to install an older version of Climada, but that's my next step. I was hoping that someone among you might have a fix before I go there.

In river_flood.RiverFlood.set_from_nc line 106 is opened and remains open over line 120, when set_raster is called, passing the same file as argument, leading to rasterio.errors.RasterioIOError: '...' not recognized as a supported file format.

Please, try and fix it in the develop branch.

Creating LitPop-entities yields an error for some countries. Upon calling the following lines of code I obtained the output pasted below:

# Import required packages:
import numpy as np
import pandas as pd
from matplotlib import colors
from iso3166 import countries as iso_cntry
from climada.entity import LitPop

# Initiate GDP-Entity for Tanzania, Rwanda, and Burundi:
ent = LitPop()
countries_list = ['ABW', 'AIA', 'ANT']
ent.set_country(countries_list, res_arcsec=120, reference_year=2014, fin_mode='gdp')
ent.set_geometry_points()

2020-10-13 14:01:57,431 - climada - DEBUG - Loading default config file: /Users/simonameiler/Documents/WCR/CLIMADA_develop/climada_python/climada/conf/defaults.conf
/Users/simonameiler/opt/anaconda3/envs/climada_env/lib/python3.7/site-packages/pandas_datareader/compat/__init__.py:7: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  from pandas.util.testing import assert_frame_equal
Traceback (most recent call last):

  File "<ipython-input-2-f4f6a39a9cf7>", line 11, in <module>
    ent.set_country(countries_list, res_arcsec=120, reference_year=2014, fin_mode='gdp')

  File "/Users/simonameiler/Documents/WCR/CLIMADA_develop/climada_python/climada/entity/exposures/litpop.py", line 183, in set_country
    _get_country_info(country_new)

  File "/Users/simonameiler/Documents/WCR/CLIMADA_develop/climada_python/climada/entity/exposures/litpop.py", line 956, in _get_country_info
    cntry_info = [iso_num, country_name, country_shp]

UnboundLocalError: local variable 'country_name' referenced before assignment

It seems that the error might be related to the input rather than the function itself. In the above example, it's 'ANT' which causes the problem. Has anyone encountered similar problems using LitPop or does anyone have an idea of what the problem is? Maybe changes in the util.finance or util.coordinates cause the problem?

... found a vulnerable dependency in a repository you have security alert access to.
| CLIMADA-project/climada_python
Known moderate severity security vulnerability detected in Pillow < 6.2.2 defined in setup.py.

Hi,

I THINK I noticed an inconsistency in the definition of time_step in TC tracks:

• In read_ibtracs_netcdf, time_step is defined in 'seconds' (typically, 10800 which is 3h in seconds, 36060).
• In equal_timesteps, time_step is defined in 'hours'.
• in read_file_emanuel, my understanding is it's in hours
• In read_one_gettelman, I think it's in hours, too.
• In _read_ibtracs_csv_single, I don't know.

This is an issue, because time_step is used in compute_windfields, more specifically:

• _vtrans
• _bs_hol08
  @tovogt Do I guess correctly that for the holland wind model part, time should be in hour? We have to be 100% sure of that and I guess you're best placed to know/check this.

Can someone please check

1. Whether I am correct? If so I guess it's a matter of correcting read_ibtracs_netcdf.
2. Whether there are other usages of 'time_step' in a TCtracks (or TropCyclone) object.

Essentially this wouldn't impact the result if equal_timestep is applied before doing anything else. But if not, translation speed would be great underestimated. That is my guess so far.

I discovered this while attempting to make the random walk application independent of the timestep. I then got tracks all over the world if using data before applying 'equal_timestep' ;-)

Thanks

in CropYieldIsimip.set_from_single_run, lines 273 and 279 the variable self._metadata is modified.

        (self._metadata).append('crop')

        #Method set_to_usd() is called to compute the exposure in USD/y (per centroid)
        #the exposure in t/y is saved as 'value_tonnes'
        if unit == 'USD':
            self['value_tonnes'] = self['value']
            (self._metadata).append('value_tonnes')
            self.set_to_usd(dir_fao=os.path.join(input_dir, 'FAO'))

That is problematic and leads to errors in the Unit test run.
There are two problems. One is the old, infamous Python pitfall. _metadata is a class variable:

class Exposures(GeoDataFrame):
    _metadata = GeoDataFrame._metadata + ['tag', 'ref_year', 'value_unit',
                                          'meta']
    
    ...

    def __init__(self, *args, **kwargs):
        if len(args):
            for var_meta in self._metadata:
                try:
                    val_meta = getattr(args[0], var_meta)
                    setattr(self, var_meta, val_meta)
                except AttributeError:
                    pass
        super(Exposures, self).__init__(*args, **kwargs)

In Python variable initialisation in the class body implicitly creates class variables. (That's different from Java where it doesn't matter whether variables are initialised in the constructor or in the class body.) Object Variables must be assigned with at self reference, i.e. in the constructor or another method.

The other problem is the leading underscore of _metadata. This is a request for privacy and indicates that the variable is meant for internal use. One can manipulate such variables, but in general - and in particular! - it is not advisable.

Quick and dirty solution

Override the Exposures._metatdata in CropYieldIsimip.
Assign self.crop and self.value_tonnes.

class CropyieldIsimip(Exposures):

    _metadata = Exposures._metadata + ['crop', 'value_tonnes']
    ...
    def set_from_single_run(...):
        ...

        #Method set_to_usd() is called to compute the exposure in USD/y (per centroid)
        #the exposure in t/y is saved as 'value_tonnes'
        if unit == 'USD':
            self.value_tonnes = self['value']
            self.set_to_usd(dir_fao=os.path.join(input_dir, 'FAO'))
        else:
            self.value_tonnes = None
        ...

TODO

Re-evaluate options to implement CropYieldIsimip without overriding _metadata.

I just updated develop and incorporated all changes made to the trop_cyclone module from June 11 on

problem is, after running the same analysis (i.e. using same centroids and tracks), I got two different answers between the previous version, and the new one.

previous:

current:

is this expected? if not, any idea of what is happening?

Thanks

Hi all,
using the plot_basemap() function in the centroids base script seems to invert the underlying data grid with respect to the open streetmap basemap. Using the plot_basemap() function for Germany, I always end of with the OSM being upside-down, Baltic and North Sea in the South, with the scatter plot showing correctly.

Running the function line by line individually, I can avoid this behavior when replacing the CLIMADA plot_scatter() function by the geopandas plot() functionality, everything else being equal. Any experience with this behavior?
I obtain the same issue when running the Exposures notebook tutorial, see below:

import contextily as ctx
import matplotlib.pyplot as plt
# select the background image from the available ctx.sources
### standard example
ax = exp_templ.plot_basemap(buffer=30000, cmap='brg') # using open street map
#ax = exp_templ.plot_basemap(buffer=30000, url=ctx.sources.ST_WATERCOLOR, cmap='brg', zoom=9) # set image zoom

### my test example
test=exp_templ.copy()
test.to_crs(epsg=3857, inplace=True)
f, ax = plt.subplots(1, figsize=(8, 8))
ax = test.plot(axes=ax)
ctx.add_basemap(ax, zoom=13)
plt.show()

The elevation package is used by the climada.hazard.tc_surge module and also imported in methods of climada.util and climada.hazard.centroid.

As for now, this package can be well installed on a Windows computer through pip. But it will not work, not in version 1.0.6 in any case.

The reason why it doesn't work is that in the background it basically runs bash or bash commands.

• make can be installed with conda
• unzip can be globally installed with UnxUtils
• mv dito
• find dito, but the PATH must be set before the winows command find
• mkdir mkdir from UnxUtils won't work because option -p is ignored, but can easily be implemented. However there seems to be no way mkdir from Windows can be overridden (probably because it is an "internal command").

Bottomline: elevation must be adapted to work for Windows. Alternatively we relinquish its use and switch to another package or simply implement the required functionality.

TL;DR: climada.util.constants could (mostly) be migrated to climada/conf/defaults.conf; this thread should resolve whether we should do this.

We got to talking about the proper place for storing constants and configuration aspects at today's coding convention meeting; I offered to summarise and start a general discussion here:

Distinct use cases

At least those that I know of, and that were discussed.

Reader support, e.g. through constant dictionary mappings:

Unit conversion, e.g. through floats:

Index-based mappings, e.g. ISIMIP_NATID_TO_ISO

Resource locaters, local or remote, e.g. relating to IBTRACS

Options

I may have missed something in the meeting, and there are of course other options. Please mention them.

• @Evelyn-M Move constant definitions that do not clearly belong with a class (e.g. those making assumptions about file formats) to a separate location for better overview.
• Let things be as they are.
• @mmyrte Start by moving URLs to the configuration file. May also include moving the config file to the (in my experience slightly more orthodox) root of the repository.

What should we do?

Dear Climada Team,

Could you please help me to better understand the tropical cyclone calc_random_walk parameter "max_angle"?

a) If pi is undirected, as stated in the code, I would expect a random spread of the tracks in all directions. Instead the probabilistic tracks are rather narrow.

b) If I set max_angle to 0 I would expect that every replication is the same as the original track. Instead all the replications move somehow parallel northward (is it maybe because cosinus of 0 is 1?).

c) I don't understand that e.g. pi/10 returns a larger spread of probabilistic storms than e.g. pi/4. Why does a smaller angle allow for more variation?

Thank you for your help and best regards,
David

There are two global grid datasets in the CLIMADA repository, but there is no documentation about how they have been generated. One grid is in 150 arc-seconds resolution (GLB_CENTROIDS_NC). The other grid is in 360 arc-seconds resolution and does only contain grid points on land (GLB_CENTROIDS_MAT). Both contain grid points together with country codes:

1) Replace old grids

First, I propose to remove the two datasets GLB_CENTROIDS_NC and GLB_CENTROIDS_MAT in favor of the more recent NATEARTH_CENTROIDS_150AS and NATEARTH_CENTROIDS_360AS:

The file GLB_CENTROIDS_NC is part of the ISIMIP project: https://www.isimip.org/gettingstarted/input-data-bias-correction/details/13/ It is based on the GPWv3 "National Identifier Grid": https://sedac.ciesin.columbia.edu/data/set/gpw-v3-national-identifier-grid Maybe we could download it from ISIMIP or from SEDAC instead of including it in CLIMADA?

2) Replace or remove look-up table

The look-up table NAT_REG_ID defines geographical groups of countries (called "regions"). Furthermore, it contains a mapping from countries to impact function IDs which is only used together with the river flood module.

2.1) Geographical grouping of countries

This particular grouping of countries is similar to the UN geoscheme (M49, https://en.wikipedia.org/wiki/List_of_countries_by_United_Nations_geoscheme), but not exactly the same. Furthermore, it is not in accordance with any other geographical grouping covered by the Python package country-converter (https://pypi.org/project/country-converter/).

The grouping seems to be of some interest for river flood simulation. I would suggest to make this clear in the naming of the variables and of the CSV-file. It should be clear that this data is somehow related to the river flood module.

2.2) Impact function IDs

In NAT_REG_ID, there are 6 different impact function IDs according to the continental regions Africa, Asia, Europe, North America, Australia/Oceania, South America. We could easily list the countries for each ID directly in the Python code or use something like the country-converter package. The file NAT_REG_ID is not really the best place for that purpose - especially since the impact functions listed there are specific for the flood module.

Hi,

Using the new 1.5.0 version I had an issue updating the conda environment. I hence removed the existing, old one and tried to re-create it from scratch using:

conda env create -f requirements/env_climada.yml --name climada_env

However I got the following error:

Collecting package metadata (repodata.json): done
Solving environment: -
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:



Package lz4-c conflicts for:
cartopy=0.18.0 -> pillow -> libtiff[version='>=4.0.9,<5.0a0'] -> zstd[version='>=1.3.7,<1.3.8.0a0'] -> lz4-c[version='>=1.9.2,<1.10.0a0']
pillow=7.2.0 -> libtiff[version='>=4.1.0,<5.0a0'] -> zstd[version='>=1.3.7,<1.3.8.0a0'] -> lz4-c[version='>=1.9.2,<1.10.0a0']
pytables=3.6.1 -> blosc[version='>=1.16.3,<2.0a0'] -> lz4-c[version='>=1.8.1.2,<1.9.0a0|>=1.9.2,<1.10.0a0']
fiona=1.8.13.post1 -> libgdal[version='>=3.0.2,<3.1.0a0'] -> tiledb[version='>=1.6.3,<2.0a0'] -> lz4-c[version='>=1.8.1.2,<1.9.0a0']
geopandas=0.6.1 -> fiona -> libgdal[version='>=2.2.2,<2.3.0a0'] -> zstd[version='>=1.3.7,<1.3.8.0a0'] -> lz4-c[version='>=1.8.1.2,<1.9.0a0|>=1.9.2,<1.10.0a0']
dask=2.25.0 -> bokeh[version='>=1.0.0,!=2.0.0'] -> pillow[version='>=4.0'] -> libtiff[version='>=4.0.9,<5.0a0'] -> zstd[version='>=1.3.7,<1.3.8.0a0'] -> lz4-c[version='>=1.9.2,<1.10.0a0']
Note that strict channel priority may have removed packages required for satisfiability.

I am on Mac OS X (10.14.6). Any idea why and how to solve this?

Issue Description

• Where is the appropriate location for functions shared between modules?
  No matter what solution, under all circumstances we should avoid "cross imports", i.e. situations where module A imports a method from B and vice versa

I'd like to deviate a bit from this location-only focus of the issue and add the following:

• How do we deal with functions that should be util functions (because they perform a frequently-requested action), but are coded individually (in slightly differing ways) in several modules at the moment?
• How do we deal with existing util functions that could be used in more modules than they currently are?

Some background

What is a util function? What is a helper function? Are there meaningful differences?
Most sources admit the terms util and helper are used ambiguously and inconsistently. There is no very clear consensus about their distinction.

A helper method is a term used to describe some method that is reused often by other methods or parts of a program. Helper methods are typically not too complex and help shorten code for frequently used minor tasks. Using helper methods can also help to reduce error in code by having the logic in one place

Some point out that "helper" functions could simply be sub-functions of a main function, used for refactoring & readability purposes within one single module, whereas the term util makes the inter-modular use purpose relatively explicit.

I'll use the term util functions from now on for the scope of this issue, referring to short, re-usable, relatively general functions performing repetitive tasks.

How are we handling util functions at the moment in climada?

• In folder climada/util we have 10 files that contain some sort of util functions (and 4 other items: config.py, constants.py, test/, and earth_engine, which is a module that never quite made it to a stand-alone feature and was hence dumped into util)

• Checking "references" of use of those util functions via github's code navigation tool reveals that, while indeed many are used in several modules, quite a few of those util functions are used only by one module or not at all (i.e. apart from its definition and a corresponding unit test)

• The logic of how util functions are currently grouped within those 10 files is not always very intuitive (e.g. there is a file_handler, but also a hdf5_handler, which is a file-format, after all)

• Checking some of the code base, there are many instances of module-agnostic, repetitive tasks that could be handled by a util function (mostly reading / loading files, converting dates / times, handling strings, converting object-types, etc.)

Propositions

• Location in folder "utils" generally makes sense. However: Should there be some naming hint in the scripts to distinguish scripts containing util funcs form other scripts in that folder (config, earth_engine, constants), e.g. ufuncs_scriptname.py?

• Make a list of currently "unused / single-used" functions in utils, try to find author & critically reflect its potential future use in utils vs relocation to specific module.

• "Awareness raising" on new util functions: Should people send around some kind of notification to currently active climada developers when they add a new util function that is potentially useful to others, incl. short description of what it can do? E.g. via slack / mail? If on the other hand, no use-case can be imagined --> reflect on whether this function really needs to go to utils

• Re-structuring of some of the scripts to logically group util functions (I did not find a good overview on "common util classes", I guess that's a bit project specific. Ideas for further "groups" will be welcome.

• To think about: what about gentle, gradual re-factoring of exisiting modules on master branch, which could integrate more util funcs? Who would take care of this?