tum-ens / pygreta

python Generator of REnewable Time series and mAps

python Generator of REnewable Time series and mAps: a tool that generates high-resolution potential maps and time series for user-defined regions within the globe.

Features

• Generation of potential maps and time series for user-defined regions within the globe
• Modeled technologies: onshore wind, offshore wind, PV, CSP (user-defined technology characteristics)
• Use of MERRA-2 reanalysis data, with the option to detect and correct outliers
• High resolution potential taking into account the land use suitability/availability, topography, bathymetry, slope, distance to urban areas, etc.
• Statistical reports with summaries (available area, maximum capacity, maximum energy output, etc.) for each user-defined region
• Generation of several time series for each technology and region, based on user's preferences
• Possibility to combine the time series into one using linear regression to match given full-load hours and temporal fluctuations

Applications

This code is useful if:

• You want to estimate the theoretical and/or technical potential of an area, which you can define through a shapefile
• You want to obtain high resolution maps
• You want to define your own technology characteristics
• You want to generate time series for an area after excluding parts of it that are not suitable for renewable power plants
• You want to generate multiple time series for the same area (best site, upper 10%, median, lower 25%, etc.)
• You want to match historical capacity factors of countries from the IRENA database

You do not need to use the code (but you can) if:

• You do not need to exclude unsuitable areas - use the Global Solar Atlas or Global Wind Atlas
• You only need time series for specific points - use other webtools such as Renewables.ninja
• You only need time series for administrative divisions (countries, NUTS-2, etc.), for which such data is readily available - see Renewables.ninja or EMHIRES

Outputs

Potential maps for solar PV and onshore wind in Australia, using weather data for 2015:

Contributors ✨

Thanks goes to these wonderful people (emoji key):

kais-siala 💬 🐛 💻 📖 🤔 🚧 👀 ⚠️ 📢	HoussameH 💬 💻 📖	Pierre Grimaud 🐛	thushara2020 👀	lodersky 📖 💻 👀	sonercandas 📖	patrick-buchenberg 📦
molarana 🎨

This project follows the all-contributors specification. Contributions of any kind welcome!

Please cite as:

Kais Siala, & Houssame Houmy. (2020, June 1). tum-ens/pyGRETA: python Generator of REnewable Time series and mAps (Version v1.1.0). Zenodo. https://doi.org/10.5281/zenodo.3727416

pygreta's Issues

Issue warning from certain util functions

reg_miss_folder and reg_miss_files can be converted to warning issues.

PV does not seem to work correctly

There were two main errors:

1. The code always returned CF_csp. I could not fully understand why.
   Instead of writing...

   aux = np.zeros(len(reg_ind[0]))
   aux[filter] = CF_pv
   CF_pv = aux
   aux[filter] = CF_csp
   CF_csp = aux
   

I added a new line that seemed redundant to me and it worked (= returned CF_pv).

    aux = np.zeros(len(reg_ind[0]))
    aux[filter] = CF_pv
    CF_pv = aux
    aux = np.zeros(len(reg_ind[0]))
    aux[filter] = CF_csp
    CF_csp = aux

I suspect that before, all the names CF_pv, aux and CF_csp used to point to the same thing in the memory.

2. The filter is incorrect. I turned it off and it works now. I think I filtered the afternoon hours out, which I should not do. This needs to be fixed. Without filtering, the code works but is slow.

Integrate PVLib model for Solar Cf calculations

PVlib is a python module used for PV modeling. Integrating it to the script might improve readability and reliability of PV capacity calculations.

Documentation found under:
https://pvlib-python.readthedocs.io/en/stable/package_overview.html#credits

add progress bar for reporting

Include contributions

@all-contributors please add @kais-siala for bug, code, doc, ideas, maintenance, review, test and talk

Paths for report

Put the paths in the config file
Add the region name to the file name, to be consistent with other files

Generate Timeseries Speed boost possible

generate_timeseries still reads the landuse and weather data in each process of the pool
Speed up possible.

calc_gcr

move function to Master and rename it as generate_ground_cover_ratio.
It should only run if "PV" is in the list of technologies and the TIF file does not exist yet.

Look into GPU Acceleration

Numba might be used to accelerate and parallelize calculations, by making use of the Nvidia GPU installed in a system.

Requires:

• Investigate feasibility
• List required changes
• Test speed improvements

Numba documentation: http://numba.pydata.org/numba-doc/latest/index.html

Use matrix convolution for the creation of buffers

Currently, the code creating buffers is quite long. I am sure it can be done easily with matrix convolution. Check:

• https://de.mathworks.com/matlabcentral/answers/395258-most-efficent-way-of-finding-submatrices-of-a-matrix
• https://stackoverflow.com/questions/43373521/how-to-do-convolution-matrix-operation-in-numpy

cProfile

is the cporfile at the end of the script usable?

paths not needed in calc_FLH_tech and calc_TS_tech

paths dictionary can be removed from calc_FLH_solar and wind, and from Calc_TS_solar and wind

Broken Link: Download of MERRA data via FTP not possible any longer

Hey,

I was going through the documentation and found that the link for the MERRA2 weather data is not available any longer.

Deprecation of the MERRA and MERRA-2 Unique Web Interface Subsetter 

It seems that now users need to use the Web API. Is there already a script as a template available for the the download of required data? Otherwise I would probably write one which could be also added to the docs for example?

References

Currently, the references are displayed correctly in the web version, but not in the PDF.
In fact, they appear as footnotes in the PDF, whereas I would like to have only one section at the end called "References", which shows the sources as it would do in a bibliography.

Masking and weighting function not compatible with multiple technology inputs

Masking and weighting are not compatible with multiple technologies listed in the tech input variable

Regression TS not what needed by stratified_timeseries

The module clean_TS_regression creates a file that is not what is needed by stratified_timeseries.
Please pull the latest commit in my branch quick-fixes and test with it.

Save calc_region in memory instead of tif

New name for repo

Any suggestions?
Apparently, I can rename it and users who search for the other link get redirected to the new one.

generate_area

The calculation of the area should not be done for each technology, but once in the beginning.
And... let's save it as a mat file, not a tif. A tif file for Europe is 1 GB!

Regression paramters

It would be nice if we could differentiate between hub_heights for onshore and offshore.
Can we run the script for CSP as well?

list_regions for regression

Right now it is the intersection of EMHIRES and IRENA.
In the future, I want it to be based on the file of IRENA that is saved in the regression folder. If that region is not in EMHIRES, then a back-up time series should be used instead.
Any thoughts on this?

Code style and contributors

Can you look into this project and see how we can implement the code style "Black" and the all-contributions standard?
https://github.com/pyproj4/pyproj/blob/master/README.md

Add examples to README.md and to readthedocs

Add examples of report, time series, potential maps.
These can be used in the first page (index), and in expected outputs (user manual).

Add module for reporting

It would be nice to have a table that provides a report after masking and weighting.
The rows will be regions.
Possible columns (not exhaustive):

• number of points before masking (using A_land, A_eez)
• number of points after masking
• area after weighting in km²
• mean, median, max, min, std before masking
• mean, median, max, min, std after masking
• power potential in TW peak
• energy potential in TWh before weighting and masking
• energy potential in TWh after weighting
• energy potential in TWh after weighting and masking (could be redundant, if weighting includes masking)
• sorted sample of FLH values before masking (length: 10000 pixels?)
• sorted sample of FLH values after masking (same length)
• sorted sample of FLH values after weighting (same length)

I have a Matlab script that does some of these things. Also a zonal_statistics module written in Python. They can be useful!

save FLH as tif as well

Currently it is only saved as mat file

Describe modules and functions

Use the triple quotation mark to provide a description of modules that could appear if we type "help module_name".
The description should include a definition of the arguments and the outputs (size, unit...).

Are the topics on the first page ok?

@all-contributors please add @kais-siala for question

Minimal environment requirements

We have to check which libraries are actually used, and create a yaml file for them.

Fix Index in documentation

I used :noindex: in Master.py to avoid warnings, but it led to no indices for the functions within that module. I would like to fix this without getting any warnings.

The issue does not arise for util.py, because I am not splitting it into thematic groups of functions.

Pysal 2.0

Check where we use Pysal, and how we should rewrite the functions so that we can use Pysal 2.0 instead of v1.14.
Also update the env file.

complete documentation

I left "?????" as a place holder in the documentation, wherever I was not sure.

Use matrix convolution for the calculation of the slope?

After we try it with the buffers, we should try using it for the slope.

Logo for the documentation ?

Separate regression

The regression modules requires that the user reruns the script after moving the files manually anyway. I suggest separating it from "Master.py" and creating another script just for it. The script might still use the assumptions in config.py

Param["year"] int not string, find conflicts and resolve

generate the maps for any resolution (flexible res_high)

Right now only res_high = [1/240 1/240] is possible.
For this to work correctly, we have to rewrite many parts of the code.
A_lu will not be a 2-D matrix any more, but a 3-D matrix, where each pixel is replaced by a 1x17 vector that counts the occurrence of certain land use types. In the trivial case of res_high = [1/240 1/240], those vectors will be filled with 16 zeros and 1 one. Or it will be replaced with the most common land use type (easy).
For float values (elevation, bathymetry), we would calculate the average.

Warning in pv_parameters

convert print("Warning: ...") into a proper warning
and document it

Expected outputs and recommended workflow

Should we merge these two sections of the User Manual? This could be done with a general graph a the start describing the sequence of functions, followed by a quick description of the major modules with examples of their outputs and a link to their function description?

Write down LU categories in config.py

you can copy them from create_buffer

generate_wind_correction

Right now we read res_correction of onshore wind in the beginning. However, we should also read it for offshore wind. That part is missing.

Clean commit history

Currently @HoussameH has almost 2 million lines of code added and deleted... could you identify the exact commit where that happened? It should be in the beginning of May, I think.
We can rebase the history to fix that error. It must be some files that you have forgot to ignore.

Can we run cProfile to see what takes time?

Here is the command:
https://stackoverflow.com/questions/8283112/cprofile-saving-data-to-file-causes-jumbles-of-characters

Add timestamp at various steps

We can add timestamps before we start and after we finish calculating one region or creating a file.
@HoussameH can you do this please?

Missing functions' description

The functions Generate_wind_correction(), weighting() , calc_gwa_correction(), and sumnorm_MERRA2() are still missing descriptions.

Complete documentation

Some "??????" are still present in the documentation, e.i. regression_paths

clean EMHIRES

instead of copying EMHIRES text files, I would rather read them, clean/filter them, and save the part that is used by the code in the regression folder, similarly to IRENA.

Replace (m x n) arrays with vectors of valid points

If we keep working with regions in the calculation of FLH, we can store the indices of the valid pixels using A_region (one long vector, linear indexing) and filter the matrices with that vector.
We can also scrap the for-loop of regions and just filter using A_land / A_eez.

Objective: avoid redundancy, and only calculate what is needed at the end.
The size of the matrices will be smaller --> better use of memory and CPU power.
Also, the scripts calculating the FLH will be similar to those calculating particular points at particular locations. The list of points will simply be longer.

Explanations for downloading the weather data in bulk confusing, add more details

sub2ind and ind2sub

Verify whether these two functions are still needed. If not, delete them.

Regression Coefficients

Implement Regression_coefficient module:
Finds the best fit for EMHIRES TS shape, with IRENA FLH constraint.

Required steps:

• reorganize the script to generate multiple heights
• Set up Pyomo abstract model
• implement data_input function
• Test results