agrdatasci / ag5tools

R toolbox to download and extract data from the "Agrometeorological indicators from 1979 to present derived from reanalysis" (AgERA5) dataset.

ag5Tools

Toolbox for downloading and extracting data from the Copernicus AgERA5 dataset

Description

The R package ag5Tools is a toolbox to download and extract data from the "Agrometeorological indicators from 1979 to present derived from reanalysis" dataset (AgERA5).

https://cds.climate.copernicus.eu/cdsapp#!/dataset/10.24381/cds.6c68c9bb?tab=overview

The download function provides programmatic access to the Copernicus Climate Data Store to download AgERA5 data.

A detailed description can be found in the companion paper: https://doi.org/10.1016/j.softx.2022.101267

Data license

The ag5Tools package does not distribute data, it only provides access to Climate Data Store through the python cdsapi.

For specific details about the license agreement on downloading and using the data please check the license at: https://cds.climate.copernicus.eu/api/v2/terms/static/licence-to-use-copernicus-products.pdf

Installation

The stable version of ag5Tools can be installed from CRAN using

install.packages("ag5Tools")

The development version could be installed using

devtools::install_github("agrdatasci/ag5Tools", build_vignettes = TRUE)

Downloading AgERA5 data

Install Python and the CDS API

Before using the ag5Tools package, you need to intall Python and the CDS API, please see the details here: https://cds.climate.copernicus.eu/api-how-to.

Get your CDS API-Key

To download AgERA5 data you should first register at the Climate Data Store and get your API key. Please follow the instructions in: https://cds.climate.copernicus.eu/api-how-to

You need to create a file to store the API key, following the instructions from: https://cds.climate.copernicus.eu/api-how-to.

Examples

Downloading data

The following example downloads daily '2m_temperature' data for year 2015

ag5_download(variable = "2m_temperature",
             statistic = "night_time_minimum",
             day = "all",
             month = "all",
             year = 2015,
             path = "C:/custom_target_folder"
             )

Extracting data

To extract maximum day temperature ("Max-Day-Time") of "2m_temperature"

ag5_extract(coords = c(35.726364, -2.197162), 
            dates = "1995-01-23", 
            variable = "2m_temperature",
            statistic = "Max-Day-Time", 
            path = "C:/agera5_data")

Acknowledgements

The ag5Tools package relies on the functionality available from other open source packages.

• The Python cdsapi

• The R package reticulate is used to access the Python CDS API functions from R.

• The R package terra is used to extract data from nc files.

• The R package fs is used for efficiently search and list files.

License

Please be aware that ag5Tools is released under MIT license, please find details in the MIT license document

Citation

To cite the ag5Tools in scientific publications please use:

Brown, D., de Sousa, K., & van Etten, J. (2023). ag5Tools: An R package for downloading and extracting agrometeorological data from the AgERA5 database. SoftwareX, 21, 101267. https://doi.org/10.1016/j.softx.2022.101267

Code of Conduct

Please note that the ag5Tools project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

ag5tools's Issues

Permission to edit the "About" section

I cannot edit the "About" section. Not critical yet, but opening now so I don't forget it.

Unable to download a subset of the map

In the Python API there is an option to extract only a part of the land, namely through the "area" argument. Would be super nice to be able to do it here as well, because the reduction in size of the downloaded files is huge!

Unable to run example after installing

I get an error about a missing Python module after installation. I then tried installing the module with pip, I still get the same error after it successfully installed for me.

  ag5_download(variable = "2m_temperature",
                           statistic = "night_time_minimum",
                           day = "all",
                           month = "all",
                           year = 2015,
                           path = "C:/custom_target_folder"
                           )
List of 20
 $ python              : chr "/opt/homebrew/Cellar/[email protected]/3.10.12_1/bin/python3.10"
 $ libpython           : chr "/opt/homebrew/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/Python"
 $ pythonhome          : chr "/opt/homebrew/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10:/opt/homebrew/Cellar/pytho"| __truncated__
 $ pythonpath          : chr "/Users/adamsparks/Library/R/arm64/4.3/library/reticulate/config:/opt/homebrew/Cellar/[email protected]/3.10.12_1/Fram"| __truncated__
 $ prefix              : chr "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.10"
 $ exec_prefix         : chr "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.10"
 $ base_exec_prefix    : chr "/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.10"
 $ virtualenv          : chr ""
 $ virtualenv_activate : chr ""
 $ version_string      : chr "3.10.12 (main, Jun 20 2023, 19:43:52) [Clang 14.0.3 (clang-1403.0.22.14.1)]"
 $ version             : chr "3.10"
 $ architecture        : chr "64bit"
 $ anaconda            : logi FALSE
 $ conda               : chr "False"
 $ numpy               :List of 2
  ..$ path   : chr "/opt/homebrew/lib/python3.10/site-packages/numpy"
  ..$ version: list()
  .. ..- attr(*, "class")= chr "numeric_version"
 $ required_module     : chr "cdsapi"
 $ required_module_path: NULL
 $ available           : logi TRUE
 $ python_versions     : chr "/opt/homebrew/Cellar/[email protected]/3.10.12_1/bin/python3.10"
 $ forced              : chr "the current process"
 - attr(*, "class")= chr "py_config"
Error: Python module cdsapi was not found.

Detected Python configuration:

Extracting scattered date from several coordinates using ag5_extract

Hi there,

I recently came across this package and found it amazing.

I am able to download the AgERA5 data that I wanted to my computer and can extract single coordinates and locations, however, I am struggling to batch download climate data that are not sequential.

For instance, I can download data from scattered days using the below code:

library(ag5Tools)
ag5_download(variable = "2m_temperature",
             statistic = "day_time_mean",
             day = c(01,05,10,15,20,25),
             month = c(10, 11, 12, 01, 02, 03),
             year = 2015,
             path = "/Users/myfolder"
)

But when trying to extract such specific dates I got the obvious error message below, which I understand that I have not downloaded all the dates within the interval between the first and second dates, and the argument date seems to do not support non-sequential dates, as explained elsewhere :

If dates is one value it extracts a single observation for the specified variable and location. If dates is a character vector of length == 2, it will extract a time series of the specified variable and location, where the first dates value is the start date and the second the end date.

Please find here what I did the and error message.

# Create vectors for days, months, and year
days <- c(01, 05, 10, 15, 20, 25)
months <- c(10, 11, 12, 01, 02, 03)
year <- 2015

# Generate all possible combinations of days and months
all_dates <- expand.grid(days = days, months = months)

# Create a data.frame with the dates column
df <- data.frame(dates = as.character(paste(year, all_dates$months, all_dates$days, sep = "-")))

# sample coordinates
df_coord <- data.frame(lat = c(-20, -24), lon = c(-80, -90))

ag5_extract(coords = df_coord, 
            lon = "lon",
            lat = "lat",
            # dates = all , 
            variable = "Temperature-Air-2m",
            statistic = "Mean-Day-Time", 
            path = "/Users/nicolaycunha/Desktop/Kaoma2015",
            celsius = TRUE)

Error in ag5_extract.numeric(coords = c(35.726364, -2.197162), dates = 2015 -  :   object 'ag5_data' not found

In summary, I am reaching out to inquire whether it is possible to extract scattered dates from different coordinates within a row. My current project involves analyzing a vast dataset encompassing multiple locations in South America, and I specifically require data ranging from 1980 to 2023 for approximately 5000 specific coordinates. Unfortunately, I have encountered challenges in efficiently utilizing the available package to achieve my objectives.

Ideally, I would like to download climate data solely for the required coordinates and a subset of dates. This approach would alleviate the need to manage extensive datasets on a local computer, thus enhancing the overall efficiency of the analysis process.

I would greatly appreciate any advice or assistance you can provide in this regard. Please let me know if the current version of the functions supports such capabilities or if there are alternative approaches to achieve the desired outcome.

Thank you kindly for your attention and support.

Best regards,
Nicolay

ag5_extract not locating some variables

I have four variables I would like to extract from files in my directory, downloaded using the ag5_download function but I can only extract two of them: Mean 24 hour temperature and Precipitation Flux. The function can't seem to locate the files for vapour pressure or wind speed. Can you help?

Filenames:

Temperature-Air-2m-Mean-24h_C3S-glob-agric_AgERA5_20110101_final-v1.1.nc

Precipitation-Flux_C3S-glob-agric_AgERA5_20110101_final-v1.1.nc

Vapour-Pressure-Mean_C3S-glob-agric_AgERA5_20110101_final-v1.1.nc

Wind-Speed-10m-Mean_C3S-glob-agric_AgERA5_20110101_final-v1.1.nc

When I run the following code, temperature and precipitation are found but vapour pressure and wind speed are not.

ag5_extract(coords = c(-54.45582, -25.64690),
dates = "2011-01-01",
variable = "Temperature-Air-2m",
statistic = "Mean-24h",
celsius = T,
path = "my_directory/")

2011-01-01
1 23.09246

ag5_extract(coords = c(-54.45582, -25.64690),
dates = "2011-01-01",
variable = "Precipitation-Flux",
path = "my_directory/")

2011-01-01
1 11.4

ag5_extract(coords = c(-54.45582, -25.64690),
dates = "2011-01-01",
variable = "vapour_pressure",
statistic = "24_hour_mean",
path = "my_directory/")

Error in get_file_path(date_to_search = dates, variable, statistic, time, :
File not found

ag5_extract(coords = c(-54.45582, -25.64690),
dates = "2011-01-01",
variable = "10m_wind_speed",
statistic = "24_hour_mean",
path = "my_directory/")

Error in get_file_path(date_to_search = dates, variable, statistic, time, :
File not found

Error with extraction of wind variable

Dear David

It's me again, I downloaded all de files but when a want to extract "wind" from netcdf daily files I get de following error:

****dk_wind <- ag5_extract(coords = c(9.546480, 56.437384),

•                    dates = c("2015-01-01", "2023-12-31"), variable = "10m_wind_speed", statistic="24_hour_mean",path = "D:/ERA5")
  

Error in get_file_path(date_to_search = X, variable, statistic, time, :
File not found****

All the other variables i can extract without any problem. My netcdf files are stored on a external drive because of size but that didn't give any problems for the other variables. What is the correct way for Variable and statistic for wind data sets?

Kind regards,
Howard

Ag5tools cdsapi client error

Dear David,
I am a PhD student at Wagenen University, I came across your Ag5tools Package. I followed all the steps but a get this error massage when I call download function.

Error in cdsapi$Client() : attempt to apply non-function

session_python gives following info
python: C:/Users/meer164/AppData/Local/anaconda3/envs/wk_ecmwf/python.exe
libpython: C:/Users/meer164/AppData/Local/anaconda3/envs/wk_ecmwf/python36.dll
pythonhome: C:/Users/meer164/AppData/Local/anaconda3/envs/wk_ecmwf
version: 3.6.15 (default, Dec 3 2021, 18:25:24) [MSC v.1916 64 bit (AMD64)] Architecture: 64bit
numpy: C:/Users/meer164/AppData/Local/anaconda3/envs/wk_ecmwf/Lib/site-packages/numpy numpy_version: 1.19.5
cdsapi: C:\Users\meer164\AppData\Local\ANACON~1\envs\wk_ecmwf\lib\site-packages\cdsapi_init_.p

NOTE: Python version was forced by RETICULATE_PYTHON

Can you please help me out?
Thanks in advance

Howard van Meer