bluegreen-labs / daymetr Goto Github PK

I recently tried using the daymetr package to download Daymet data for specific tiles (e.g., 11570 for the years 2017-2020). The files downloaded using the daymetr package are different in size from the ones I download manually from the Daymet (ornldaac) webpage and cannot be opened in R using any method I have tried. I obtain an R error message saying that the files are of an unknown file format. This seems to be a more recent issue as in 2021 when I tried to use the daymetr package to download the files for years up until 2020, I obtained the exact same files that I downloaded manually from the webpage. For some reason, I was never able to download files that could be opened from the year 2020 using the daymetr package, but now when I try to download the files for any year, they are all unreadable. The files I have recently tried to download using the daymetr package are all 628 bytes in size, whereas the files that I download manually from the webpage range in size from 1.5 MB to 79.8 MB. Is this an issue with the daymetr package, or do you know how to resolve this issue?

Just curious - when will daily meteorological estimates in daymetr be available for the year 2022?

This package depends on (depends, imports or suggests) raster and one or more of the retiring packages rgdal, rgeos or maptools (https://r-spatial.org/r/2022/04/12/evolution.html, https://r-spatial.org/r/2022/12/14/evolution2.html). Since raster 3.6.3, all use of external FOSS library functionality has been transferred to terra, making the retiring packages very likely redundant. It would help greatly if you could remove dependencies on the retiring packages as soon as possible.

I am attempting to query netCDF subset data for NW Wyoming and convert nc files to tif for manipulation and analysis. Here is a reproducible example:

download_daymet_ncss(location = c(44.49073, -111.0388, 42.9683, -109.7625), 
                     start = 2020, end = 2020, 
                     param = c("tmin", "tmax"), 
                     frequency = "monthly", 
                     path = tempdir())

nc2tif(path = tempdir(), overwrite = TRUE)

When I run this code on my personal machine (Mac OS 12.6.2), I get the following error message:

Error : [rast] cannot open this file as a SpatRaster: /private/var/folders/75/5_f2nf9136d7jlc_4jx157xm0000gn/T/RtmplE3WQJ/tmax_monavg_2020_ncss.nc
In addition: Warning message:
/private/var/folders/75/5_f2nf9136d7jlc_4jx157xm0000gn/T/RtmplE3WQJ/tmax_monavg_2020_ncss.nc' not recognized as a supported file format. (GDAL error 4)
Conversion error...corrupt file?
Error : [rast] cannot open this file as a SpatRaster: /private/var/folders/75/5_f2nf9136d7jlc_4jx157xm0000gn/T/RtmplE3WQJ/tmin_monavg_2020_ncss.nc
In addition: Warning message:
/private/var/folders/75/5_f2nf9136d7jlc_4jx157xm0000gn/T/RtmplE3WQJ/tmin_monavg_2020_ncss.nc' not recognized as a supported file format. (GDAL error 4)
Conversion error...corrupt file?

Terra has known compatibility issues with Mac OS, particularly with respect to netCDF, as documented here. Following the advice in the link, I installed the development version of terra, but still received the same error message when running the nc2tif() function. To see if I could work around this, I tried running the code on a Windows 10 machine. The code ran without error.

Is this an issue the Daymetr developers are aware of? Ideally, all my data processing would be done locally on my Mac OS machine. Is there any way to work around terra, within daymetr, without splitting my workflow across two separate machines?

Thank you and I greatly appreciate any advice.

Package and dependencies info (versions are identical across platforms): daymetr (1.7), sf (1.0-12), terra (1.7-35), ncdf4 (1.21), tidyr (1.3.0), tibble (3.2.0).

I am unable to download any daymet data with the download.daymet function. I receive the following error message, even when using the example code found on the help site.

cldat<- download.daymet(site="Oak Ridge National Laboratories",lat=36.0133,lon=-84.2625,start_yr=1980,end_yr=2010,internal=TRUE)
Downloading DAYMET data for: Oak Ridge National Laboratories at 36.0133/-84.2625 latitude/longitude !
Error in download.daymet(site = "Oak Ridge National Laboratories", lat = 36.0133, :
You requested data is outside DAYMET coverage,
the file is empty --> check coordinates!

I am running R version 3.3.1 and using the following packages as dependencies: sp 1.2-3; rgdal 1.1-10; downloader 0.4. I am using Windows 7 Enterprise.

Some variables are not available in the monthly / yearly summaries. Remove from docs.

Check unit tests, most likely a non issue

This example should remove all doubt on how to reproject tiled Daymet data. The example downloads a single tile around Cape Cod, a distinct geographic feature. I plot the data in the corrected projection, as well as reprojected to EPSG 4326.

In short, the correct CRS projection of the tiled data is:
+proj=lcc +lat_1=25 +lat_2=60 +lat_0=42.5 +lon_0=-100 +x_0=0 +y_0=0 +ellps=WGS84 +units=m +no_defs

unlike what is reported in the tile meta-data!

library(daymetr)
library(raster)
library(sp)
library(rnaturalearth)

# download data
download_daymet_tiles(tiles = 11755, param = "prcp", start = 1980, end = 1980)

# read raster
r <- raster::raster(file.path(tempdir(),"prcp_1980_11755.nc"))

# set the correct projection
raster::projection(r) <- "+proj=lcc +lat_1=25 +lat_2=60 +lat_0=42.5 +lon_0=-100 +x_0=0 +y_0=0 +ellps=WGS84 +units=m +no_defs"

# reproject to lat long
r_ll <- raster::projectRaster(r, crs = "+init=epsg:4326")

# grab usa outlines from rnaturalearth
usa_ll <- ne_countries(scale = 50, country = "United States of America", returnclass = "sp")
usa <- spTransform(usa_ll, CRSobj = CRS("+proj=lcc +lat_1=25 +lat_2=60 +lat_0=42.5 +lon_0=-100 +x_0=0 +y_0=0 +ellps=WGS84 +units=m +no_defs"))

# plot simple overview graphs
par(mfrow = c(2,1))
plot(r, main = "LLC projection")
lines(usa)

plot(r_ll, main = "lat lon projection")
lines(usa_ll)

When I run this,

download_daymet_tiles(
  location = c(36.61,-83.37,35.57,-81.29),
  start = 1980,
  end = 1980,
  param = "tmin"
)

I get this

Downloading DAYMET data for tile: 11209; year: 1980; product: tmin
  |==================================================| 100%

Downloading DAYMET data for tile: 11210; year: 1980; product: tmin
  |==================================================| 100%

Downloading DAYMET data for tile: 11389; year: 1980; product: tmin
  |==================================================| 100%

Downloading DAYMET data for tile: 11390; year: 1980; product: tmin
  |==================================================| 100%

But, only the downloaded file for the first tile (11209) seems to have actual data in it. This behavior is consistently observed for other geographic extents I tried. Am I missing something?

Thornton, P.E., M.M. Thornton, B.W. Mayer, Y. Wei, R. Devarakonda, R.S. Vose, and R.B. Cook. 2017. Daymet: Daily Surface Weather Data on a 1-km Grid for North America, Version 3. ORNL DAAC, Oak Ridge, Tennessee, USA. https://doi.org/10.3334/ORNLDAAC/1328

download_daymet_ncss() or download_daymet_tiles() on daily tmin,tmax or prcp and others are downloading blank (NULL) files to my path folder.

https://thredds.daac.ornl.gov/thredds/ncss/ornldaac/1840/daymet_v4_daily_na_prcp_2019.nc

on the Daymet websites' thredds server link, seems like they provide different paths on their server.

See-
https://thredds.daac.ornl.gov/thredds/catalog/ornldaac/2129/catalog.html?dataset=2129/daymet_v4_daily_na_tmax_2009.nc

Is this a simple version mismatch issue?

Thank you in advance!

I can download single pixel data for year 2020 using the following code:

download_daymet(site = "test", lat = 44.598173, lon = -111.092613, start = 2020, end = 2020, path = tempdir(), silent = FALSE, force = TRUE)

However, when I try to download gridded data for the same time period, the following code fails:

download_daymet_ncss(location = c(44.598173, -111.092613, 42.842560, -109.616358), start = 2020, end = 2020, param = c("tmin", "tmax"), frequency = "daily", path = tempdir(), silent = FALSE, force = TRUE)

Error message:
Error: Requested data download failed!
Common issues involve a mismatch between
the location and the mosaic used or downloads
exceeding 6GB in size.

In both code chunks I am using the argument "force = TRUE" to override the default maximum end year. Note that I can run the download_daymet_ncss() function on year 2019 with no issue. Also note that the location of the single pixel download is the top-left coordinate of the bounding box for the gridded download.

Please advise. Thank you.

prepare for release:

• Check CRAN issues
• urlchecker::url_check()
• devtools::check(document = FALSE)
• rhub::check_for_cran()
• Update cran-comments.md
• Update NEWS.md
• R CMD check --as-cran
• git push

Submit to CRAN:

• devtools::submit_cran()
• Approve email
• Accepted email

Tile download does nothing when bounding box is passed as location argument:

https://github.com/khufkens/daymetr/blob/e9faa371d1c8e7da174a078cceddc20e22a7533c/R/download_daymet_tiles.r#L102

loops through tiles instead of tile_selection
also
https://github.com/khufkens/daymetr/blob/e9faa371d1c8e7da174a078cceddc20e22a7533c/R/download_daymet_tiles.r#L47

should set tile_selection not tiles

Hi,

I'm sure that the projection from LCC to EPSG 4326, as in the daymetr vignette (https://cran.r-project.org/web/packages/daymetr/vignettes/daymetr-vignette.html), is not working correctly.

I've tried it many times and the projection transformation is not correct. It ends up as a very small extent, with exceptionally high resolution output in the wrong place.

I'm referring to the section of the vignette: "Querying netCDF subset (ncss) data"

# to set the correct projection use
raster::projection(r) <- "+proj=lcc +lat_1=25 +lat_2=60 +lat_0=42.5 +lon_0=-100 +x_0=0 +y_0=0 +a=6378137 +b=6356752.314706705 +units=m +no_defs"

# reproject to lat lon
r <- raster::projectRaster(r, crs = "+init=epsg:4326")
plot(R)

Even if you look at the ouput of plot(r), i think this is visible..it seems to be an extent of only approx 42.4929 to 42.4933 degrees latitude, which is less than 1km (0.00833) so less than 1 pixel (since daymet is 1 km resolution). The output image however shows many pixels..

I tried with CDO and NCO. I think this is a problem in how the netcdf file is being read. The units of measure in the netcdf are listed as km. Should it be m?

Also, Daymet documentation says that the LCC transform is based on wgs 84 datum an dhas only one parameter a listed (no b parameter is listed). Should that info be included?

Hope you can resolve this, as i'm counting on using this package which i think is great!!

Thank you,
Vishal

Similar to the advice during ropensci review on MODISTools, convert to a tidy data format for easier manipulation and concatenation of data (simple merge / rbind rather than more convoluted lapply loops).

NCSS downloads have a fault in their projection information, set the correct values as such:

# set the correct projection
raster::projection(r) <- "+proj=lcc +lat_1=25 +lat_2=60 +lat_0=42.5 +lon_0=-100 +x_0=0 +y_0=0 +ellps=WGS84 +units=km +no_defs"

Is it possible to identify which variables a user is interested in when accessing many sites using download_daymet_batch? For instance, I am only interested in TMAX and TMIN in my analysis and do not need the other variables that daymetr provides. Being able to identify which variables are of interest inside the download_daymet_batch() function could cut down on memory usage and increase processing speed.

See pull request Mike.

#20

This is with version 1.2 installed today with commit a68b39c:

The call download_daymet(path = tempdir(), internal = FALSE) fails with message

Error in file.copy(daymet_tmp_file, daymet_file, overwrite = TRUE, copy.mode = FALSE) :
file can not be copied both 'from' and 'to'

because the arguments are such that daymet_tmp_file and daymet_file are identical.

The current code that throws the error is:

  file.copy(daymet_tmp_file, daymet_file, overwrite = TRUE,
    copy.mode = FALSE)
  file.remove(daymet_tmp_file)

A suggestion to fix this behavior is to only copy and delete tmp-file if not identical to the one from the user requested path, e.g., wrap the above lines in a if-condition:

  if (!identical(normalizePath(daymet_tmp_file), normalizePath(daymet_file))) {
    file.copy(daymet_tmp_file, daymet_file, overwrite = TRUE,
      copy.mode = FALSE)
    file.remove(daymet_tmp_file)
  }

Thank you very much for making your package available! We are using it in our research work https://github.com/Burke-Lauenroth-Lab/rSFSW2

Daymet data headers are seemingly not set in stone, so fixed read to extract meta-data can not be relied on. Do a grep on the header to find meta-data instead.

The code below counts the number of days that meet a user-specified condition, and returns a spatially-explicit result as either a raster object or file. This tool is able to extract useful indices like Growing Degree Days.

I put this together to extract GDD, freezing degree days, and as a data exploration tool.

This code is currently reliant on sapply. I am working on making it more efficient using raster::overlay and raster::calc.

@khufkens do you think that this would fit within the package?

#' @param data string containing the name of a file
#'  containing the daily Daymet data to be evaluated
#' @param startday numeric day of year at which counting should begin. 
#' Default = 0 (January 1)
#' @param endday numeric day of year at which counting should end. 
#' Default = 365 (December 31)
#' @param criteria logical expression (e.g. ">") to be evaluated
#' @param value numeric the value that the criteria is to be evaluated
#'  against
#' @param internal TRUE / FALSE (if FALSE, write the output to file)
#' using the Daymet file format protocol.
#' 
#' @description This tool counts the number of days in a given time period
#' that meet a given set of criteria. This can be used to extract indices 
#' such as Growing Degree Days (tmin > 0), or days with precipitation 
#' (prcp != 0).

calc_nd <- function(data = NULL,
                    startday = 0,
                    endday = 365,
                    criteria = NULL,
                    value = NULL,
                    internal = FALSE){
  
  # perform input checks
  if(is.null(data) | is.null(criteria) | is.null(value)){
    stop('Please specify data, criteria and value.')
  }
  
  # load data and convert to dataframe
  data <- suppressWarnings(raster::brick(data))
  data.df <- raster::as.data.frame(data,
                                   stringsAsFactors = FALSE)
  
  # select desired dates from data to create working dataset
  data.wrk <- as.data.frame(data.df[ , ,startday:endday],
                            stringsAsFactors = FALSE)
  
  # use boolean indexing in sapply to subset working data
  sel <- sapply(data.wrk,
                criteria,
                value)
  
  # convert df to array
  nr       <- raster::nrow(data)
  nc       <- raster::ncol(data)
  nbnd     <- ncol(sel)
  dim(sel) <- c(nr,nc,nbnd)
  
  # convert array to raster
  crs      <- raster::crs(data)
  ext      <- raster::extent(data)
  sel.ras  <- raster::brick(crs = crs,
                            nrows = nr,
                            ncols = nc,
                            nl = nbnd)
  sel.ras <- raster::setValues(sel.ras,
                               sel)
  
    
  # use SUM to gather the number of days that meet the criteria
  result <- raster::calc(sel.ras,
                         sum,
                         na.rm=TRUE)
  
  # return all data to raster, either as a geotiff or as a local object
  if (internal == FALSE){
    # create output file name
    year <- strsplit(input_file, "_")[[1]][3]
    output_file <- sprintf('nd_%s.tif',year)
    
    # write result to file
    raster::writeRaster(result,
                        output_file,
                        overwrite = TRUE)
  } else {
    # return result
    return(result)
  }
}

The function download_daymet returns an error like:

Your requested data is outside DAYMET (temporal) coverage, or the server can't be reached. Check your the connection to the server or the coordinates and start/end years!

simply when the coordinates are outside of the spatial coverage. It think the error message should be edited to become accurate, otherwise it is confusing.

cr_daymet<-download_daymet_batch("CR_header_site.csv",
start = 1980,
end = 2019,
internal = TRUE)

Error in curl::curl_fetch_disk(url, x$path, handle = handle) :
Could not resolve host: daymet.ornl.gov
Downloading DAYMET data for: 134 at 51.0313052819043/-117.983126697915 latitude/longitude !

I'm so happy to find such a valuable package today. Thank you!

I am running this in Windows, which may be the source of the problem, but when I download data for both a single site, or for a batch, it is always saved as a .csv in the working directory.

I can work like this, but thought you might want to know.

Check error message and trapping of these. Seems that in some instances out of range values are not trapped (especially edge cases).

Change start_yr to start and end_yr to end for brevity.

The function is not working for param = "srad" with frequency = "monthly" or frequency = "annual". Tested with CRAN version and GitHub version. Also, param = "ALL" only downloaded tmin, tmax, prcp, and vp.

download_daymet_ncss(location = c(50, -114, 40, -97), start = 1980, end = 1980, param = "ALL", frequency = "monthly", mosaic = "na", path = "D:/NGP/data/daymet")

This returns the generic "data download failed" error message.

As per email from Michele:

The ORNL DAAC has posted a guide to the THREDDS NetCDF Subset Service. This REST API allows subsetting of netCDF datasets in coordinate space. Due to file sizes, it may be cumbersome to download the entire North American extent of the Daymet data. Subsetting the data through THREDDS is one option available for users who don't need access to entire data set. The THREDDS data server provides an interactive GUI web form which allows users to perform NCSS-based spatio-temporal subsetting and download of one Daymet variable_year per submission. These interactive requests can be automated through programmatic, machine-to-machine requests which involve the construction and submission of an HTTP request to the THREDDS data server through an extended URL with defined parameters. Gridded data subsets are returned in CF-compliant netCDF-3 or netCDF-4 formats.

https://daymet.ornl.gov/web_services.html

Remove the tidyr dependency, revert to base R for stability, use reshape() instead of gather() and spread().

VPD_normalized = (daylength / 24) * VPD

- This is `raster` library bug, not a daymetr or ORNL one!!! 
- In short, raster fails to read the CF-1.6 netcdf format properly. 
- Only GDAL >=2.3 will do so properly through readGDAL().

The default projection data in returned tiles seems to be faulty. Mainly the second parallel North is set to 45 where it should be 60. Below is a reproducible example of the issue.

First download an example tile with a recognizable outline for reference

# set libraries and lat / lon projection
library(daymetr)
library(raster)
library(maps)
lat_lon <-  CRS("+proj=longlat +datum=WGS84")

# download a test tile with a faulty projection parameter
if(!file.exists("~/tmax_1980_11754.nc")){
  download_daymet_tiles(tile = "11754", path = "~", param = "tmax")
}

Read in the data with default projection info.

s <- raster("~/tmax_1980_11754.nc")

# print the default projection
projection(s)

Reproject this data to lat / lon and plot together with a lat / lon representation of the tiles and the coastal outline around NYC. Note the mismatch!

s_ll <- projectRaster(s, crs = lat_lon)
plot(s_ll)
plot(tile_outlines, add = TRUE)
maps::map("world",
          add = TRUE)

Now overwrite the original project with one where the second parallel North is 60 degrees instead of 45 (as specified on the website), and generate the plot again. Notice that all things align nicely now.

projection(s) <- CRS("+proj=lcc +lon_0=-100 +lat_0=42.5 +x_0=0\
                     +y_0=0 +lat_1=25 +ellps=WGS84 +lat_2=60")
s_ll <- projectRaster(s, crs = lat_lon)

plot(s_ll)
maps::map("world",
     add = TRUE)
plot(tile_outlines, add = TRUE)

Thanks for the excellent interface to DAYMET. I've noticed (for the last month or so) that I can no longer get DAYMET data via download_daymet(), the function always issues an error: The server is unreachable, check your connection.. Any ideas?

Example.

# using latest development version from GitHub
library(daymetr)
x <- download_daymet(site = "Oak Ridge National Laboratories",
                     lat = 36.0133,
                     lon = -84.2625,
                     start = 1980,
                     end = 2010,
                     internal = TRUE)

Session Info:

R version 3.6.3 (2020-02-29)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] RColorBrewer_1.1-2      knitr_1.28              pbapply_1.4-2           sharpshootR_1.6.3      
 [5] hydromad_0.9-26         reshape_0.8.8           polynom_1.4-0           latticeExtra_0.6-29    
 [9] lattice_0.20-41         elevatr_0.2.0           rgeos_0.5-2             sp_1.4-1               
[13] reshape2_1.4.3          soilDB_2.5.6            aqp_1.22                Evapotranspiration_1.14
[17] zoo_1.8-7               daymetr_1.4.2          

loaded via a namespace (and not attached):
  [1] colorspace_1.4-1  ellipsis_0.3.0    class_7.3-16      rio_0.5.16        rgdal_1.4-8       rprojroot_1.3-2  
  [7] htmlTable_1.13.3  base64enc_0.1-3   fs_1.4.0          rstudioapi_0.11   circular_0.4-93   remotes_2.1.1    
 [13] mvtnorm_1.1-0     fansi_0.4.1       xml2_1.3.0        codetools_0.2-16  splines_3.6.3     pkgload_1.0.2    
 [19] jsonlite_1.6.1    Formula_1.2-3     cluster_2.1.0     png_0.1-7         compiler_3.6.3    httr_1.4.1       
 [25] backports_1.1.5   assertthat_0.2.1  Matrix_1.2-18     cli_2.0.2         acepack_1.4.1     htmltools_0.4.0  
 [31] prettyunits_1.1.1 tools_3.6.3       igraph_1.2.5      gtable_0.3.0      glue_1.3.2        dplyr_0.8.5      
 [37] Rcpp_1.0.4        carData_3.0-3     cellranger_1.1.0  raster_3.0-12     vctrs_0.2.4       ape_5.3          
 [43] nlme_3.1-145      xfun_0.12         stringr_1.4.0     ps_1.3.2          openxlsx_4.1.4    testthat_2.3.2   
 [49] lifecycle_0.2.0   devtools_2.2.2    MASS_7.3-51.5     scales_1.1.0      hms_0.5.3         parallel_3.6.3   
 [55] curl_4.3          memoise_1.1.0     gridExtra_2.3     ggplot2_3.3.0     rpart_4.1-15      stringi_1.4.6    
 [61] desc_1.2.0        plotrix_3.7-7     permute_0.9-5     e1071_1.7-3       checkmate_2.0.0   boot_1.3-24      
 [67] pkgbuild_1.0.6    zip_2.0.4         rlang_0.4.5       pkgconfig_2.0.3   purrr_0.3.3       htmlwidgets_1.5.1
 [73] processx_3.4.2    tidyselect_1.0.0  plyr_1.8.6        magrittr_1.5      R6_2.4.1          Hmisc_4.4-0      
 [79] mgcv_1.8-31       pillar_1.4.3      haven_2.2.0       foreign_0.8-76    withr_2.1.2       survival_3.1-11  
 [85] abind_1.4-5       nnet_7.3-13       tibble_3.0.0      crayon_1.3.4      car_3.0-7         progress_1.2.2   
 [91] jpeg_0.1-8.1      usethis_1.5.1     grid_3.6.3        readxl_1.3.1      data.table_1.12.8 vegan_2.5-6      
 [97] callr_3.4.3       forcats_0.5.0     digest_0.6.25     munsell_0.5.0     sessioninfo_1.1.1

I am interested in downloading daymet v4 data so that I can access the most recent year, 2020. Can you add this option to the download_daymet_ncss function or simply update it to use the current version?

Daymet produces daily data for their entire suite of variables, but only produces aggregate data for some. I want to create a script that could create this aggregate data for variables (particularly snow water equivalent). This script would likely be useful to other Daymet users.

@khufkens, do you think that this would be useful to include in this package?

Ideally the script would:

• Take daily data from the download_daymet functions
• Aggregate the data by a user-specified interval (e.g. month, year)
• Calculate the sum or mean (user-selected) for the aggregated data at the given interval
• Write the file as a tif, or output it to the work space

I have a start on this, but have not tested the outputs that this code produces. What I have so far is below.

#' Aggregates daily 'Daymet' gridded products to monthly or annual
#' time scales and creates a gridded (geotiff) output for easy
#' post processing and modelling. Optionally a raster object is
#' returned to the current workspace.
#' 
#' This is intended to work for swe, dayl and srad data, for which
#' ORNL-DAAC does not provide aggregated data.
#' 
#' Idea - what if it can be specified whether to calculate mean
#' or total. One or the other make sense for dayl, but there
#' may be applications for both with swe and srad...
#'
#' @param path full path location of the daymet tiles
#' @param product which tile to process
#' @param year which year to process
#' @param int interval to aggregate by. Options are "month" or "annual"
#' @param internal TRUE / FALSE (if FALSE, write the output to file)
#' using the Daymet file format protocol.

daymet_grid_agg = function(path ='.',
                           product = NULL,
                           year = NULL,
                           int = NULL,
                           internal = FALSE){
  
  # exit on missing tile
  if ( is.null(year)|is.null(int) ) {
    stop('No year or interval provided ...')
  }

  # list all files
  if(is.null(product)){
    data <- sprintf('%s/%s_%s_ncss.nc',path,int,year)
    output_file <- sprintf('%s/%s_%s_ncss.tif',path,int,year)
  } else {
    data <- sprintf('%s/%s_%s_%s.nc',path,int,year,product)
    output_file <- sprintf('%s/%s_%s_%s.tif',path,int,year,product)
  }

  # load everything into a raster brick
  data <- suppressWarnings(raster::brick(path))
  
  # extract time variable from data and covert to R-readable
  # date format
  dates <- as.Date(
           sub(pattern="X",
           replacement="",
           x=names(data)),
           format="%Y.%m.%d")
    
  ## use base r functions to aggregate data by month
  # get months for reference in aggregate function
  month <- as.list(months(dates))
  
  # data prep
  data.df <- raster::as.data.frame(data,
                                   row.names=TRUE,
                                   optional=FALSE)
  data.df <- as.data.frame(t(rbind(month,data.df)))
  
  ## aggregate bands by month and calculate mean
  result <- aggregate(x=data.df,
                      by=list(data.df[1]),
                      FUN=mean,
                      na.rm=FALSE)
  # doing a little cleanup
  result <- t(result)
  names <- result[1,]
  result <- as.data.frame(result[-1,])
  colnames(result) <- names
  
  # aggregate bands by month and calculate total
  
  ## OR use base r functions to aggregate data by year
  #same as above but aggregating by year
  
  # return all data to raster, either as a geotiff
  # or as a local object
  if (internal == FALSE){
      raster::writeRaster(result,
                          output_file,
                          overwrite = TRUE)
  } else {
    return(result)
  }
}

Hi, I would like to use your R code for downloading tiles from Daymet. I receive an error that the package is not available for R version 3.2.2. Has anyone else encountered this? I'm running on a windows machine through R Studio.

end <- as.numeric(format(as.Date(Sys.Date()),"%Y"))

daymet_test <- TRUE

while (inherits(daymet_test,"try-error")){
daymet_test <- try(daymetr::download_daymet(
start = end,
end = end,
internal = TRUE,
silent = TRUE))
end = end - 1
}

return(end)

Concerns functions with calculating T_mean for NCSS tiles, as well as annual and monthly summary downloads (can probably be integrate in original functions).

I having problems downloading a subset from a location in Hawaii.

library(daymetr)
download_daymet_ncss(
             location = c(19.3,-156.2,19.2,-155.1),
             start = 2019,
             end =  2020,
             param = c('tmin'),
             mosaic = "hi",
             path = "/tmp/data/daymet_bounding_box_19.3_-156.2_19.2_-155.1")
Creating a subset of the Daymet data
        be patient, this might take a while!

https://thredds.daac.ornl.gov/thredds/ncss/ornldaac/2129/daymet_v4_daily_hawaii_tmin_2019.nc

Downloading DAYMET subset: year: 2019; product: tmin

  |======================================================================| 100%
Error: Requested data download failed! 
             Common issues involve a mismatch between 
             the location and the mosaic used or downloads 
             exceeding 6GB in size.

I installed v1.7 of daymetr from cran.

If I check the THREDDS server at https://thredds.daac.ornl.gov/thredds/catalog/ornldaac/2129/catalog.html?dataset=2129/daymet_v4_daily_hi_tmin_2019.nc the ncss url is https://thredds.daac.ornl.gov/thredds/ncss/grid/ornldaac/2129/daymet_v4_daily_hi_tmin_2019.nc/dataset.html

Notice it uses hi instead of hawaii.

The correction at https://github.com/bluegreen-labs/daymetr/blob/master/R/download_daymet_ncss.r#L146-L151 should be removed.

The same issue is for puerto rico, it should be kept pr.

Code coverage assess how well code is tested and documented. I'm still missing a lot of comments in if() statements. Where:

# an error trap
 if(TRUE){
 something()
}

is not counted by codecov and should read:

# an error trap
if(TRUE){
 # error message
 something()
}

(I think) to be valid.

Similar issues apply to all if statements which aren't documented within the statement.

Downloads with the default query:
download_daymet_ncss(path = "~")

Which queries:
https://thredds.daac.ornl.gov/thredds/ncss/ornldaac/1328/1980/daymet_v3_tmin_1980_na.nc4?var=lat&var=lon&var=tmin&north=34&west=-82&east=-81.75&south=33.75&time_start=1980-01-01T12:00:00Z&time_end=1980-12-31T12:00:00Z&timeStride=1&accept=netcdf

Will results in different downloads for unknown reasons!

At times a query will download 54K:
Creating a subset of the Daymet data
be patient, this might take a while!
Downloading DAYMET subset: year: 1980; product: tmin
Downloading: 54 kB

Which only seems to contain one pixel of data (gdalinfo)

Driver: netCDF/Network Common Data Format
Files: tmin_daily_1980_ncss.nc
Size is 512, 512
Coordinate System is `'
Metadata:
  NC_GLOBAL#citation=Please see http://daymet.ornl.gov/ for current Daymet data citation information
  NC_GLOBAL#Conventions=CF-1.6
  NC_GLOBAL#geospatial_lat_max=12.01033018985965
  NC_GLOBAL#geospatial_lat_min=11.93425580314189
  NC_GLOBAL#geospatial_lon_max=-79.48915510825471
  NC_GLOBAL#geospatial_lon_min=-79.76852761774354
  NC_GLOBAL#History=Translated to CF-1.0 Conventions by Netcdf-Java CDM (CFGridWriter2)
Original Dataset = /daymet/V3/CFMosaic/1980/daymet_v3_tmin_1980_na.nc4; Translation Date = 2018-11-20T17:21:42.020Z
  NC_GLOBAL#references=Please see http://daymet.ornl.gov/ for current information on Daymet references
  NC_GLOBAL#source=Daymet Software Version 3.0
  NC_GLOBAL#start_year=1980
  NC_GLOBAL#Version_data=Daymet Data Version 3.0
  NC_GLOBAL#Version_software=Daymet Software Version 3.0
Subdatasets:
  SUBDATASET_1_NAME=NETCDF:"tmin_daily_1980_ncss.nc":lat
  SUBDATASET_1_DESC=[1x34] latitude (32-bit floating-point)
  SUBDATASET_2_NAME=NETCDF:"tmin_daily_1980_ncss.nc":lon
  SUBDATASET_2_DESC=[1x34] longitude (32-bit floating-point)
  SUBDATASET_3_NAME=NETCDF:"tmin_daily_1980_ncss.nc":tmin
  SUBDATASET_3_DESC=[365x1x34] tmin (32-bit floating-point)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)

A short time later it will download 1.3MB:
Creating a subset of the Daymet data
be patient, this might take a while!
Downloading DAYMET subset: year: 1980; product: tmin
Downloading: 1.3 MB

Which contains data (gdalinfo)

Driver: netCDF/Network Common Data Format
Files: tmin_daily_1980_ncss.nc
Size is 512, 512
Coordinate System is `'
Metadata:
  NC_GLOBAL#citation=Please see http://daymet.ornl.gov/ for current Daymet data citation information
  NC_GLOBAL#Conventions=CF-1.6
  NC_GLOBAL#geospatial_lat_max=34.05315791533045
  NC_GLOBAL#geospatial_lat_min=33.70520494123185
  NC_GLOBAL#geospatial_lon_max=-81.68213717040001
  NC_GLOBAL#geospatial_lon_min=-82.06525821633383
  NC_GLOBAL#History=Translated to CF-1.0 Conventions by Netcdf-Java CDM (CFGridWriter2)
Original Dataset = /daymet/V3/CFMosaic/1980/daymet_v3_tmin_1980_na.nc4; Translation Date = 2018-11-20T17:24:15.970Z
  NC_GLOBAL#references=Please see http://daymet.ornl.gov/ for current information on Daymet references
  NC_GLOBAL#source=Daymet Software Version 3.0
  NC_GLOBAL#start_year=1980
  NC_GLOBAL#Version_data=Daymet Data Version 3.0
  NC_GLOBAL#Version_software=Daymet Software Version 3.0
Subdatasets:
  SUBDATASET_1_NAME=NETCDF:"tmin_daily_1980_ncss.nc":lat
  SUBDATASET_1_DESC=[32x28] latitude (32-bit floating-point)
  SUBDATASET_2_NAME=NETCDF:"tmin_daily_1980_ncss.nc":lon
  SUBDATASET_2_DESC=[32x28] longitude (32-bit floating-point)
  SUBDATASET_3_NAME=NETCDF:"tmin_daily_1980_ncss.nc":tmin
  SUBDATASET_3_DESC=[365x32x28] tmin (32-bit floating-point)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)

I'm unsure if this is an httr issue or a Daymet issue.

ORNL backend seems to have an issue with delivering netcdf4 data on the NCSS system. The netcdf v3 support is still working however.

Clearly upsets PBR.

First of all, thanks fot his package, it has been very usefull.

Today I ran my routine to extract data and this error pop up. I dig a little and find out that apparently Daymet changed the download file and added some lines (or modified them). I don't know if this change is permanent, but changing the skip value in tile (line 182) to 4, alt to 5 (line 188) and data to 8 (line 195) fix the problem for the actual format.

When attempting to call the code:

batch.download.daymet(file_location='TestDoc.csv',start_yr=2011,end_yr=2015,internal=FALSE)

from an R script in RStudio (Version 0.99.903) the following error code is generated:

Error: could not find function "batch.download.daymet"

The newest version of DaymetR (downloaded 3 Oct 2016 10am AZ time) has been downloaded and loaded using the library() command. Important to note also is that when I run the code:

daymetr::batch.download.daymet(file_location='TestDoc.csv',start_yr=2011,end_yr=2015,internal=FALSE)

from the RStudio command window the following error code is generated:

Error in download.daymet(site = site, lat = lat, lon = lon, start_yr = start_yr, :
You requested data is outside DAYMET coverage,
the file is empty --> check coordinates!

The coordinates used were: 40.22458 -88.85675 (LAT/LON) which worked with a previous version of DaymetR and is not outside DAYMET coverage.

Attached is the TestDoc I used in my request code (converted from .csv to .xlsx for GitHub upload)
TestDoc.xlsx

2021 DAYMET data through June is now available: https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1904

Does the download_daymet() function need to be updated to access this data? When I use 'end = 2021' as an argument, I get the following error: End year exceeds valid data range!

Thanks for providing this great tool!