jwagemann / era5_in_gee

Functions and Python scripts to ingest ERA5 data into Google Earth Engine

ERA5 reanalysis data in Google Earth Engine

This repository contains a set of functions and example scripts to ingest Copernicus ERA5 reanalysis data into Google Earth Engine based on manifest uploads.

The functions were developed during the process of making a subset of the ERA5 reanalysis data available in Google Earth Engine.

Workflow overview

The workflow consists of six major steps:

1. Downloading hourly data as daily files or monthly aggregates in NetCDF format from the Climate Data Store with cdsapi
2. Aggregating hourly files to daily means or sums (total precipitation) with xarray
3. Converting NetCDF data files to GeoTiff with gdal
4. Uploading hourly, daily and monthly GeoTiff files to Google Cloud Platform (GCP) with google-cloud-storage Python API
5. Creating image manifests (JSON-based files) describing the metadata and band names of the resulting Earth Engine asset
6. Ingesting data files uploaded to GCP as assets into Earth Engine with earthengine-api and manifest uploads

Repository content

• ERA5 in GEE functions
  • Python script
  • Jupyter notebook
• Example workflows
  • Hourly assets
  • Daily assets
  • Daily single asset
  • Monthly assets
• Example manifest files
  • Hourly multiple assets
  • Daily multiple assets
  • Daily single asset
  • Monthly multiple assets
• Example scripts for
  • Retrieve ERA5 reanalysis from the CDS with cdsapi
  • Delete blobs from GCP buckets

Python packages required

• cdsapi
• EarthEngine Python API
• google-cloud-storage Python API
• gdal
• xarray

Naming convention and folder structure

The scripts base on the following naming convention and folder_structure:

Naming Convention

Data is downloaded from the Climate Data Store with the following syntax:

• era5_ + name_of_variable_ + year_ + month_ + day + .nc
  • Example for a file of 2m dewpoint temperature from 1 January 1980: era5_2m_dewpoint_temperature_1980_01_01.nc

Folder Structure:

(example for 2m air temperature)

• era5_t2m
  • nc
    • hourly
      • year (e.g. 1979)
    • daily
      • year (e.g. 1979)
    • monthly
      • year (e.g. 1979)
  • tiff
    • hourly
      • year (e.g. 1979)
    • daily
      • year (e.g. 1979)
    • monthly
      • year (e.g. 1979)
• manifests
  • era5_hourly
    • year (e.g. 1979)
  • era5_daily
    • year (e.g. 1979)
  • era5_monthly
    • year (e.g. 1979)
  • manifest_template.json

References

• ERA5 reanalysis data in the Climate Data Store
• Google Earth Engine
• ERA5 DAILY | GEE Public Data Catalog
• ERA5 MONTHLY | GEE Public Data Catalog
• ERA5 in GEE | Slides from the EarthEngineVirtualMeetup in Dec 2019
• ERA5 reanalysis data available in Earth Engine - ECMWF Newsletter Number 162 Winter 2020

License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

era5_in_gee's Issues

Offset in image

First, thank you for putting these tools together and making them public! This is really helpful.

I was comparing the ERA5 and ERA5-Land image collections and noticed that there seems to be a slight shift or offset in the ERA5 images (https://code.earthengine.google.com/c71edc087f903a9cb61b55e1bfaf0e3f) that seemed to be about a half an ERA5 cell down and to the right.

Looking through these scripts I noticed that you are defining the geotransform is -180.0, 0.25, 0.0, 90.0, 0.0, -0.25 (https://github.com/jwagemann/era5_in_gee/blob/master/era5_in_gee_functions.py#L144). Since the latitude and longitude values in the NetCDF are for the "center" of the cells but the GeoTIFF is expecting the geotransform to be the upper-left corner "edge", shouldn't the geotransform be something like -180.125, 0.25, 0.0, 90.125, 0.0, -0.25?