sentinel-5p / data-on-s3

A repository of documents and notebooks related to Sentinel-5P

MEEO_AWS_SENTINEL-5P

Jupyter Notebooks and assciated documents for working with Sentinel-5P Level 2 data stored in the AWS S3 bucket S3://meeo-s5p

Requirements

The following are some details about the software requirements to run the notbook.

Python 3 & Libraries

The notebooks included here use Python 3 and the floowing libraries:

• boto3. The library for working programmatically with AWS
• netcdf4. The library for reading netcdf file formats.
• gdal. The Geospatial Data Abstraction Library
• cartopy. A cartographic python library with Matplotlib support for visualisation.
• rasterio. Rasterio is a highly useful module for raster processing which you can use for reading and writing several different raster formats in Python. It depends on gdal.

Sentinel-5p Data

The Sentinel-5P data stored on AWS can be browsed here.

Summary

This data set consists of the Level 2 products generated from observations from the Sentinel-5P satellite. The Sentinel data is divided by type, described below:

• Near Real Time (NRTI): Products are available within 3 hours after sensing, shortly after that they will appear in the S3 bucket. These files are in the netCDF file format. Over time NRTI data gets replaced by OFFL data.

• Off Line (OFFL): For level 2 products this data is available about a week after the NRTI data, shortly afterwards they will appear in the S3 bucket. These files are in the netCDF file format and are of higher quality to the NRTI data.

• Cloud Optimised Geo-TIFF (COGT): These files are generated by MEEO from OFFL data shortly after the official Sentinel-5P data becomes available. These files are in the TIFF image file format.

• Reprocessed File (RPRO): From time to time there are new advances in product generation and ESA issues reprocessed files. These are converted into cloud optimised geo-tiff by MEEO.

The bucket contains the following S5P Level 2 products:

Parameter	Product Type (10 Characters wide)	Parameter used to generate COGT
Ozone (O3) total column	L2__O3____	ozone_total_vertical_column
Nitrogen Dioxide (NO2)	L2__NO2___	nitrogendioxide_tropospheric_column
Sulphur Dioxide (SO2) total column	L2__SO2___	sulfur_total_vertical_column_1k
Carbon Monoxide (CO) total column	L2__CO____	carbonmonoxide_total_column
Methane (CH4) total column	L2__CH4___	methane_mixing_ratio
Formaldehyde (HCHO) total column	L2__HCHO__	formaldehyde_tropospheric_vertical_column
UV Aerosol Index	L2__AER_AI	aerosol_index_354_388

The full data description document can be read here.

data-on-s3's Issues

Sentinel-3 (OL_2_WFR) Full Resolution data

The OL_2_WFR COG product is lower resolution than expected. It seems to have a resolution of around 1km, but I am expected 300m. Has this data been sub-sampled or is this correct?

is the Full Archive stored in your bucket?

Hey all,
do you store the full archive of Sentinel-3 (LST for example) in your bucket?
Best,
Zayd

Cloud product

The L2__CLOUD_ product appears to be available as well; is this deliberately omitted from the documentation? If not, minor documentation issue. Thanks!

Data updated till may for offline and NRTI

Hi,
I was checking the data in https://meeo-s5p.s3.amazonaws.com/?t=catalogs and it's updated till May 2020.
Is it going to updated anytime soon ? and if so what is the time lag

Missing and corrupted CH4 files

My group is developing a cloud-based methane inversion workflow on the AWS Cloud, and we would like to use the TROPOMI methane NetCDF files available in the Sentinel-5P Level 2 AWS S3 Bucket. While comparing the full list of OFFL and RPRO CH4 files in your S3 bucket with our local copies of the datasets, I noticed that there were several hundred missing files across the two methane datasets in the S3 bucket. These files all seem to correspond with pass times that span multiple calendar dates (but not all such files are missing). I was wondering if you knew why these files might be missing and if they could be added to the S3 bucket. I also identified three files in the S3 bucket that appear to be corrupted. I've attached a list of missing files (accurate through June 30th 2020 for the OFFL product), plus the corrupted ones.
missing_files.txt