metoffice / aws-earth-examples Goto Github PK

Example code of how to freely use Met Office's weather datasets through Earth on AWS.

License: Other

aws-earth-examples's Introduction

Met Office on AWS Earth

Some of the most important weather datasets produced by the Met Office are now available for free on Amazon Web Services, as part of the Earth on AWS initiative.

This repo contains example code of how to access these datasets. We also give you instructions on how to request support or contribute.

Getting Started

Data is available on AWS in S3 buckets.
Each forecast simulation is comprised of many files stored in NetCDF.
You can subscribe to SNS topics to be notified when new data is made available.
You can find a list of buckets and topics on the AWS Open Data Registry.

We have made some examples available in the form of Jupyter Notebooks.

Getting Support

The team behind the Met Office on AWS Earth project is happy to help you with any question you might have about the datasets or the example code. Please submit your question on the repo's Issues section and we will get back to you as soon as possible.

Contributing

You are welcome to contribute with example code, ideas or fixes. Here are the ways you can do it:

If you want to get seriously involved and become a member of the team, please get in touch with any of the contributors.

License

All Met Office's datasets made available through Earth on AWS can be freely used under an attribution, non-commercial, no derivatives Creative Commons license. Please read the full license text; by downloading any of the datasets you will be accepting the license.

aws-earth-examples's People

Contributors

Stargazers

Watchers

Forkers

rastandy jaynetra fotisz phanisankar-nidadavolu gmkprabhu1983 jackkelly hyunjay asalahuddin90 chicha1986 uk-gov-mirror vaquarkhan ravi98sankar dmcg michaelli916 rth-10

aws-earth-examples's Issues

Add examples for working with Lambert Azimuthal projection (UKV, MOGREPS-UK)

Code along the lines of the following can be used to turn the lambert azimuthal x / y co-ordinates into lat/lon. It'd be good to get a full example up to show one way of working with regional UK model data.

import pyproj
p = pyproj.Proj("+proj=laea +lon_0=-2.5 +lat_0=54.9 +x_0=0.0 +y_0=0.0 +ellps=GRS80")
# lon, lat = p(x, y, inverse=True)
lon, lat = p(-36000, -158000, inverse=True)
print (lat, lon)

53.47926025120268 -3.0422171310489534

Originally posted by @boristyukin in #19 (comment)

examples outdated?

hey guys, really neat architecture and documentation! very impressive work!

I noticed that MOGREPS-UK files now have projected x/y coordinates vs. lat/lons. Being new to all of this, I have hard time using projected x/y. I was only able to figure out that recent files are using
iris.coord_systems.LambertAzimuthalEqualArea

can you tell me how to get regular lat/lons from MOGREPS-UK so I can use them with Google Map? I need to extract data from cubes and store it in another system in a simple table for my needs.

For example, one of projected y/x in a cube is -158000.0,-36000.0 but google maps does not recognize this as valid coordinate.

I understand I need to change projection of the cube but have no clue now.

Would be great to update examples as well.

Insufficient descriptive information in messages on UK Deterministic

Transforming NETCDF file to Relational Databases

Hi, I am in a process to convert the MET Office Cube Data which is in NETCDF format to relational world for reporting and dashboards. Do you know how can i do that?

Known issue: no data available/no notifications

We were alerted yesterday that the service has been down since 14/12/2019.
I completed a fix for it this morning, but it will be 24 hours before the data can be released as the newly ingested data is live data.
You should start to see data from approx 10:30 UTC 21/12/2019 although, for less frequent models such as Global, it may be slightly longer before a run or the specific parameters you want come through. Your existing subscriptions will continue to function - the issue was with data getting into the system, not getting data out.

Do add a comment to this issue if you're unable to get data after this time.

Data info link in Open Data description unavailable

In the description of the dataset on AWS Open Data here, it says 'For information about the data see the Met Office website', and links to https://www.metoffice.gov.uk/services/data-provision/big-data-drive/wholesale/categories/planetary-scale-applications

Attempting to follow the link, I get a 'page not found' error.

Not able to get any message

Not able to get any message from below one

Description
Met Office UK Atmospheric Hi-Res Model (UKV) - 2km Deterministic
Resource type
SNS Topic
Amazon Resource Name (ARN)
arn:aws:sns:eu-west-2:021908831235:aws-earth-mo-atmospheric-ukv-prd
AWS Region
eu-west-2

SNS subscription

i try to create an aws SNS subscription but it doesnt work
I use:

arn:aws:s3:::aws-earth-mo-atmospheric-global-prd

and i use Region EU(LONDON)

regards

s3-website

Hi,

just a short question,
are you planing to give the data free over the s3-website mechanism

like: http://aws-earth-mo-atmospheric-ukv-prd.s3-website-eu-west-1.amazonaws.com/

regards
Ulrich

MOGREPS-UK SNS - Matching SNS name attribute to the description of data

Hello,

I have subscribed to the MOGREPS-UK SNS ARN (arn:aws:sns:eu-west-2:021908831235:aws-earth-mo-atmospheric-mogreps-uk-prd) and I am trying to determine the link between the data returned in the message and the description of the data from your website, referenced from the Registry of Open Data on AWS.

For example, in the JSON data below there is a "Name" attribute of "lwe_thickness_of_graupel_and_hail_fall_amount", but I can't see this at all in description of the data. Can you help me with the link (lookup list or similar), or point me in the direction for how I can work this out, or correct me in my interpretation of the logic if I've misunderstood things?

Many thanks, Rob.

{
   "Type":"Notification",
   "MessageId":"fb9e5411-a34a-54ae-b2ef-038a344bfc09",
   "TopicArn":"arn:aws:sns:eu-west-2:021908831235:aws-earth-mo-atmospheric-mogreps-uk-prd",
   "Message":"{\"model\": \"mo-atmospheric-mogreps-uk-prd\", \"long_name\": \"lwe_thickness_of_graupel_and_hail_fall_amount\", \"ttl\": 1596641086, \"time\": \"2020-07-30T23:00:00Z\", \"bucket\": \"aws-earth-mo-atmospheric-mogreps-uk-prd\", \"created_time\": \"2020-07-28T15:16:28Z\", \"name\": \"lwe_thickness_of_graupel_and_hail_fall_amount\", \"forecast_period_bounds\": \"205200 208800\", \"object_size\": 139662.0, \"forecast_period\": \"208800\", \"forecast_reference_time\": \"2020-07-28T13:00:00Z\", \"forecast_period_units\": \"seconds\", \"cell_methods\": \"time: sum\", \"key\": \"a6cd8ec46f068a5a13a58567b136b3f2c2595b2a.nc\", \"realization\": \"23 24 25\"}",
   "Timestamp":"2020-07-29T15:24:49.781Z",
   "SignatureVersion":"1",
   "Signature":"rY0t50duiMfuEVcNKhznucuubMQ5/dfPZpwdIck2t1FD6jYFmfiJJVxfDqFCAmYEF4TaIIE8AoZimryjl25geulL22dbQ7te+/g82frQQ8YPPKUPgqDKFPp/tLeAVOqadcVctpr7Osry1w7v/yv9u45IKDNxvnJy5NUyUyA2nan0H2hbkNU5e4DURbZSIBRrRQhWQKJapvMcqP0Hv+wz4Ilye8ba6Q3ZBXSe5RgKcM0MrBMbdSwvI3WhR3EFqhSGZNGRVcD/Yus+CowrreVSuWufDMVupcj4Ud08/CO92UHWtqb42u8dWjKQ/msmO2yi33V5CvWhNZjtr5rrZjihjw==",
   "SigningCertURL":"https://sns.eu-west-2.amazonaws.com/SimpleNotificationService-a86cb10b4e1f29c941702d737128f7b6.pem",
   "UnsubscribeURL":"https://sns.eu-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-2:021908831235:aws-earth-mo-atmospheric-mogreps-uk-prd:f39322bf-8548-4781-937e-d590d1487a51",
   "MessageAttributes":{
      "forecast_reference_time":{
         "Type":"String",
         "Value":"2020-07-28T13:00:00Z"
      },
      "name":{
         "Type":"String",
         "Value":"lwe_thickness_of_graupel_and_hail_fall_amount"
      },
      "model":{
         "Type":"String",
         "Value":"mo-atmospheric-mogreps-uk-prd"
      }
   }
}

GRIB files

Hi Guys
I'm an offshore sailor training for the Mini Transat hopefully next year and I use GRIB files a lot for weather routing. I realise this probably isn't a new question but I was wondering whether its possible to convert the Netcdf format into GRIB2 so I would be able to use the MO models in my routings?
Thanks
Julian

Number of SQS requests

Hello,

I have subscribed to the UK Deterministic model via SQS. I cannot control the number of messages using the queue configuration. I have used the service for about 5 days and I have already received over 700,000 messages using (70%) of the Free usage allowance.

Is there some way to control the messages for specific cycles (03Z and 15Z)? for instance?

Thanks,

Frank

UKV Deterministic service stopped

Hello,

The UKV Deterministic model output has stopped around midnight today. Could you please advise?

Many thanks,

Frank

Message cell_methods' time interval on UK Deterministic Model SQS

Hello,

For periods of 3 hourly data on the 03Z and 15Z UK Deterministic runs, the SQS messages have cell_methods with time interval of 1 hour rather than 3 hours.

Frank

List the version of Iris module in Example/1. Getting Started

some APIs has deprecated in Iris latest version, such as:

List the Iris version used in the example.

Regards!

Historic Data

Hi, I would like to get the SNS messages for historic data.

The example provided is only to get newly available data, however I am looking for data in 2018.

I can see the list of files in: https://s3.eu-west-2.amazonaws.com/aws-earth-mo-atmospheric-ukv-prd so can make some intelligent guesses, however it would be ideal if I can get the SNS Message data for them.

Any help greatly appreciated.

Thanks,

Martin

Cannot access data in MOGREPS-UK S3 bucket

Hi,
Believe it allows public access, I am able to list the bucket, but unable
to retrieve files from it:
aws-earth-mo-atmospheric-mogreps-uk-prd

Command:
aws s3 cp s3://aws-earth-mo-atmospheric-mogreps-uk-prd/
8732490bb7e5a710eea5ff25830aaad8a48c57cc.nc sample.nc
Got:
"fatal error: An error occurred (403) when calling the HeadObject
operation: Forbidden"

I know it is holiday season, appreciate your help!

Thanks,
Yiqun

How to download entire time step in single model run through S3?

Hello:

I have a high interest in "Met Office Global Atmospheric Hi-Res Model - 10km Deterministic" released on AWS Open data platform. However, while I reviewed data on S3 bucket, I was confused by file categorized. Seemed data was named by random characters. I tried to comprehend what involved in one file, but I guess it only single time step be released instead of whole forecast time.

Leon

how to get the complete data for the whole forecast period

Hello,
I want to download the "Met Office Global Atmospheric Hi-Res Model - 10km Deterministic" dataset, and I don't subscribe the AWS SNS yet. Instead, I got the keys on the bucket: https://s3.eu-west-2.amazonaws.com/aws-earth-mo-atmospheric-mogreps-g-prd, then according to the bucket and keys. I have downloaded the data every hour by URL(bucket+key) for a week, but today when I analyze the data I downloaded, I found that the dataset is not complete, for certain variable, such as "wind speed at 10m", I only got two files that forecasted at 2018-11-18 12:00, the time is not consequent for one forecasting reference time, I was wondering that have you uploaded all or part of the keys on "https://s3.eu-west-2.amazonaws.com/aws-earth-mo-atmospheric-mogreps-g-prd", and how can I get the complete hourly data for the whole forecast period?

Thanks
He Shuangshuang