metno / nwpdocs Goto Github PK

View Code? Open in Web Editor NEW

14.0 55.0 4.0 2.7 MB

Description of the numerical weather prediction products at MET Norway

License: GNU General Public License v3.0

nwpdocs's Introduction

NWPdocs

Description of the numerical weather prediction products at MET Norway

Check out the wiki page for more information.

nwpdocs's People

Contributors

Stargazers

Watchers

Forkers

johtoblan tnipen andreasbordvik haukhauk

nwpdocs's Issues

Strange behaviour of MAE in temperature prediction

Dear developers,
I checked mean average error of metno historical predictions for year 2021 and compared to fmi-station data. One would expect prediction to get worse for a longer prediction time but that seems not to be the case for temperature and wind speed (for pressure it seems ok).

Do I have a bug in my code or can this be explained some other way, a typical plot of MAE with respect to time (0-54h) is below (here for temperature).

High precipitation for October 23rd 2016

Hi,

First of all, thank you for all of your great products!

I'm downloading data from the MET post-processed products Rerun version 2 product for October 23rd, 2016 for Denmark. I find that some hours have precipitation that seems unreasonable high leading to daily precipitation above 400 mm (see the figure below for the sum of the hourly precipitation).

If one looks at the hour to hour precipitation for the coordinate (in lambert conformal conic the coordinate is (-291442.2, -764322)) with the highest precipitation they seem to be just consistently high, no specific hour stands out. Do any of you know why one would observe such high values?

Thanks!

Jens

Provide a recipe on how to calculate wind at 100m height

X, Y = proj.transform(lat, lon) ?

Dear developers,
I have to change the order of lat lon in
https://github.com/metno/NWPdocs/wiki/Examples#plotting-timeseries-for-the-nearest-neighbour-of-a-lonlat-point
to get it work for Helsinki (MET Nordic dataset version3)

With code below helsinki position is ok (lat, lon order changed in transform compared to example above)

ds=xr.open_mfdataset("met_analysis_1_0km_nordic_20200912T12Z.nc")
lat=60.205791,
lon=24.929186,
crs= "+proj=lcc +lat_0=63 +lon_0=15 +lat_1=63 +lat_2=63 +no_defs +R=6.371e+06",
xname='x',
yname='y',
proj = pyproj.Proj.from_crs("4326", crs)
X, Y = proj.transform(lat, lon)
# Find nearest neighbour
x = ds.variables[xname].to_numpy()
y = ds.variables[yname].to_numpy()
Ix = np.argmin(np.abs(x - X))
Iy = np.argmin(np.abs(y - Y))

MEPS deterministic files from 2023-12-05 00:00 to 2024-03-05 21:00 are missing

Hello metno,

I have noticed that the MEPS deterministic files from meps_det_2_5km_20231205T00Z.nc to meps_det_2_5km_20240305T21Z.nc are missing from https://thredds.met.no/thredds/catalog/meps25epsarchive/catalog.html.

Is it possible to know the reason of the recall and when they will be available again? Many thanks in advance.

Where are the historical forecasts? Are they only in the operational and not in reruns?

In the documentation in
https://github.com/metno/NWPdocs/wiki/MET-Nordic-dataset#files
it says:
"The Archive/Rerun version 3 folder contains the most recent rerun of analyses and forecasts. "

Questions: where are the forecasts? I only find analyses.
For instance
wget https://thredds.met.no/thredds/fileServer/metpparchivev3/2020/06/01/met_analysis_1_0km_nordic_20200601T12Z.nc gives me data
but
wget https://thredds.met.no/thredds/fileServer/metpparchivev3/2020/06/01/met_forecast_1_0km_nordic_20200601T12Z.nc gives
ERROR 404:

If one browses manually analysis files are there but not the historical forecasts

Surface downwelling shortwave data values

Hi!
I extracted data from the operational dataset (https://thredds.met.no/thredds/catalog/metpparchive/2022/catalog.html) for the period 01/01/2021-01/07/2022. For the last few months (since about mid-March) the integral_of_surface_downwelling_shortwave_flux_in_air_wrt_time file has unexpected values.

wind direction/speed vs x_wind/y_wind

Some files (e.g. https://thredds.met.no/thredds/catalog/metpparchive/2019/03/04/catalog.html?dataset=metpparchive/2019/03/04/met_analysis_1_0km_nordic_20190304T10Z.nc) have x_wind/y_wind variables, while others have wind_speed/wind_direction

"wind_direction_10m.standard_name" states "wind_from_direction"

is x_wind/y_wind intended as the "wind_from_direction" as well?

Update projection documentation

We are removing the proj4 attribute from the projection variable. This must be reflected in the wiki examples.

Radiation data in Post Processed Analysis Product

There seems to be many outliers in the radiation variable for the analysis products (http://thredds.met.no/thredds/catalog/metpparchivev1/catalog.html) . Here I made the assumption that the radiation variable represents deaccumulated values and that they represent deaccumulated values over a period of 1 hour. So I just divided the values by 3600s to get W/m^2. But I see that some values are too high. Look at the plots below. I just took one point inside one of Statkrafts catchments and plotted the radiation for the whole period (2013.09.01-2018.09.01). I have also included a plot where only the values from one of the years with the outliers are plotted.
@tnipen , @cristianlussana do you know the cause for this?

file with missing variables 2016-11-08 02:00Z

Missing variables in file MET Nordic analysis > Archive > Rerun_version_1 > http://thredds.met.no/thredds/dodsC/metpparchivev1/2016/11/08/met_analysis_1_0km_nordic_20161108T02Z.nc.html

3d data for Finland is rather sparse...

Hei,
I'd like to get 3d data for Finland. Tried from "https://thredds.met.no/thredds/dodsC/mepslatest/meps_det_vc_2_5km_latest.nc"
In principle this data is good, but lat lon grid is pretty sparse in Finland (about 100km from one point to another...).

Is there any 3d data from metno that would be usable in Finland?
Or could you point out where in (Finland?) to get this data for free?

terveisin, Markus

Spatial grid values (x, y) have changed since `meps_det_2_5km_20230130T09Z.nc`

Hello metno,

Since 2023-01-30 09:00, the MEPS files have a slightly different spatial grid values. They no longer are exactly spaced in increments of 2500 m.

import xarray as xr
import pandas as pd

T06 = xr.open_dataset(
    'https://thredds.met.no/thredds/dodsC/meps25epsarchive/2023/01/30/meps_det_2_5km_20230130T06Z.nc'
)
T09 = xr.open_dataset(
    'https://thredds.met.no/thredds/dodsC/meps25epsarchive/2023/01/30/meps_det_2_5km_20230130T09Z.nc'
)
diff_stats = pd.concat(
    objs=[
        T06['x'].diff(dim='x').to_series().rename('T06_x').describe(),
        T06['y'].diff(dim='y').to_series().rename('T06_y').describe(),
        T09['x'].diff(dim='x').to_series().rename('T09_x').describe(),
        T09['y'].diff(dim='y').to_series().rename('T09_y').describe(),
    ],
    axis='columns'
)
print(diff_stats)

Is this intended or an error? Thanks in advance

ground height information (oreography) for 2.5 km MEPS grid

Dear Developers,

Could not find oreography info in MEPS dataset nc files (info that near Galdhøpiggen it data should show about 2400m).
Is this information available somewhere in the same 2.5km MEPS grid?

Terveisin, Markus

Update precipitation parameter specifications

Under https://github.com/metno/NWPdocs/wiki/MEPS-dataset#meps-parameters we should add description of the various precipitation parameters and which we recomment to use. Possibly some metadata are wrong as well

Python scripts to download nc files are not working?

Dear developers,

I try to run (copy pasted from https://github.com/metno/NWPdocs/wiki/Examples#extracting-and-plotting-a-variable )

import netCDF4
filename = "https://thredds.met.no/thredds/dodsC/metpparchive/"+\
           "2021/08/01/met_analysis_1_0km_nordic_20210801T12Z.nc"
ncfile   = netCDF4.Dataset(filename)

This results to

Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url
Traceback (most recent call last):
  File "/home/hu-mka/git/linux-weatherstation/machinelearning/dev/metno_demo1.py", line 7, in <module>
    ncfile = netCDF4.Dataset(filename)
  File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: b'https://thredds.met.no/thredds/dodsC/metpparchive/2021/08/01/met_analysis_1_0km_nordic_20210801T12Z.nc'

pip command given (as instructed in https://github.com/metno/NWPdocs/wiki/Examples#extracting-and-plotting-a-variable)
pip install matplotlib netCDF4 numpy pyproj xarray

pip freeze shows

certifi==2022.6.15
cftime==1.6.1
cycler==0.11.0
fonttools==4.37.1
kiwisolver==1.4.4
matplotlib==3.5.3
netCDF4==1.6.0
numpy==1.23.2
packaging==21.3
pandas==1.4.3
Pillow==9.2.0
pyparsing==3.0.9
pyproj==3.3.1
python-dateutil==2.8.2
pytz==2022.2.1
six==1.16.0
xarray==2022.6.0

I have ubuntu 22.04

Are there some special versions of the libraries one should use? I remember this did work earlier (but with different versions of the libraries).

Terveisin, Markus

MEPS global attributes show wrong geospatial longitude

geospatial_lon_min: 18.1, should be -18.1

e.g. https://thredds.met.no/thredds/dodsC/meps25epsarchive/2021/01/01/meps_det_2_5km_20210101T12Z.nc.html

Wrong metadata for radiation data in post-processed forecast product

Some of the files in the post-processed forecast dataset (http://thredds.met.no/thredds/catalog/metpparchivev1/catalog.html) have unit for the radiation which does not match with the magnitude of the values in this variable. In the example below the unit in the metadata is W m^-2 while the variable seem to represent aggregated (integral) values. @tnipen, @cristianlussana, @ivarsei can this be fixed?

f="met_forecast_1_0km_nordic_20140701T06Z.nc"
ds=nc.Dataset(f,'r')
rad = ds.variables['integral_of_surface_downwelling_shortwave_flux_in_air_wrt_time']
rad
Out[32]:
<class 'netCDF4._netCDF4.Variable'>
float32 integral_of_surface_downwelling_shortwave_flux_in_air_wrt_time(time, y, x)
    coordinates: longitude latitude
    units: W/m^2
    standard_name: surface_downwelling_shortwave_flux_in_air
    grid_mapping: projection_lcc
unlimited dimensions: time
current shape = (61, 1544, 1270)
filling on, default _FillValue of 9.969209968386869e+36 used
rad[:,:,0].max()
Out[33]: 75324344.0

Restructure examples

Failed file reads

I try to read data, one day at the time, and I get the following issues. I do not know which of 24 files in each file that ails
Day: 0
http://thredds.met.no/thredds/dodsC/metpparchivev1/2017/10/17/met_analysis_1_0km_nordic_20171017T00Z.nc
Day: 1
http://thredds.met.no/thredds/dodsC/metpparchivev1/2017/10/18/met_analysis_1_0km_nordic_20171018T00Z.nc
Read failed: 'air_pressure_at_sea_level' <class 'KeyError'>
Day: 2
http://thredds.met.no/thredds/dodsC/metpparchivev1/2017/10/19/met_analysis_1_0km_nordic_20171019T00Z.nc
Read failed: NetCDF: Not a valid ID <class 'RuntimeError'>
etc.
Day: 1
http://thredds.met.no/thredds/dodsC/metpparchivev1/2019/03/01/met_analysis_1_0km_nordic_20190301T00Z.nc
Read failed: 'time' <class 'KeyError'>

Does anyone know where these errors come from? The "Not a Valid ID" message seems to coma and go unpredictable.

Read failed: NetCDF: Not a valid ID
http://thredds.met.no/thredds/dodsC/metpparchivev1/2019/02/26/met_analysis_1_0km_nordic_20190226T00Z.nc
Read failed: 'time'
http://thredds.met.no/thredds/dodsC/metpparchivev1/2019/03/01/met_analysis_1_0km_nordic_20190301T00Z.nc
Read failed: NetCDF: Not a valid ID
http://thredds.met.no/thredds/dodsC/metpparchivev1/2019/03/13/met_analysis_1_0km_nordic_20190313T00Z.nc
Read failed: NetCDF: Not a valid ID
http://thredds.met.no/thredds/dodsC/metpparchivev1/2019/04/03/met_analysis_1_0km_nordic_20190403T00Z.nc
Read failed: NetCDF: Not a valid ID
http://thredds.met.no/thredds/dodsC/metpparchivev1/2019/04/06/met_analysis_1_0km_nordic_20190406T00Z.nc
Read failed: NetCDF: Not a valid ID
http://thredds.met.no/thredds/dodsC/metpparchivev1/2019/04/20/met_analysis_1_0km_nordic_20190420T00Z.nc
Read failed: NetCDF: Not a valid ID
http://thredds.met.no/thredds/dodsC/metpparchivev1/2019/04/26/met_analysis_1_0km_nordic_20190426T00Z.nc

files with missing data

Missing variables in file MET Nordic analysis > Archive > Rerun_version_1 >

Jan 5, 2019, 2:00 missing data
Jan 7, 2019, 18:00 missing wind data

Missing files in operational

Dear All,
Are there many files missing in the archive of operational forecasts,
could not find at least these:

https://thredds.met.no/thredds/dodsC/metpparchive/2018/03/07/met_forecast_1_0km_nordic_20180307T15Z.nc
https://thredds.met.no/thredds/dodsC/metpparchive/2018/03/09/met_forecast_1_0km_nordic_20180309T16Z.nc

Terveisin, Markus

The precipitation_amount in Helsinki on 15th of June 2022 6 am is 9.96921E+036

Dear Developers,
I download file https://thredds.met.no/thredds/dodsC/metpparchive/2022/06/15/met_forecast_1_0km_nordic_20220615T00Z.nc
And cut coordinates to Helsinki region
The other values below look ok, but the precipation amount (sorry its off in the copy paste below) at the first time stamp is 9.96921E+036

Whats going on? ( I read into xarray doing cut in projected coordinates and taking nanmean over spatial coordinates, then to pandas and to csv shown below)

Here is minimal example code to reproduce the result:

import xarray as xr
import pyproj

ds = xr.open_dataset(
    "https://thredds.met.no/thredds/dodsC/metpparchive/2022/06/15/met_forecast_1_0km_nordic_20220615T00Z.nc"
)
proj = pyproj.Proj(ds.projection_lcc.proj4)
X, Y = proj(longitude=25.1, latitude=60.2)
dm = 1500
east = X + dm
west = X - dm
north = Y + dm
south = Y - dm
ds = ds.sel(x=slice(west, east), y=slice(south, north))
ds = ds.mean(dim=("x", "y"), skipna=True)
df = ds.to_dataframe()
df.to_csv("helsinki_20220615_00.csv")

Terveisin, Markus

time,projection_lcc,forecast_reference_time,altitude,land_area_fraction,air_temperature_2m,air_pressure_at_sea_level,relative_humidity_2m,cloud_area_fraction,wind_speed_10m,integral_of_surface_downwelling_longwave_flux_in_air_wrt_time,integral_of_surface_downwelling_shortwave_flux_in_air_wrt_time,precipitation_amount,wind_speed_of_gust,wind_direction_10m
2022-06-15 00:00:00,-2147483647.0,2022-06-15,5.0,0.7633333,287.91003,100622.59,0.7065686,0.9068469,5.9370203,7261794.0,172984.89,9.96921e+36,10.124577,322.83487
2022-06-15 01:00:00,-2147483647.0,2022-06-15,5.0,0.7633333,287.61575,100620.016,0.74328095,0.780403,5.9827876,8463072.0,172984.89,0.0,10.053935,322.14633
2022-06-15 02:00:00,-2147483647.0,2022-06-15,5.0,0.7633333,287.3064,100627.5,0.73399603,0.6613028,5.8526263,9633541.0,230882.69,0.0,10.06389,320.86154
2

Missing data in Rerun version 2

I've found some periods with missing data in the latest rerun.

For 2019 there is one period of 30 hours missing from thredds:

2019-03-02T21Z to 2019-03-04T02Z

For 2020 the files that I could not read are:

2020-01-13T18Z to 2020-01-14T23Z
2020-01-27T19Z to 2020-01-31T23Z
2020-02-04T03Z to 2020-02-05T02Z
2020-07-07T14Z to 2020-07-07T23Z
2020-07-11T03Z to 2020-07-14T23Z

Error:curl error: Problem with the SSL CA cert (path? access rights?)

Was trying to run copy pasted example from metno examples:

import xarray
import numpy as np

opendap_url = "https://thredds.met.no/thredds/dodsC/aromearcticlatest/"+\
              "archive/arome_arctic_det_2_5km_latest.nc"
#Ask for time=0 to make the calculation smaller, height0=0 to remove the 4th dimension
ds = xarray.open_dataset(opendap_url).isel(time=0,height0=0)

# Formula to calculate pressure from hybrid: p(n,k,j,i) = ap(k) + b(k)*ps(n,j,i)"
ap = ds.ap
b = ds.b
surface_pressure = ds.surface_air_pressure

# Note that k = 0 is top of atmosphere (ToA), and k = 64 is surface
pressure_at_k = (ap + (b*surface_pressure))
temperature_at_k = ds.air_temperature_ml
R = 287.058
g = 9.81

height_at_k = xarray.full_like(pressure_at_k, fill_value=0)

max_k = len(height_at_k.hybrid) -1
# Compute the height of the lowest model level
height_at_k.values[max_k,:,:] = (R*temperature_at_k.isel(hybrid=max_k))/g
height_at_k.values[max_k,:,:] *= np.log(surface_pressure / (pressure_at_k.isel(hybrid = max_k)))

# Loop over the rest of the model levels
for hybrid in range(max_k - 1,-1,-1):
    height_at_k.values[hybrid,:,:] = (R*temperature_at_k.isel(hybrid=hybrid))/g
    height_at_k.values[hybrid,:,:] *= np.log((pressure_at_k.isel(hybrid = hybrid + 1)) / \
                                             (pressure_at_k.isel(hybrid = hybrid)))
    height_at_k.values[hybrid,:,:] += height_at_k.isel(hybrid=hybrid + 1)

but got error
Error:curl error: Problem with the SSL CA cert (path? access rights?)

google says this is server side error due to
outdated SSL certificate(s) for cURL installed on the server.

terveisin, Markus

Accumulated rainfall

There are several parameters with accumulated values (accumulated total precipitation, accumulated rainfall at surface, etc.) in the deterministic MEPS control data files. Over which time frame is the accumulated values calculated? We are interested in rain intensity per hour, how do we relate the accumulated rainfall to rain intensity in mm/hr?

Finer gridded vertical cross section data

First off all, thanks alot for providing the MEPS data free of charge. It has been a great playground for me lately :)

I am a enthusiastic paraglider pilot, and have developed a small service (see https://flysogn.streamlit.app/ and select "forecast") that shows wind and thermal activity forecasted by METS so that we can use it for planning and expectations. This seems to work quite well.

However, after some investigations, I have realized that you output a quite sparse grid in your meps_lagged_6_h_vc_2_5km and meps_det_vc_2_5km files.

For example, in my local area (sogndal), the two closest points are 10 and 16 km away, respectively. This would of course be better if I got proper output of the 2.5km grid (I assume an interpolation here would not be as good).

Is there a possibility to add this as model outputs? Or am I thinking wrongly here? The way I load the forecast can be seen [here].(https://github.com/simeneide/flysogn/blob/main/meps.py#L21).

Secondly, is it possible to add lat lon dimensions to the dataset? The way I can filter down a position now is through the following code (which is slow):

    subset = dataset.where(
        (np.abs(dataset.latitude - lat) < tol) & (np.abs(dataset.longitude - lon) < tol), 
        drop=True
    )

Parameters MET Nordic

Are the parameters cloud_area_fraction and integral_of_surface_downwelling_shortwave_flux_in_air just the nearest neighbour of the raw model output at 2.5 km resolution? Or is there some post-processing or downscaling applied to the analysis and forecast products like nearest neighbourhood area selection? In the pdf about MEPS linked to on the MEPS AROME Arctic wiki page, it is written that post-processed cloud area fractions are the median value of a neighbourhood area of 37.5x37.5 km, but I notice that the information is a few years old and applied to the old pp products.

metno / nwpdocs Goto Github PK

nwpdocs's Introduction

NWPdocs

nwpdocs's People

Contributors

Stargazers

Watchers

Forkers

nwpdocs's Issues

Recommend Projects

Recommend Topics

Recommend Org