Feature Request: Finer vertical resolution about open-meteo

[patrick-zippenfenig]

Hi,
I would like to add more ICON model levels, however there are 2 issues:

1. It requires high CPU resources to convert the ICON grid to regular grid. Adding more vertical levels leads to processing times of multiple hours of ICON. This can be improved by better parallelising data processing, but it still requires a lot of CPU time. Currently I am using smaller virtual machines, one for each model, to process data. Afterwards data is distributed to API nodes. This is designed to make the processing system more fault tolerant and easier to scale. But of course larger VMs are more expensive.
2. Disk space and bandwidth required. Coping and storing data to all API nodes would significantly increase. This is a scalability issue, because each API nodes requires 2 TB "hot" data to serve forecast data from the past 90 days and 15 days forecast. Most API nodes are bare metal server with large NVMe SSDs. Regular VMs do not handle high load well. It is not easy to scale up the Forecast API nodes.

There are a couple of options. I could setup a, dedicated API endpoint (e.g. "Atmospheric Forecast API") that contains all pressure and model level data. This way I only need a smaller number of nodes with large disks (4TB NVMe should be ideal).

As you might imagine this is a bit more work increases operational cost, therefore I would only start working on it, if there is larger interest.

Alternatively, you can also try to adapt the Open-Meteo source code to add model levels. I am currently mapping wind levels 80/120 m to some model levels. It is therefore possible to add more levels in the code and download data self-hosted. However, ICON model levels are technically defined on varying elevation. Therefore the code may require larger adaptation to get correct results for model levels.

[MrMorice]

Hi @patrick-zippenfenig! Thank you very much for the super quick and detailed answer. Offering all of the great open-meteo functionality for this really seems like a huge project. As I only need the data for several hundred locations I'll probably go with a custom fetch, process and throw away approach as I don't have the hardware for running the full open-meteo. Have a great week and thank you for this amazing project! Best, Mo