panodata / dwdweather2 Goto Github PK

Hi,

i'm getting the older columnnames e.g. "soiltemp_temperature_005" when requesting data on macos. When running on windows, i get the newer "soil_temperature_005" for the same value.

Is there a way to control which columnnames are used?

thanks!

markus

@JohannesRol follows a more object-oriented library interface approach with his dwdopendata package. There, you will create a location object from a factory function obtaining the geolocation and subsequently be able to invoke appropriate methods on that object for acquiring actual weather information.

We like that approach, so it would be nice to see a similar interface here.

Original example

import dwdopendata as dwd

# Acquire location object from coordinates of nearest station.
location = dwd.location(51.898, 8.9876)

# Acquire wind speed data for specific location and time range from specified resolution dataset.
wind_speed = location.wind('2019-05-01T00:00:00', '2019-05-02T00:00:00', '10min')

The library usage example is wrong. It lacks the required parameter resolution is missing and the hour parameter of query was renamed.
It should be the following:

from datetime import datetime
from dwdweather import DwdWeather

# Create client object.
- dw = DwdWeather()
+ DW = DwdWeather(resolution="hourly") 

# Find closest station to position.
closest = dw.nearest_station(lon=7.0, lat=51.0)

# The hour you're interested in.
# The example is 2014-03-22 12:00 (UTC).
query_hour = datetime(2014, 3, 22, 12)

- result = dw.query(station_id=closest["station_id"], hour=query_hour) 
+ result = dw.query(closest["station_id"], query_hour)
print(result)

Took me several hours to figure that out. :/

Hello @amotl,

first of all thank you for your great work.
After exploring dwdweather2, I noticed that solar_* returns always None.

However, e.g. for station ID 5856 (hourly resolution) there clearly is data available (see https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/solar/)

e.g:

query_hour = datetime(2019, 7, 17, 11)
result = dw.query(station_id=5856, timestamp=query_hour)
for item in result.items():
     print(item)

should return solar-values unequal None:
5856;2019071711:12; 1; -999; 57.0; 321.0; 60; 28.00;2019071712:00;eor

Am I missing something here?
Kind regards

I have recurring queries on several stations, each time asking the same data. However, for some stations, the information seems not to be stored correctly.
Instead of using local sqlite db, the data is queried from the ftp server.

dw = DwdWeather(resolution="hourly", categories='air_temperature')
dw.categories = [{'key': 'TU', 'name': 'air_temperature'}]

Stations i'm querying e.g.:

4275 Rotenburg (Wümme)
3667 Nürnberg-Netzstall

Hi,

I am currently working with the dwdweather2 package and everything is working just fine. Thanks for the nice package!

What I want to get as a result is daily data from different stations. Until now, the package only supports hourly and 10-minutes resolution. Is it possible to add the daily data functionality to the package? Then I don't need to aggregate the data and that would save me a lot of time.

The data is already available in CDC.

Many thanks in advance.

This looks promising, thanks! From the code it appears to be fetching observations data. Are there any plans to add functionality to access forecast data too?

Hey!

Just wanted to see if you would entertain adding Continuous Integration via Microsoft Azure. Looking to try out Azure Pipelines and would be happy to add this to the project.

e.g. https://azure.microsoft.com/en-in/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/

System:

• Windows 10
• Python 3.7

I run in to the following problem when running pip install dwdweather2:

PS D:\workspaces\robotframework-dwdweather> pip install dwdweather2
Collecting dwdweather2
  Using cached https://files.pythonhosted.org/packages/b9/0c/fb48f9e43f263a493f0a5e13d00bbe6ca0a31d11958b444836d218951a25/dwdweather2-0.8.0.tar.gz
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Daidalos\AppData\Local\Temp\pip-install-pa4hwo8d\dwdweather2\setup.py", line 6, in <module>
        README = open(os.path.join(here, 'README.rst')).read()
      File "d:\programs\python\python37\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2062: character maps to <undefined>
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in C:\Users\Daidalos\AppData\Local\Temp\pip-install-pa4hwo8d\dwdweather2\

Installation works with dwdweather (is this project related to you) but not with dwdweather2.

Installing dwdweather:

PS D:\workspaces\robotframework-dwdweather> pip install dwdweather
Collecting dwdweather
  Downloading https://files.pythonhosted.org/packages/ee/4f/4bd3ceff7b6158133d6e7cbfd68f27e4d72fd2f80c83d032944e16bbbb2f/dwdweather-0.7.tar.gz
Installing collected packages: dwdweather
  Running setup.py install for dwdweather ... done
Successfully installed dwdweather-0.7

@JohannesRol has an implementation to build a list of multiple stations sorted by distance to a given geolocation [1]. It would be nice to add a similar feature here. See also #14.

[1] https://github.com/JohannesRol/dwdopendata/blob/8c412ba9/dwdopendata.py#L32-L58

Am I mistaken or is this module about retrieving historical data from DWD. They seem to have several "interfaces". If I see correctly, the files used to query in this library are for data that is at least a few hours old.
Is that right?

Dear @mmaelicke,

within jeremiahpslewis/dwdbulk#5, you asked for RADOLAN support. Most probably, reading the binary files would prove to be more efficient.

Integrating wradlib [2] or radproc [3] into dwdweather2 should be possible, but would it make any sense? I am happy to receive any thoughts about this.

With kind regards,
Andreas.

[1] https://opendata.dwd.de/climate_environment/CDC/grids_germany/hourly/radolan/recent/bin/
[2] https://wradlib.org/
[3] https://github.com/jkreklow/radproc

@jlewis91 maintains a well done implementation at [1]. While it is somehow derived from this project, it is based on modern Python (Python 3.7+ & Pandas). Thanks for this, @jlewis91!

It currently seems to implement acquisition of "observation" data [2] of

• precipitation for the 1_minute resolution
• air_temperature, precipitation, solar, wind, extreme_temperature and extreme_wind for the 10_minutes resolution

as well as acquisition of "forecast" data.

It would be cool to join both projects again in any way. The best thing would be to build upon @jlewis91's efforts and add back improved versions of a CLI interface and a caching layer. SQLite support should be dropped altogether.

[1] https://github.com/jlewis91/dwdbulk
[2] https://github.com/jlewis91/dwdbulk/blob/6892c2c/tests/test_dwdbulk.py#L64-L65

cc @wetterfrosch

Trying to get dwdweather 0.11.1 running on Python 3.7.4.

Problem:
Issuing the command
dwdweather weather 02667 20190717T11 --resolution hourly --categories air_temperature
leads to
2020-01-07 16:33:48,803 [dwdweather.client ] INFO : Requesting https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/air_temperature/recent
2020-01-07 16:33:48,928 [dwdweather.client ] WARNING: Station "2667" has no data for category "air_temperature"

The issue is also present when importing DwdWeather in python using the minimal example in the readme.

I tracked the problem to the client.py. For some reason the find_resource_file function in get_measurements gets stuck in the try block, but there is no Error raised.

When installing dwdweather2 I find dwdweather2 0.14.0 has requirement requests<2.23,>=2.22

Is there a reason why requests 2.24.0 is banned?

Hi,

the paths are still correct?

I do not get daily data or hourly data for any station...

dwdweather weather 3987 2022-02-04T19:00

Messages like:
Station "3987" has no data for category "air_temperature"
Station "3987" has no data for category "cloud_type"
....

If I check https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/air_temperature/
then I see only directories "historical", "recent","timeseries_overview" - but not "now"

I think something was there updated?

Thanks!

@Noordsestern asked over at #5

Any plans migrating to python3?

Hi!

I cannot embed this into my project as the request library is outdated.

dwdweather2 (0.14.0) depends on requests (>=2.22,<2.23)

Does it have to be limited to <2.23?

Hi,

I want to get a list of the x nearest stations (and not just the closest). How can I do this? It seems in the command line this is possible, but I am using the library. Am I missing it or is it not possible yet?

Just some context: I want to get the e.g. 3 closest stations that have data for e.g. wind speed around a given point.

TIA
Antje

Hello,

I am trying to locate nearest dwd sensors from a list, therefore wanted to install the new library of dwd. However when I try to run the sample code seen at the Usage as a library documentation, I get this error.

dw = DwdWeather() # Sample code 
No schema information for resolution "None" found in knowledge base.
An exception has occurred, use %tb to see the full traceback.

SystemExit: 1

/home/halden/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3333: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

Edit: Apparently I forgot to specify the resolution parameter. It was a problem from my end. therefore I closed the topic.
I googled possible solutions but nothing worked so far. Any ideas about troubleshooting?

Station "662" has no data for category "soil_temperature" 50%|████████████████ | 271257/541989 [01:20<01:05, 4125.29it/s]Error in converting field 'solar_duration', value '118.0' to int. File "/home/xxxxxxx/.local/lib/python3.6/site-packages/dwdweather/core.py", line 469, in import_measures_textfile cell = int(cell)

How to reproduce:

from dwdweather import DwdWeather
from datetime import datetime

dw = DwdWeather(resolution="hourly")

dw.query(station_id=662, timestamp=datetime.now())

My Fix:
#20

When using print(DwdWeather.query()) it only returns this message: TypeError: query() missing 3 required positional arguments: 'self', 'station_id', and 'timestamp' and not the expected list of all the possible keys.

I used Python 3.8.5 and the latest dwdweather2 release.

If I try to restrict the categories in library usage (daily resolution), no precipitation is found for any station. Without the restriction to category_names I can derive "precipitation_height" from results. Is that wanted behaviour or a bug?

Coming from marians/dwd-weather#8, we want to add PostGIS support in the future to unlock querying the station list using GIS operations as we are already doing to the station list from luftdaten.info with Luftdatenpumpe, see also LDI data plane v2.

The outcome of that will converge into one or more Grafana Dashboards @wetterfrosch is currently working on over at weather.hiveeyes.org and we will be happy to share all results with the community.