ejep / datapoint-python Goto Github PK

Currently requesting data for coordinates which are outside the UK results in misleading data or no data (see #15, #37). Once haversine is implemented (issue #10, pull request #69) the distance from the requested location to the nearest site will be known in kilometres. The problem of the requested point being out of bounds would be solved by requiring that the point be within a set distance of the nearest forecast site. @jacobtomlinson do you have an idea of what a suitable distance would be?

Running simple_forecast.py with

site = conn.get_nearest_forecast_site(52.603370,-2.1413670)

gives the following error

Renaming functions in #77 and limiting the distance of sites for responses (#74, #76) have the potential to break things. Along with some bug fixes, this seems like a good time to move to version 0.7.0

@jacobtomlinson what are your thoughts?

Currently Forecast.now() returns the latest time step on the day of the forecast which is in the past. If this does not exist, a problem near midnight, this is worked around. E.g. if it is currently 16:00, the 15:00 forecast will be returned. If it is 17:45 the 15:00 forecast will be returned, not the 18:00

I think better logic would be to return the nearest forecast in time, whether that is in the future or the past. This should cope with the midnight issue (#19) as the second day in the forecast (days[1]) could be checked.

GitHub is raising security concerns with the versions of requests being used. It has suggested updating to 2.20.0 or above.

Implement the Observations product from Datapoint.

http://www.metoffice.gov.uk/datapoint/product/uk-hourly-site-specific-observations

I created this project a few years ago to simplify accessing the Met Office Datapoint API for my projects. I was working in IT Support at the Met Office at the time and specifically wanted it to automate some vanity features like changing wallpaper automatically based on the weather. Despite being employed at the Met Office this is an unofficial and unsupported way of accessing the APIs, hence why it lives under my personal github namespace.

Around 4 years ago I moved to the @informatics-lab in the Met Office and my maintenance on this project decreased with time as I was not actively using it in my day to day activities.

Recently @EJEP picked up maintenance of the project and has been extremely helpful. I've only weighed in from time to time with some project direction.

I've recently left the Met Office and therefore my interest in this project has decreased even further. I know it is a dependency of other projects such as @home-assistant and so I would like to see it maintained, but I just can't justify this activity myself any longer.

Therefore I would like to discuss our options here. Would @EJEP like to take on sole ownership of the project? If so should it be transferred to your GitHub account? If not then what other options are there?

On March 1st there are bool is sometimes returned. I suspect this is related to the previous bugs that give this problem.

Possibly the code can't cope with the possibility of leap years?

The function forecast.now() always returns None. It looks like the problem appeared in commit cd82da5, in which the return logic was altered.

The documentation has quite a few tables in, which are written as html in the markdown files. These are rather hard to read. ReStructured Text has a built in table syntax which is easier to read. Changing this would not require changing the documentation text. It would require changing the documentation build system to use sphinx

Add a new comment for a unique idea for an example use of datapoint.

There are tools which can help automate the change log between versions.This would be nice to have.

Should really be latitude first

Within the function _distance_between_coords in the Manager object I initially implemented a haversine function for finding the nearest forecast site. However it didn't work properly and so I replaced it with a far simpler formula.

Despite this being more acurate than my original broken haversine function it probably isn't 100% correct. It should be replaced with a correct haversine function.

Python 2.6 was end-of-lifed some time ago. @jacobtomlinson do you have any objections to removing it from the listed versions supported and from the tests? This probably ought to be done in a point release.

Once the site list has been requested there is no need to request it again as it is unlikely to change.

The get_all_sites() function should cache the site list for a specified amount of time.

There is no guidance in the DataPoint documentation on how frequently this list changes, but I expect it to be so infrequently that caching the list for up to 24 hours wouldn't be overkill. To keep it responsive though I'd be tempted to suggest 1 hour would be fine.

It seems that the API occasionally returns weather values which are specified as "night" or "day" (e.g "hail shower night" or "hail shower day") but at the wrong time of day. Usually around changeover times, it sometimes stays as night until around 9am, 3 hours after sunrise. And very occasionally returns a day value in the middle of the night.

I can imagine two solutions for this:

A) Calculate when sunrise/sunset is and just correct the values if they are wrong.
This remains inline with the API and just cleans the data.

B) Strip the day/night from weather types completely (both "hail shower night" and "hail shower day" would just become "hail shower"). Then calculate day/night and store as a separate attribute.
Allows most flexibility when using the data.

Implement the Text Forecast product from Datapoint

http://www.metoffice.gov.uk/datapoint/product/regional-text-forecast

It would be nice to be able to get every element in a timestep easily, for instance

for element in timestep:
    print(element.value)

@jacobtomlinson The observation code is now finished (pull request #53). Along with swapping longitude and latitude (pull request #31) and adding timeouts (a minor change in __call_api, pull request #54) there are probably enough changes to justify a bump in version number to 0.5.0, especially since #31 is not backwards compatible. What are your thoughts?

There are also some other things to do like:

• The PyPi page doesn't list python versions 3.5 and 3.6 which are tested by Travis. This was correct in the master setup.py
• The version of the requests module used in travis can be increased
• python 3.7 could be tested against.
• change longitude and latitude in observation code to ensure consistency
• Add unit test for observation
• Update object image in documentation to include observations

I will list anything else I think of over the next few days.

Occasionally the connection to the datapoint API fails. It would be nice if this module could handle retrying the connection. The requests module has support for this.

At present errors with Forecast functions result in a return value of None, which causes later errors. Fatal errors should instead raise errors, like is done for some other types of error in Manager.

Hi,

Having updated the API key and location, I am running current_weather.py but get the following error:

Worthing (Beach)
Traceback (most recent call last):
File "current_weather.py", line 21, in
print now.weather.text
AttributeError: 'bool' object has no attribute 'weather'

I then tried with:

current_timestep = forecast.now()
print current_timestep.weather.text

And that gives an error too:

Worthing (Beach)
Traceback (most recent call last):
File "temp.py", line 22, in
print current_timestep.weather.text
AttributeError: 'bool' object has no attribute 'weather'

I've been investigating and can't work out what is wrong. I saw a recent bug due to the forecast.now problem but this seems to have been resolved and I do have the latest version.

Any ideas what might be wrong?

Rob

Cloning the repository and running python setup.py install resulted in errors due to print statements not using parentheses.

The checking of the data is a bit haphazard at the moment. Putting it all in one place would make it easier to modify in the future.

Despite returning basically the same data types for a 'Day', 'Night' and '3hourly' timestep Datapoint feels the need to provide different keys for the same thing.

For example Probability of Precipitation will be 'Pp' if you requested 3hourly. But if you request Daily you will get 'PPd' in the Day timestep and 'PPn' in the Night timestep. And it is not even consistent with some things being the same (Visibility is always 'V') and some being different again e.g Humidity is either 'H' for 3hourly, or 'Hn' and 'Hm' for 'Humidity Noon' and 'Humidity Midday'.

The only difference between the data returned for any of the timesteps is that you don't get UV data for a 'Night' timestep in a Daily. Even though you do for midnight in a 3hourly.

Also Temperature (and Feels Like Temperature) in a 3hourly is the average temperature for that timestep but in a daily it's the minimum temperature for the Night timestep and maximum temperature for the Day timestep.

To work around this the Manager object holds a hardcoded array of mappings and has some additional logic to read the correct array items into the Timestep object properties.

This makes the code a but ugly and overly complicated. I'd like to neaten this up if possible.

Sometimes 'None' is returned from get_all_sites in Manager.py. This has happened a few times, but I have not managed to reproduce it in code trying to look at the bug. I suspect that it is related to the way I am trying to retry connections in code outside this module.

My (unconfirmed) suspicion is that the sites_last_update_time is set, then there is a connection error which is raises an exception. manager.sites_last_request is then never changed from None.

If the same connection is then used (as in my code), the sites list is not updated by get_all_sites as very little time has passed.

The Timestep object currently has a 'name' property. The maps directly to the 'name' property returned from Datapoint and is the number of minutes after midnight (UTC) the timestep starts.

This should probably be converted into a datetime object and perhaps be renamed to something like 'date' or 'time'

The web service used by the postcodes module is dead (see e-dard/postcodes#5) so the postcodes example no longer works.

Hi I got this error when trying to reproduce one of the examples.
AttributeError Traceback (most recent call last)
in ()
----> 1 print now.weather.text

AttributeError: 'NoneType' object has no attribute 'weather'

This example is using Zurich, Switzerland with the coordinates from the Metoffice website.

$ python3
Python 3.6.6 (default, Jul 19 2018, 14:25:17) 
[GCC 8.1.1 20180712 (Red Hat 8.1.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datapoint
>>> conn = datapoint.connection(api_key="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee")
>>> site = conn.get_nearest_site(47.383, 8.567)
>>> forecast = conn.get_forecast_for_site(site.id, "3hourly")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/fab/Documents/repos/ha/home-assistant/lib64/python3.6/site-packages/datapoint/Manager.py", line 217, in get_forecast_for_site
    forecast.continent = data['SiteRep']['DV']['Location']['continent']
KeyError: 'Location'
>>> 

It works for Paris and London.

Original issue: home-assistant/core#17492

At present the version number is set in three difference places. It will be more convenient to store it in one place only.

From log file for site Clifton Hill Ski Slope Exeter

Tue Dec  2 21:00:03 2014 - Partly cloudy (night)
Tue Dec  2 22:00:02 2014 - Partly cloudy (night)
Tue Dec  2 23:00:01 2014 - Partly cloudy (night)
Unexpected error: <type 'exceptions.NameError'> on line 64
Wed Dec  3 01:00:01 2014 - Clear night
Wed Dec  3 02:00:03 2014 - Clear night
Wed Dec  3 03:00:04 2014 - Clear night

This issue happens every day when requesting data at midnight.

If longittude and latitude are the wrong way round, it returns st margarets bay beach as that is furthest point on edge england. Could make a check to make sure that this long/lat it actually in UK else return error.

The date property for a Day object should probably be a datetime object.

Currently it is a string returned from datapoint in the format yyyy-mm-ddZ.

Datapoint provides capability information. This states what data is available for a specific site or sites without returning the data itself.

This would be useful for error checking and setting bounds rather than assuming a 5 day forecast will be returned.

http://www.metoffice.gov.uk/datapoint/product/uk-3hourly-site-specific-forecast/detailed-documentation#5,000 UK locations three hourly forecasts capabilities feed

If you get the API key wrong, you get an obscure message:

>>> import datapoint
>>> 
>>> # Create datapoint connection
... conn = datapoint.Manager(api_key="aaa")
>>> 
>>> site = conn.get_nearest_site(-0.124626, 51.500728)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
...
  File "/Users/pelson/miniconda/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

@jacobtomlinson, setting up the webhook needs access to the settings page of the repository on GitHub, which I think only the owner has. From the readthedocs documentation all that needs doing is copying the URL from the Admin->Integrations page on the read the docs settings to the GitHub integrations settings section and ticking a few boxes.

(Latitude as well)

This is mostly a housekeeping thing

Sometimes the connection times out rather than retrying and isn't wrapped in a MaxRetries error. This needs investigating.

Please add timeout to all requests. Thanks.

In the daily forecast, there are two timesteps per day. The first timestep in the list is later than the second. The first is for 12:00 and the second is for 00:00. This should be the other way around. This looks like it is related to the way that the data is supplied by the API.

Python 3.4 has now reached end-of-life as of 2019-03-18 (see here). I think we should remove support.

@jacobtomlinson what are your thoughts?

The version numbers currently follow x.y.z format. I think it would be a good idea to explicitly use semantic versioning as defined at https://semver.org/ to set version numbers. We are in 0.y.z so things are still fluid.

@jacobtomlinson what do you think?

Currently the forecast object has a method called now() which returns the current timestep from that forecast.

It would be good to be able to provide a datetime object as an argument to return different timesteps.

At the moment the names of the functions for getting sites are untidy: The ones for observations mention observations in the name; the ones for forecasts don't and therefore appear global.

Tidying this would make it more obvious what the functions do.

The documentation states that element.value is an int. This is not always the case.