seitabv / flexmeasures-openweathermap Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 1.0 51 KB

Integration to the OpenWeatherMap API from FlexMeasures.

License: Apache License 2.0

Makefile 2.36% Python 96.87% Shell 0.77%

flexmeasures-openweathermap's People

Contributors

Stargazers

Watchers

flexmeasures-openweathermap's Issues

Update the tests according to the new code

Fix the tests that are failing after the latest commits.

link : https://github.com/SeitaBV/flexmeasures-openweathermap/actions/runs/5730725655/job/15530130433

Limit lookup for weather stations by account

Currently, all weather stations will be searched and potentially be used to store data (if they are the closest to the location).

While it makes sense conceptually that weather stations are public (to be used by all accounts on a FlexMeasures server), it doesn't have to be like that. Maybe two accounts use different weather services. It can lead to unwanted data situations, in any case.

Sensor.find_closest() already supports adding an account_id argument. We could support this from FM-OWM, by configuring an account_id (as CLI parameter or a setting).

It would also be good to query only public weather stations per default, however, the account_id parameter has no effect when set to None. Maybe we'd need to add our own filter (disregarding non-public) in this case.

Separate data source for forecasts and measurements

I'd like forecasts and measurements to be saved under a separate DataSource (type), so we can treat them separately in e.g. the FlexMeasures UI.

I propose to use the following. For measurements:

Source(
    name="OpenWeatherMap",
    model="onecall/timemachine",  # name of the API, see https://openweathermap.org/api/one-call-api#history
    version="2.5",
    type="archive",  # alternatives: "historian" or "logger"
)

And for forecasts:

Source(
    name="OpenWeatherMap",
    model="onecall"  # name of the API, see https://openweathermap.org/api/one-call-api#current
    version="2.5",
    type="forecaster",
)

Two notes:

FlexMeasures could use a database migration to fill in the model and version (to this date, this plugin only used the onecall endpoint and not the onecall/timemachine endpoint). Even though such a migration would be specific to this plugin, I reckon it should be part of the db versioning of FlexMeasures.
As a related issue, I'm considering using a db migration to rename the "forecasting script" data source type to "forecaster" in FlexMeasures.
OWM's latest API version is actually 3, see https://openweathermap.org/api/one-call-3, so let's consider updating (in a separate issue).

Use .in for requirement when installing the package

In setup.py, we should (just as FlexMeasures does) not use the fixed dependencies but our required ranges.

Save data for those sensor which are present

Presently, the plugin scans for various sensors such as wind speed, temperature, cloud cover, and irradiance, subsequently saving the weather forecast. It is necessary to implement a check on these sensors to store the forecast only for those that are registered.

flexmeasures-openweathermap/flexmeasures_openweathermap/utils/owm.py

Line 100 in d5e56bd

for sensor_specs in mapping:

Allow storing both cloud cover and irradiance

The irradiance is computed from the cloud cover, which is nice, but it can be useful to store the raw cloud cover data as well. The sensor mapping in sensor_specs.py doesn't easily allow to do this (the clouds key is already in use). The way we set up the sensor mapping in https://github.com/SeitaBV/flexmeasures-zinfo/ better serves the purpose of both setting up sensors for raw data and setting up sensors for derived data.

Make OWM's API version configurable

Currently hard-coded to 2.5

`make install-for-dev` is failing to install python modules

I tried to install the plugin for development using make, and I had this below issue which is not able to install some dependencies.

pip-compile -o requirements/test.txt requirements/test.in
  ERROR: Cannot install -r requirements/test.in (line 8) because these package versions have conflicting dependencies.
Traceback (most recent call last):
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 316, in _backjump
    name, candidate = broken_state.mapping.popitem()
KeyError: 'dictionary is empty'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 434, in resolve
    success = self._backjump(causes)
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in _backjump
    raise ResolutionImpossible(causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('redis<4.4'), parent=LinkCandidate('https://files.pythonhosted.org/packages/6b/d1/05dcf84b58ad53ac21e17fc48045f1932a39ef3b41f115013e44abac2f8e/fakeredis-1.9.1-py3-none-any.whl (from https://pypi.org/simple/fakeredis/) (requires-python:>=3.7,<4.0)'))]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/piptools/scripts/compile.py", line 630, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/home/ubuntu/Projects/Seita/flexmeasures-openweathermap/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 101, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
make[1]: *** [Makefile:36: freeze-deps] Error 1
make[1]: Leaving directory '/home/ubuntu/Projects/Seita/flexmeasures-openweathermap'
make: *** [Makefile:17: install-for-dev] Error 2

Make a Pypi package (pip-installable)

This plugin should be pip-installable.

See the latest state in Inspired by the flexmeasures-plugin-template for inspiration.

Support building dependencies, or find out which Python versions we support

When trying the make-supported installation, a user reported an error (see below, they say it works neither with Python 3.11 nor 3.9).

However, for me on Python3.10, make install works well (in a virtual env).

We might have an issue with requirements here, and maybe different Python versions. This requires a little research.

make install
make install-pip-tools
pip3 install -q "pip-tools>=6.2"
make freeze-deps
make install-pip-tools
pip3 install -q "pip-tools>=6.2"
pip-compile -o requirements/app.txt requirements/app.in

error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully
│ exit code: 1
 ─> [29 lines of output]
Package hdf5 was not found in the pkg-config se
Perhaps you should add the directory containing
to the PKG_CONFIG_PATH environment variable
No package 'hdf5' found
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <
File "/private/var/folders/xq/wbc5yvh91bz8dc7
_populate_hdf5_info(dirstosearch, inc_dirs,
File "/private/var/folders/xq/wbc5yvh91bz8dc7

raise ValueError('did not find HDF5 headers
ValueError: did not find HDF5 headers
reading from setup.cfg...


HDF5_DIR environment variable not set, chec
checking /Users/edgar/include ...
hdf5 headers not found in /Users/edgar/include
checking /usr/local/include ...
hdf5 headers not found in /usr/local/include
checking /sw/include ...
hdf5 headers not found in /sw/include
checking /opt/include ...
hdf5 headers not found in /opt/include
checking /opt/local/include ...

Allow to pass asset ID and/or name to CLI commands instead of full location

This is useful because the location is cumbersome to look up and type.

For the name, we'd need a UniqueConstraint on GenericAsset.account_id and GenericAsset.name, which sounds like a good idea to me anyway.

Add setting for minimal degrees search distance for weather station

We use the variable max_degree_difference_for_nearest_weather_sensor with a default value of 2, when looking up weather sensors. So both latitude and longitude must be within this margin of error.

Isn't 2 on the high side?
It should become a setting which can be overwritten in flexmeasures.cfg. We can easily add this in __init__.py

Refactoring of irradiance computations

Right now, we compute irradiance ourselves. In save_forecasts_to_db, we catch this sensor name ("irradiance") and do the actual computation to get from cloud cover data to the irradiance.

That code should live in a better place. A function store, let's say.
save_forecasts_to_db should just use that.

seitabv / flexmeasures-openweathermap Goto Github PK

flexmeasures-openweathermap's People

Contributors

Stargazers

Watchers

flexmeasures-openweathermap's Issues

Recommend Projects

Recommend Topics

Recommend Org