home-assistant / home-assistant-notebooks

:notebook: Sample Jupyter Notebooks to explore Home Assistant data

Jupyter notebooks for Home Assistant

Sample Jupyter Notebooks to inspect data of your Home Assistant instance.

• For help to get started, see the Home Assistant Data Science Portal
• Package to help you explore Home Assistant data: Hass Data Detective

home-assistant-notebooks's Issues

Getting started note book errors

• Add-on version: 0.9.2
• System: Home Assistant OS 9.3 (amd64 / qemux86-64)
• Home Assistant Core: 2022.11.2
• Home Assistant Supervisor: 2022.10.2

%%time

df = db.fetch_all_sensor_data()

The returned Pandas dataframe has 50000 rows of data.
CPU times: user 346 ms, sys: 48.3 ms, total: 395 ms
Wall time: 457 ms

df = df[df['domain']=='sensor']
df = functions.generate_features(df)
df = functions.format_dataframe(df)

---

KeyError Traceback (most recent call last)
File /usr/local/lib/python3.9/dist-packages/pandas/core/indexes/base.py:3803, in Index.get_loc(self, key, method, tolerance)
3802 try:
-> 3803 return self._engine.get_loc(casted_key)
3804 except KeyError as err:

File /usr/local/lib/python3.9/dist-packages/pandas/_libs/index.pyx:138, in pandas._libs.index.IndexEngine.get_loc()

File /usr/local/lib/python3.9/dist-packages/pandas/_libs/index.pyx:165, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:5745, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:5753, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'domain'

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
Cell In [5], line 1
----> 1 df = df[df['domain']=='sensor']
2 df = functions.generate_features(df)
3 df = functions.format_dataframe(df)

File /usr/local/lib/python3.9/dist-packages/pandas/core/frame.py:3804, in DataFrame.getitem(self, key)
3802 if self.columns.nlevels > 1:
3803 return self._getitem_multilevel(key)
-> 3804 indexer = self.columns.get_loc(key)
3805 if is_integer(indexer):
3806 indexer = [indexer]

File /usr/local/lib/python3.9/dist-packages/pandas/core/indexes/base.py:3805, in Index.get_loc(self, key, method, tolerance)
3803 return self._engine.get_loc(casted_key)
3804 except KeyError as err:
-> 3805 raise KeyError(key) from err
3806 except TypeError:
3807 # If we have a listlike key, _check_indexing_error will raise
3808 # InvalidIndexError. Otherwise we fall through and re-raise
3809 # the TypeError.
3810 self._check_indexing_error(key)

KeyError: 'domain'

Moving docs to home-assistant.io

Hi home-assistant-notebooks maintainers,

We are about to add a new section to home-assistant.io titled "Ecosystem". It's a place where any external tools can share their documentation in an official and easy to reference capacity. I think that home-assistant-notebooks would be a great addition. The PR to add this section should be merged tonight or tomorrow and docs can start being written shortly thereafter. I don't think you would need to do anything more than taking the existing docs and splitting them up into pages for easy linking.

Obviously you don't have to do this but we thought we'd let you know.

The PR that needs to be merged is here.

Thanks!

Propose to not point people to nbviewer

Since Github renders notebooks, I don't see the point in sending people to nbviewer to view the notebooks, its just one more UI to confuse people.

Add example .db file

To enable reproducible runs of the getting started notebook it is ideal to have a small .db file which can be used as the data source for the notebook run.

DataExploration-1.ipynb needs update: Column 'origin' doesn't exist

In DataExploration-1.ipynb a cell isn't working anymore:

# name the dataframe rows for usability
allqueryDF.columns = ['state_id', 'domain', 'entity_id', 'state', 'attributes',
                      'origin', 'event_id', 'last_changed', 'last_updated',
                      'created']

should now read

# name the dataframe rows for usability
allqueryDF.columns = ['state_id', 'domain', 'entity_id', 'state', 'attributes',
                      'event_id', 'last_changed', 'last_updated', 'created']

Getting Started Notebook fails with Execute error

After a clean install of the Jupyter plugin I received the following error. The error persisted after updating the getting-started notebook to latest from this repository:

import detective.functions as functions
import pandas as pd
​
db = detective.db_from_hass_config()
YAML tag !include_dir_merge_named is not supported
Successfully connected to database sqlite:////config/home-assistant_v2.db
Error with query:
SELECT DISTINCT(entity_id) FROM states

'Engine' object has no attribute 'execute'

AttributeError Traceback (most recent call last)
Cell In[10], line 5
2 import detective.functions as functions
3 import pandas as pd
----> 5 db = detective.db_from_hass_config()

File /usr/local/lib/python3.9/dist-packages/detective/core.py:19, in db_from_hass_config(path, **kwargs)
16 path = config.find_hass_config()
18 url = config.db_url_from_hass_config(path)
---> 19 return HassDatabase(url, **kwargs)

File /usr/local/lib/python3.9/dist-packages/detective/core.py:57, in HassDatabase.init(self, url, fetch_entities)
55 print("Successfully connected to database", stripped_db_url(url))
56 if fetch_entities:
---> 57 self.fetch_entities()
58 except Exception as exc:
59 if isinstance(exc, ImportError):

File /usr/local/lib/python3.9/dist-packages/detective/core.py:85, in HassDatabase.fetch_entities(self)
79 """Fetch entities for which we have data."""
80 query = text(
81 """
82 SELECT DISTINCT(entity_id) FROM states
83 """
84 )
---> 85 response = self.perform_query(query)
87 # Parse the domains from the entities.
88 self.entities = [e[0] for e in response]

File /usr/local/lib/python3.9/dist-packages/detective/core.py:73, in HassDatabase.perform_query(self, query, **params)
71 """Perform a query."""
72 try:
---> 73 return self.engine.execute(query, params)
74 except:
75 print(f"Error with query: {query}")

AttributeError: 'Engine' object has no attribute 'execute'

The Getting Started notebook is out of date

The Getting Started notebook is actually incompatible with the latest releases of HASS-data-detective, currently at 2.4. This is causing issues for users of the jupyterlab addon since it is installing the latest data detective. This notebook needs to be updated to use the same version of detective as the addon (which is currently on 2.4)

Update notebooks with Hass.io details

The notebooks were written to be uses directly beside a Home Assistant installation. With Hass.io around for quite some time I guess that people start to use the notebooks as well. To make their live easier, additional details for Hass.io should be added, e.g. path to the DB and a like.

@frenck is also working on a Hass.io add-on for Jupyter Labs. And this will boost the usage even more.

Getting Started Error

I have been trying to get this to work for a long time, on and off, I was told before the books were outdated, but I always get an error when I try to run it.

---

AttributeError Traceback (most recent call last)
File /usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/base.py:1412, in Connection.execute(self, statement, parameters, execution_options)
1411 try:
-> 1412 meth = statement._execute_on_connection
1413 except AttributeError as err:

AttributeError: 'str' object has no attribute '_execute_on_connection'

The above exception was the direct cause of the following exception:

ObjectNotExecutableError Traceback (most recent call last)
Cell In[10], line 13
9 # We keep track of contexts that we processed so that we will only process
10 # the first service call in a context, and not subsequent calls.
11 context_processed = set()
---> 13 for event in db.perform_query("SELECT * FROM events WHERE event_type = 'call_service' ORDER BY time"):
14 entity_ids = None
16 # Skip if we have already processed an event that was part of this context

File /usr/local/lib/python3.11/dist-packages/detective/core.py:75, in HassDatabase.perform_query(self, query, **params)
73 try:
74 with self.engine.connect() as conn:
---> 75 return conn.execute(query, params)
76 except:
77 print(f"Error with query: {query}")

File /usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/base.py:1414, in Connection.execute(self, statement, parameters, execution_options)
1412 meth = statement._execute_on_connection
1413 except AttributeError as err:
-> 1414 raise exc.ObjectNotExecutableError(statement) from err
1415 else:
1416 return meth(
1417 self,
1418 distilled_parameters,
1419 execution_options or NO_OPTIONS,
1420 )

ObjectNotExecutableError: Not an executable object: "SELECT * FROM events WHERE event_type = 'call_service' ORDER BY time"

What am I doing wrong ?

Python API is deprecated

As state here, the python homeassistant API is deprecated. So the home-assistant-python-api.ipynb does not work anymore.

Getting started notebook: show alternative to random

For sensors and binary sensors, we currently use a random.choice. This is embedded in the data extraction code, making it unclear to users how they could change it to their own entity ID. We should clarify that by extracting the random.choice to a variable and adding commented out code and a comment on how to change that to your own entity ID.