Category: IoT-Data | Home Assistant

Github-style calendar heatmap of device data

2016-08-19T06:00:00+00:00

Thanks to [Anton Kireyeu](https://github.com/kireyeu) we are able to present another awesome [Jupyter notebook]. I guess that you all know the graph which Github is using to visualize your commits per day over a time-line. It's a so-called [heatmap]. If there are more commits, it's getting hotter. The latest [notebook][nb-prev] is capable to do the same thing for your devices. To be more precise, for the hours your devices are home.

Heatmap

[heatmap]: https://en.wikipedia.org/wiki/Heat_map [Jupyter notebook]: https://jupyter.org/ [nb-prev]: http://nbviewer.jupyter.org/github/home-assistant/home-assistant-notebooks/blob/master/DataExploration-2/DataExploration-2.ipynb

IoT Data Exploration with Jupyter Notebooks

2016-07-23T18:00:00+00:00

_This is the first blog post by Anton Kireyeu. A new contributor to Home Assistant who will focus on exploring and visualizing Home Assistant data._ As we learned in the recent [blog post by Fabian], all operational data of your Home Assistant application is stored locally and is available for exploration. Our first steps were querying data with the [DB Browser for SQLite], exporting the data extract as a CSV file and graphing in LibreOffice. But what else can be done with this data and what tools are there available? This post will help you get set up using a few popular data scientist tools to allow you to locally process your data: - [Pandas]: an open source tool for data analysis for Python - [matplotlib]: a Python plotting library - [Jupyter notebook]: application for creation and sharing of documents containing live code, visualizations and explanatory text

One of the graphs created with this tutorial.

_TL; DR: Use [this Jupyter Notebook][nb-prev] to visualize of your data_ [blog post by Fabian]: https://home-assistant.io/blog/2016/07/19/visualizing-your-iot-data/ [DB Browser for SQLite]: http://sqlitebrowser.org/ [Pandas]: http://pandas.pydata.org/ [matplotlib]: http://matplotlib.org/ [Jupyter notebook]: https://jupyter.org/ [nb-prev]: http://nbviewer.jupyter.org/github/home-assistant/home-assistant-notebooks/blob/master/DataExploration-1/DataExploration-1.ipynb ### Dependencies In order to run the provided Jupyter notebook, please make sure you have the following applications/libraries installed on your computer: - Pandas - NumPy - Matplotlib - SQLAlchemy - Jupyter As a Windows user myself, I find the easiest, quickest and most hassle-free way of installing all of these dependencies is to use [WinPython]. This free open-source portable distribution includes all of the dependencies required for this notebook, as well as a few other essential Python libraries you may require for data exploration in the future. [WinPython]: https://winpython.github.io/ #### Why Jupyter? While all Home Assistant implementations can have varying setup, components and scripts, the underlying data structure is standardized and well-defined. This allows us to write Python code that is environmentally agnostic. Wrapping it in a Jupyter notebook ensures the code, visualizations and directions/explanations are kept digestible and neatly-packaged. One of the amazing features of Jupyter is the ability to change code as you go along, customizing all outputs and visualizations on the fly! #### Where do I start? This tutorial is based around a heavily commented Jupyter Notebook that we created. So to get started, you will have to open that: - [download the tutorial Jupyter Notebook][nb-prev] (leads to preview page, from there click download top-right) - launch the Jupyter Notebook App - Click the 'upload' button to add the downloaded notebook to Jupyter - Adjust the `DB_URL` at the beginning of the notebook to point at your Home Assistant database - Select in top menu: Cell -> Run All That’s it! The included code will walk you through importing the required libraries, show running raw SQL against your local database, plotting basic data from the states table, and in the end output a few plots of changes for every entity in your system as well as the mean daily value for the past 20 days. After just those few steps, you will be greeted with beautiful formatted data like this:

One of the graphs created with this tutorial.

#### What’s next? Thanks to the magic of Jupyter, all of the code is customizable: want to selectively display your data, only covering a specific entity? Sure thing! Want to change the properties of the plots? No problem! While you learn and explore your IoT data, we will be working on providing more ready-to-use Jupyter Notebooks. Feel free to ask questions or provide suggestions. Would you like to see a specific visualization? Is there a particular facet of data you’re interested in? Let’s talk about it, let’s dive into the world of data together!

Visualize your IoT data

2016-07-19T16:00:00+00:00

The [history component](/components/history/) is tracking everything that is going on within Home Assistant. This means that you have access to all stored information about your home. Our history is not a full-fledged graphical processing and visualization component as you may know from systems and network monitoring tools. The current limitation is that you only can select a day for a visual output of your information and not a period. Also, there is no possibility to drill down on a specific entity. This blog post will show you ways to export data for reporting, visualization, or further analysis of automation rules. In this blog post I use the temperature of the [Aare](https://en.wikipedia.org/wiki/Aare) river close to where I live as a show case. The temperatures were recorded with the [Swiss Hydrological Data sensor](/components/sensor.swiss_hydrological_data/) and the name of the sensor is `sensor.aare`. The database is stored at `/.homeassistant/home-assistant_v2.db` as [SQLite database](https://www.sqlite.org/). In all examples we are going to use the path: `/home/ha/.homeassistant/home-assistant_v2.db` If you are just curious what's stored in your database then you can use the `sqlite3` command-line tool or a graphical one like [DB Browser for SQLite](http://sqlitebrowser.org/). The table that is holding the states is called `states`. The `events` tables is responsible for storing the events which occurred. So, we will first check how many entries there are in the `states` table. `sqlite3` needs to know where the databases is located. To work with your database make sure that Home Assistant is not running or create a copy of the existing database. It's recommended to work with a copy. ```bash $ sqlite3 /home/ha/.homeassistant/home-assistant_v2.db SQLite version 3.11.0 2016-02-15 17:29:24 sqlite> SELECT count(*) FROM states; 24659 ``` Let's have a look at a sample [SQL](https://en.wikipedia.org/wiki/SQL) query. This query will show all states in a period for the sensor `sensor.aare`. ```sql SELECT state, last_changed FROM states WHERE entity_id = 'sensor.aare' AND last_changed BETWEEN '2016-07-05 00:00:00.000000' AND '2016-07-07 00:00:00.000000'; ``` The SQL statement can be formed that it fits exactly what you need. This means that you can process the data in any way you want for further use. Often it makes sense to eliminate certain entries like `Unknown` or peaks. If the above query is executed in DB Browser for SQLite you would be able to save the sensor's graph as png.

Visualization with DB Browser for SQLite

You may ask: Why not do this with LibreOffice Calc or another spreadsheet application? As most spreadsheet applications are not able to work directly with SQLite database we are going to export the data from the database to [CSV](https://en.wikipedia.org/wiki/Comma-separated_values). ```bash $ sqlite3 -header -csv /home/ha/.homeassistant/home-assistant_v2.db "SELECT last_changed, state FROM states WHERE entity_id = 'sensor.aare' AND last_changed BETWEEN '2016-07-05 00:00:00.000000' AND '2016-07-07 00:00:00.000000';" > sensor.csv ``` The ordering for the `SELECT` was changed to get the time stamps first and then the state. Now we can import the CSV file into the application of your choice, here it's LibreOffice Calc.

Import of the CSV file

After the import a graph can be created over the existing data.

Graph in LibreOffice

You can also use [matplotlib](http://matplotlib.org/) to generate graphs as an alternative to a spreadsheet application. This is a powerful Python 2D plotting library. With the built-in support for SQLite in Python it will only take a couple lines of code to visualize your data. ```python import sqlite3 from matplotlib import dates import matplotlib.pyplot as plt import homeassistant.util.dt as dt values = [] timestamps = [] conn = sqlite3.connect('/home/ha/.homeassistant/home-assistant_v2.db') data = conn.execute("SELECT state, last_changed FROM states WHERE " "entity_id = 'sensor.aare' AND last_changed BETWEEN " "'2016-07-05 00:00:00.000000' AND " "'2016-07-07 00:00:00.000000'") for x in data: timestamps.append(dates.date2num(dt.parse_datetime(x[1]))) values.append(float(x[0])) plt.plot_date(x=timestamps, y=values, fmt="r-") plt.ylabel('Temperature') plt.xlabel('Time line') plt.savefig('sensor.png') ``` Creating a connection to the database and executing a query is similar to the ways already seen. The return values from the query are split into two lists. The time stamps must be converted in an value which is accepted by matplotlib and then the graph is generated and saved as image.

Sensor graph generated by matplotlib

Most of the graphs are pretty ugly. So, further beautification will be needed. If you have created a nice report including some amazing graphs then the Home Assistant community would be grateful for sharing them in our [forum](https://community.home-assistant.io/).