Merge branch 'current' into next

source/developers/asyncio.markdown

@@ -0,0 +1,27 @@
+---
+layout: page
+title: "Asynchronous Programming"
+description: "Introduction to the asynchronous core of Home Assistant."
+date: 2016-10-17 21:49
+sidebar: true
+comments: false
+sharing: true
+footer: true
+---
+
+On September 29, 2016 we released [Home Assistant 0.29][0.29] as part of our bi-weekly release schedule. This release introduced a complete overhaul of the core spearheaded by [Ben Bangert][ben].
+
+The old core was set up like a “traditional” threaded application. Each resource that was not thread safe (ie. the state of entities) would be protected by a lock. This caused a lot of waiting and potential inconsistency because a task could now end up waiting halfway through it’s job until some resource got freed.
+
+Our new core is based on an Python’s built-in asyncio module. Instead of having all threads have access to the core API objects, access is now limited to a special thread called the event loop. All components will now schedule themselves as a task to be executed by the event loop. This gives us the guarantee that only one task is executed at once, meaning we no longer need any locks.
+
+The only problem with running everything inside the event loop is when a task is doing blocking I/O, what most third-party Python libraries are doing. For example while requesting new information from a device, the core will stop running until we get a response from the device. To handle this, a task is able to suspend itself until the response is available after which it will be enqueued for the event loop to process the result.
+
+For a task to be able to suspend itself, all code that it calls has to have this capability added. This means in practice that each device integration will need a full rewrite of the library that offers the integration! As this is not something that can be achieved, ever, a 100% backwards compatible API has been added so that no platform will require updating.
+
+The backwards compatible API works by scheduling a task from a different thread and blocking that thread until the task has been processed by the event loop.
+
+### [Next step: Categorizing Functions »](/developers/asyncio_categorizing_functions/)
+
+[0.29]: https://home-assistant.io/blog/2016/09/29/async-sleepiq-emoncms-stocks/
+[ben]: https://github.com/bbangert/

source/developers/asyncio_categorizing_functions.markdown

@@ -0,0 +1,79 @@
+---
+layout: page
+title: "Categorizing Functions"
+description: "A categorization of functions to work with the asynchronous core of Home Assistant."
+date: 2016-10-17 21:49
+sidebar: true
+comments: false
+sharing: true
+footer: true
+---
+
+A piece of work within Home Assistant is represented by a function that will be invoked. It will either run inside our event loop or inside our thread pool, depending on if it is async safe.
+
+Home Assistant uses the convention that all functions that must be run from within the event loop are prefixed with `async_`.
+
+## {% linkable_title The coroutine function %}
+
+Coroutines are special functions based on Python’s generators syntax which allows them to suspend execution while waiting on a result.
+
+Invoking a coroutine function will return a Generator object back, but will not actually begin execution. This object will execute the task when it is either yielded from (from within another coroutine) or it is scheduled on the event loop.
+
+To declare a function a coroutine, import the coroutine annotation from the asyncio package and annotate your function.
+
+```python
+import asyncio
+
+@asyncio.coroutine
+def async_look_my_coroutine(target):
+    result = yield from entity.async_turn_on()
+    if result:
+        print("hello {}".format(target))
+
+hass.loop.create_task(async_look_my_coroutine("world")
+```
+
+In this example, we schedule the coroutine by calling `hass.loop.create_task`. This will add the coroutine to the queue of tasks to be run. When the event loop is running `async_look_my_coroutine` it will suspend the task when `yield from entity.async_turn_on()` is called. At that point a new task will be scheduled to execute `entity.async_turn_on()`. When that job has been executed, `async_look_my_coroutine` will resume.
+
+## {% linkable_title The callback function %}
+
+This is a normal function that is considered safe to be run from within the event loop. A callback is unable to suspend itself and thus cannot do any I/O or call a coroutine. A callback is capable of scheduling a new task but it will not be able to wait for the results.
+
+To declare a function as a callback, import the callback annotation from the core package and annotate your function.
+
+A common use case for a callback in Home Assistant is as a listener for an event or a service call. It can process the incoming information and then schedule the right calls to be made. Example from the automation component.
+
+```python
+from homeassistant.core import callback
+
+@callback
+def async_trigger_service_handler(service_call):
+    """Handle automation trigger service calls."""
+    vars = service_call.data.get(ATTR_VARIABLES)
+    for entity in component.async_extract_from_service(service_call):
+        hass.loop.create_task(entity.async_trigger(vars, True))
+```
+
+In this example, `entity.async_trigger` is a coroutine function. Invoking the coroutine function will return a coroutine task. The passed in parameters will be used when the task gets executed.
+
+To execute the task we have to schedule it for execution on the event loop. This is done by calling `hass.loop.create_task`.
+
+### {% linkable_title Why even have callbacks? %}
+
+You might wonder, if a coroutine can do everything a callback can do, why even have a callback. The reason is performance and better state consistency of the core API objects.
+
+When coroutine A waits for coroutine B, it will suspend itself and schedule a new task to run B. This means that the event loop is now running A, B and then A again. If B is a callback, A will never have to suspend itself and thus the event loop is just running A. The consistency implication is that other events queued to run on the event loop continue to wait until callbacks complete, but will be interleaved when yielding to another coroutine.
+
+## {% linkable_title Event loop and thread safe %}
+
+These are functions that are safe to run both in a thread and inside the event loop. These functions are usually performing a computation or transform data in memory. Anything that does I/O does not fall under this category. Many standard library functions fall in this category. For example generating the sum of a set of numbers using sum or merging two dictionaries.
+
+There is no special annotation to mark functions as part of this category and care should be taken when using these functions from inside the event loop. When in doubt, look at their implementation.
+
+## {% linkable_title Other functions %}
+
+These are all the functions that did not fit in the previous categories. These functions are either thread-safe or not considered safe to be run within the event loop. These are functions that use sleep, or perform I/O.
+
+There is no special annotation necessary to be considered part of this category.
+
+### [Next step: Working with Async »](/developers/asyncio_working_with_async/)

source/developers/asyncio_misc.markdown

@@ -0,0 +1,22 @@
+---
+layout: page
+title: "Miscellaneous Async"
+description: "A collection of miscellaneous topics about async that didn't fit on the other pages."
+date: 2016-10-17 21:49
+sidebar: true
+comments: false
+sharing: true
+footer: true
+---
+
+## {% linkable_title What about ‘async’ and ‘await’ syntax? %}
+Python 3.5 introduced new syntax to formalize the asynchronous pattern. This is however not compatible with Python 3.4. The minimum required Python version for Home Assistant is based on the Python version shipped with Debian stable, which is currently 3.4.2.
+
+For more information, Brett Cannon wrote [an excellent breakdown][brett] on 'async' and 'await' syntax and how asynchronous programming works.
+
+## {% linkable_title Acknowledgements %}
+
+Huge thanks to [Ben Bangert][ben] for starting the conversion of the core to async, guiding other contributors while taking their first steps with async programming and peer reviewing this documentation.
+
+[brett]: http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5
+[ben]: https://github.com/bbangert/

source/developers/asyncio_working_with_async.markdown

@@ -0,0 +1,113 @@
+---
+layout: page
+title: "Working with Async"
+description: "A breakdown of all the different ways to work with the asynchronous core of Home Assistant."
+date: 2016-10-17 21:49
+sidebar: true
+comments: false
+sharing: true
+footer: true
+---
+
+Although we have a backwards compatible API, using the async core directly will be a lot faster. Most core components have already been rewritten to leverage the async core. This includes the EntityComponent helper (foundation of light, switch, etc), scripts, groups and automation.
+
+## {% linkable_title Interacting with the core %}
+
+[All methods in the Home Assistant core][dev-docs] are implemented in two flavors: an async version and a version to be called from other threads. The versions for other are merely wrappers that call the async version in a threadsafe manner using [the available async utilities][dev-docs-async].
+
+So if you are making calls to the core (the hass object) from within a callback or coroutine, use the methods that start with async_. If you need to call an async_ function that is a coroutine, your task must also be a coroutine.
+
+## {% linkable_title Implementing an async component %}
+
+We currently do not support async setup for components. We do however support using async functions as service handlers. Just define your handlers as a callback or coroutine and register them as usual.
+
+## {% linkable_title Implementing an async platform %}
+
+For platforms we support async setup. Instead of setup_platform you need to have a coroutine async_setup_platform.
+
+```python
+setup_platform(hass, config, add_entities, discovery_info=None):
+    # Setup your platform outside of the event loop.
+```
+
+Will turn into:
+
+```python
+import asyncio
+
+@asyncio.coroutine
+def async_setup_platform(hass, config, async_add_entities,
+                         discovery_info=None):
+    # Setup your platform inside of the event loop
+```
+
+The only difference with the original parameters is that the add_entities function has been replaced by the coroutine `async_add_entities`.
+
+## {% linkable_title Implementing an async entity %}
+
+You can make your entity async friendly by converting your update method to be async. This requires the dependency of your entities to also be async friendly!
+
+```python
+class MyEntity(Entity):
+    def update(self):
+        """Retrieve latest state."""
+        self._state = fetch_state()
+```
+
+Will turn into:
+
+```python
+import asyncio
+
+class MyEntity(Entity):
+    @asyncio.coroutine
+    def async_update(self):
+        """Retrieve latest state."""
+        self._state = yield from async_fetch_state()
+```
+
+Make sure that all properties defined on your entity do not result in I/O being done. All data has to be fetched inside the update method and cached on the entity. This is because these properties are read from within the event loop and thus doing I/O will result in the core of Home Assistant waiting until your I/O is done.
+
+## {% linkable_title Calling async functions from threads %}
+
+Sometimes it will happen that you’re in a thread and you want to call a function that is only available as async. Home Assistant includes a few async helper utilities to help with this.
+
+In the following example, `say_hello` will schedule `async_say_hello` and block till the function has run and get the result back.
+
+```python
+from homeassistant.util.async import run_callback_threadsafe
+
+def say_hello(hass, target):
+    return run_callback_threadsafe(
+        hass.loop, async_say_hello, target).result()
+
+def async_say_hello(hass, target):
+    return "Hello {}!".format(target)
+```
+
+## {% linkable_title Dealing with passed in functions %}
+
+If your code takes in functions from other code, you will not know which category the function belongs to and how they should be invoked. This usually only occurs if your code supplies an event helper like `mqtt.async_subscribe` or `track_state_change_listener`.
+
+To help with this, there are two helper methods on the hass object that you can call from inside the event loop:
+
+#### {% linkable_title hass.async_run_job %}
+
+Use this method if the function should be called as soon as possible. This will call callbacks immediately, schedule coroutines for execution on the event loop and schedule other functions to be run inside the thread pool.
+
+| Callback | Call immediately.
+| Coroutine | Schedule for execution on the event loop.
+| Other functions | Schedule for execution in the thread pool.
+
+#### {% linkable_title hass.async_add_job %}
+
+Use this method if the function should be called but not get priority over already scheduled calls.
+
+| Callback | Schedule for execution on the event loop.
+| Coroutine | Schedule for execution on the event loop.
+| Other functions | Schedule for execution in the thread pool.
+
+### [Next step: Miscellaneous »](/developers/asyncio_misc/)
+
+[dev-docs]: https://dev-docs.home-assistant.io/en/master/api/core.html
+[dev-docs-async]: https://dev-docs.home-assistant.io/en/dev/api/util.html#module-homeassistant.util.async

source/developers/development_validation.markdown

@@ -82,5 +82,5 @@ SENSOR_TYPES = {
 PLATFORM_SCHEMA = PLATFORM_SCHEMA.extend({
     ...
     vol.Optional(CONF_MONITORED_VARIABLES, default=[]):
-        vol.All(ensure_list, [vol.In(SENSOR_TYPES)]),
+        vol.All(cv.ensure_list, [vol.In(SENSOR_TYPES)]),
 ```

source/developers/python_api.markdown

@@ -11,9 +11,11 @@ footer: true
 
 In the package [`homeassistant.remote`](https://github.com/home-assistant/home-assistant/blob/master/homeassistant/remote.py) a Python API on top of the [HTTP API](/developers/api/) can be found.
 
-This page is not a full documentation it's more a collection of some example. A simple way to get all current entities is to visit the "Set State" page in the "Developer Tools". For the examples below just choose one from the available entries. Here the sensor `sensor.office_temperature` and the switch `switch.livingroom_pin_2` are used. 
+Note: This page is not full documentation for this API, but a collection of examples showing its use.
 
-First, import the module and setup the basics.
+A simple way to get all current entities is to visit the "Set State" page in the "Developer Tools". For the examples below just choose one from the available entries. Here the sensor `sensor.office_temperature` and the switch `switch.livingroom_pin_2` are used.
+
+First import the module and setup the basics:
 
 ```python
 import homeassistant.remote as remote
@@ -22,7 +24,7 @@ api = remote.API('127.0.0.1', 'password')
 print(remote.validate_api(api))
 ```
 
-This snippet shows how to use the `homeassistant.remote` package in another way.
+Here's another way to use the `homeassistant.remote` package:
 
 ```python
 import homeassistant.remote as remote
@@ -35,7 +37,7 @@ living_room = hass.states.get('group.living_room')
 
 ### {% linkable_title Get configuration %}
 
-Get the current configuration of a Home Assistant instance.
+Get the current configuration of a Home Assistant instance:
 
 ```python
 import homeassistant.remote as remote
@@ -47,7 +49,7 @@ print(remote.get_config(api))
 
 ### {% linkable_title Get details about services, events, and entitites %}
 
-Similar to the output in the "Developer Tools" of the frontend.
+The output from this is similar to the output you'd find via the frontend, using the DevTools console.
 
 ```python
 import homeassistant.remote as remote
@@ -72,7 +74,7 @@ for entity in entities:
 
 ### {% linkable_title Get the state of an entity %}
 
-To get the details of a single entity the `get_state` method is used. 
+To get the details of a single entity, use `get_state`: 
 
 ```python
 import homeassistant.remote as remote
@@ -86,13 +88,13 @@ print('{} is {} {}.'.format(office_temperature.attributes['friendly_name'],
       )
 ```
 
-The output is composed out of the details which are stored for this entity.
+This outputs the details which are stored for this entity, ie:
 
 ```bash
 Office Temperature is 19 °C.
 ```
 
-The exact same thing is working for a switch. The difference is that both entities have different attributes.
+Switches work the same way. The only difference is that both entities have different attributes.
 
 ```python
 import homeassistant.remote as remote
@@ -107,7 +109,7 @@ print('{} is {}.'.format(switch_livingroom.attributes['friendly_name'],
 
 ### {% linkable_title Set the state of an entity %}
 
-Of course, it's possible to set the state.
+Of course, it's possible to set the state as well:
 
 ```python
 import homeassistant.remote as remote
@@ -118,11 +120,11 @@ remote.set_state(api, 'sensor.office_temperature', new_state=123)
 remote.set_state(api, 'switch.livingroom_pin_2', new_state=STATE_ON)
 ```
 
-The state will be set to those value until the next update occurs.
+The state will be set to the new values until the next update occurs.
 
 ### {% linkable_title Blinking all entities of a domain %}
 
-If you want to turn on all entities of a domain, just use a service which was retrieved by `get_services`.
+If you want to turn on all entities of a domain, retrieve the service via `get_services` and act on that:
 
 
 ```python
@@ -139,7 +141,7 @@ remote.call_service(api, domain, 'turn_off')
 
 ### {% linkable_title Control a single entity %}
 
-To turn on or off a single switch. The ID of the entity is needed as an attribute.
+To turn on or off a single switch, pass the ID of the entity:
 
 ```python
 import time
@@ -156,9 +158,9 @@ remote.call_service(api, domain, 'turn_off', {'entity_id': '{}'.format(switch_na
 
 ### {% linkable_title Specify a timeout %}
 
-The default timeout for an API call with `call_service` is 5 seconds. Service
+The default timeout for an API call with `call_service` is 5 seconds. Services
 taking longer than this to return will raise
-`homeassistant.exceptions.HomeAssistantError: Timeout` unless provided with a
+`homeassistant.exceptions.HomeAssistantError: Timeout`, unless provided with a
 longer timeout.
 
 ```python
@@ -180,7 +182,7 @@ remote.call_service(api, domain, 'turn_on', {'entity_id': switch_name},
 
 ### {% linkable_title Send a notification %}
 
-The example uses the jabber notification platform to send a single message to the given recipient in the `configuration.yaml` file. 
+The example uses the Jabber notification platform to send a single message to the given recipient in the `configuration.yaml` file: 
 
 ```python
 import homeassistant.remote as remote
@@ -192,4 +194,4 @@ data = {"title":"Test", "message":"A simple test message from HA."}
 remote.call_service(api, domain, 'jabber', data)
 ```
 
-For more details please check the source of [homeassistant.remote](https://github.com/home-assistant/home-assistant/blob/master/homeassistant/remote.py).
+For more details, please check the source of [homeassistant.remote](https://github.com/home-assistant/home-assistant/blob/master/homeassistant/remote.py).

source/developers/rest_api.markdown

@@ -54,7 +54,7 @@ Successful calls will return status code 200 or 201. Other status codes that can
 The API supports the following actions:
 
 #### {% linkable_title GET /api/ %}
-Returns message if API is up and running.
+Returns a message if the API is up and running.
 
 ```json
 {
@@ -142,7 +142,7 @@ $ curl -X GET -H "x-ha-access: YOUR_PASSWORD" \
 ```
 
 #### {% linkable_title GET /api/events %}
-Returns an array of event objects. Each event object contain event name and listener count.
+Returns an array of event objects. Each event object contains event name and listener count.
 
 ```json
 [
@@ -193,7 +193,7 @@ $ curl -X GET -H "x-ha-access: YOUR_PASSWORD" \
 ```
 
 #### {% linkable_title GET /api/history %}
-Returns an array of state changes in the past. Each object contains further detail for the entities.
+Returns an array of state changes in the past. Each object contains further details for the entities.
 
 ```json
 [
@@ -377,7 +377,7 @@ Returns a message if successful.
 ```
 
 #### {% linkable_title POST /api/services/<domain>/<service> %}
-Calls a service within a specific domain. Will return when the service has been executed or 10 seconds has past, whichever comes first.
+Calls a service within a specific domain. Will return when the service has been executed or after 10 seconds, whichever comes first.
 
 You can pass an optional JSON object to be used as `service_data`.
 
@@ -416,7 +416,7 @@ $ curl -X POST -H "x-ha-access: YOUR_PASSWORD" \
 ```
 
 <p class='note'>
-The result will include any changed states that changed while the service was being executed, even if their change was the result of something else happening in the system.
+The result will include any states that changed while the service was being executed, even if their change was the result of something else happening in the system.
 </p>
 
 #### {% linkable_title POST /api/template %}
@@ -443,7 +443,7 @@ $ curl -X POST -H "x-ha-access: YOUR_PASSWORD" \
 ```
 
 #### {% linkable_title POST /api/event_forwarding %}
-Setup event forwarding to another Home Assistant instance.
+Set up event forwarding to another Home Assistant instance.
 
 Requires a JSON object that represents the API to forward to.
 
@@ -455,7 +455,7 @@ Requires a JSON object that represents the API to forward to.
 }
 ```
 
-It will return a message if event forwarding was setup successful.
+It will return a message if event forwarding was set up successfully.
 
 ```json
 {

source/developers/server_sent_events.markdown

@@ -22,7 +22,7 @@ $ curl -X GET -H "x-ha-access: YOUR_PASSWORD" \
        -H "Content-Type: application/json" http://localhost:8123/api/stream
 ```
 
-For more comfort put the HTML snippet below in a file `sse.html` in your `www` folder of your Home Assistant configuration directory (`.homeassistant`)
+You can create a convenient view for this by creating an HTML file (`sse.html`) in the `www` folder of your Home Assistant configuration directory (`.homeassistant`). Paste this snippet into the file:
 
 ```html
 <!DOCTYPE html>
@@ -44,7 +44,7 @@ Visit [http://localhost:8123/local/sse.html](http://localhost:8123/local/sse.htm
 
 ## {% linkable_title Examples %}
 
-A simplest way to consume server-sent events is `httpie`.
+A simple way to consume server-sent events is to use a command-line http client like [httpie][https://httpie.org/]. Installation info is on the site (if you use Homebrew, it's `brew install httpie`). Once installed, run this snippet from your terminal:
 
 ```bash
 $ http --stream http://localhost:8123/api/stream x-ha-access:YOUR_PASSWORD content-type:application/json
@@ -52,17 +52,21 @@ $ http --stream http://localhost:8123/api/stream x-ha-access:YOUR_PASSWORD conte
 
 ### {% linkable_title Website %}
 
-The [home-assistant-sse](https://github.com/fabaff/home-assistant-sse) repository contains an more advanced example.
+The [home-assistant-sse](https://github.com/fabaff/home-assistant-sse) repository contains a more advanced example.
 
 ### {% linkable_title Python %}
 
-If you want to test the server-sent events without creating a website then the Python module [`sseclient` ](https://pypi.python.org/pypi/sseclient/) can help. Install it first:
+If you want to test the server-sent events without creating a website, the Python module [`sseclient` ](https://pypi.python.org/pypi/sseclient/) can help. To install (assuming Python and pip3 are already installed):
 
 ```bash
 $ pip3 install sseclient
 ```
 
+<<<<<<< HEAD
 The simplest script to consume the SSE looks like the following snippet.
+=======
+The simplest script to consume the SSE in Python looks like this:
+>>>>>>> current
 
 ```python
 from sseclient import SSEClient

source/developers/website.markdown

@@ -72,12 +72,16 @@ You can use the default markdown syntax to generate syntax highlighted code. For
  ```{% endraw %}
 ```
 
-If you are using templates then those parts need to be [escaped](http://stackoverflow.com/a/24102537). Otherwise, they will be rendered and appear blank on the website.
-
 Note that you can replace `yaml` next to \`\`\` with the language that is within the block.
 
 When you're writing code that is to be executed on the terminal, prefix it with `$`.
 
+### {% linkable_title Templates %}
+
+For the [configuration templating](/topics/templating/) is [Jinja](http://jinja.pocoo.org/) used.
+
+If you are using templates then those parts needs to be [escaped](http://stackoverflow.com/a/24102537). Otherwise they will be rendered and appear blank on the website.
+
 ### {% linkable_title HTML %}
 
 The direct usage of HTML is supported but not recommended. The note boxes are an exception.