Add Scene Management Functionality

README.md

@@ -71,6 +71,9 @@ what they are and how to interpret messages from them.
   sensor or push a remote button to get it to download the database).
 - Trigger modem virtual scenes
 - Automatic inbound message de-duplication.
+- Scene/Link Management
+  - Add or delete links on devices from a defined configuration
+  - Import scenes defined on the network into a file for backup
 
 
 # Detailed Documentation
@@ -81,6 +84,8 @@ what they are and how to interpret messages from them.
 
 - [Full system install and automatically starting the server](docs/auto_start.md) on startup.
 
+- [Scene/Link Management](docs/scenes.md)
+
 - [Development Guide](docs/CONTRIBUTING.md)
 
 
@@ -91,11 +96,6 @@ future enhancements:
 
 - Full suite of unit tests.
 - YAML input configuration validation.
-- Heal network (remove records for missing devices, fix missing links).
-- Modem and device scene management.  Define all the links (scenes) in
-  a configuration file and have the system push that information to the
-  devices.  Eliminates the need to do any manual linking of devices and
-  serves as a backup of the Insteon network and scenes.
 
 
 # Thanks

config.yaml

@@ -51,6 +51,14 @@ insteon:
   # startup.  This may be slow depending on the number of devices.
   startup_refresh: False
 
+  # Path to Scenes Definition file (Optional)
+  # The path can be specified either as an absolute path or as a relative path 
+  # using the !rel_path directive.  Where the path is relative to the
+  # config.yaml location
+  #
+  #scenes: /home/user/insteon_mqtt/scenes.yaml
+  #scenes: !rel_path scenes.yaml
+
   #------------------------------------------------------------------------
   # Devices require the Insteon hex address and an optional name. Note
   # that MQTT address topics are always the lower case hex address or
@@ -114,12 +122,6 @@ insteon:
     thermostat:
     #  - aa.bb.cc: 'downstairs'
 
-  #------------------------------------------------------------------------
-  # FUTURE: Insteon scene definitions.
-  #scenes:
-  #  - scene_01:
-
-
 #==========================================================================
 #
 # MQTT configuration
@@ -163,10 +165,19 @@ mqtt:
   #       command_topic: "insteon/modem/scene"
   #       payload_on: { "cmd" : "on", "group" : 10 }
   #       payload_off: { "cmd" : "off", "group" : 10 }
+  # In Home Assistant use MQTT switch with a configuration like this to trigger
+  # modem scene named 'test_scene':
+  #   switch:
+  #     - platform: mqtt
+  #       name: "test_scene"
+  #       command_topic: "insteon/modem/scene"
+  #       payload_on: { "cmd" : "on", "name" : test_scene }
+  #       payload_off: { "cmd" : "off", "name" : test_scene }
   modem:
     # The output of passing the payload through the template must match the
-    # following where group is in the range 1-255.
-    #   { "cmd" : 'on'/'off', "group" : GROUP }
+    # following where group is in the range 1-255 and NAME is a string name
+    # defined in a scenes.yaml file.  Either GROUP or NAME must be specified.
+    #   { "cmd" : 'on'/'off', ["group" : GROUP, "name": NAME] }
     # Available variables for templating are:
     #   value = the input payload
     #   json = the input payload converted to json.  Use json.VAR to extract

docs/mqtt.md

@@ -44,8 +44,10 @@ be identified by it's address or the string "modem".
 
 When adding a new device or after performing a factory reset on a device it
 is necessary to perform a three step process to setup the device to work
-properly with insteon-mqtt.  The steps are 1) Join, 2) Pair, and 3) Sync (Not
-implemented yet).  These three commands can be re-run at anytime without harm.
+properly with insteon-mqtt.  The steps are 1) Join, 2) Pair, and 3) Sync
+(only if you have scenes defined for this device in a scenes.yaml file).  
+The Join and Pair commands can be re-run at anytime without harm.  You can also
+run sync in dry-run mode at any time to see what changes would be made.
 
 From the command line these actions can be performed as follows:
 
@@ -83,12 +85,18 @@ brackets [], then it's optional.  If a value can be one of many like
 true or false, then the possible values are separated by a slash /.
 
 The MQTT topic to publish management commands to is (aa.bb.cc is the
-device address or nice name from the config.yaml file):
+example device address):
 
    ```
    insteon/command/aa.bb.cc
    ```
 
+Alternatively you can use the nice names from the config.yaml file too:
+
+   ```
+   insteon/command/NICE NAME
+   ```
+
 ### Join a New Device to the Network
 
 Supported: devices
@@ -136,12 +144,89 @@ This command can also be run from the command line:
    ```
 
 
-### Sync Device Links (Placeholder)
+### Sync Device Links
+
+Supported: modem, device
+
+This function will alter the device's link database to match the scenes
+defined in the scenes.yaml file.  This includes adding new links as well as
+deleting un-defined links.  Details can be found in [Scene Management](scenes.md)
+
+The command payload is below.  Setting the dry_run flag to true will cause the
+changes to be made to the device, the default false will only report what would
+happen:
+
+  ```
+  { "cmd" : "sync", ["dry_run" : true/false]}
+  ```
+
+  This command can also be run from the command line:
+
+   ```
+   insteon-mqtt config.yaml sync aa.bb.cc
+   ```
+
+### Sync All Device Links
+
+Supported: modem
+
+This function will perform the sync command on all devices.
+
+The command payload is as follows.  Setting the dry_run flag to true will cause
+the changes to be made to the device, the default false will only report what
+would happen:
+
+  ```
+  { "cmd" : "sync_all", ["dry_run" : true/false]}
+  ```
+
+ This command can also be run from the command line:
+
+  ```
+  insteon-mqtt config.yaml sync-all
+  ```
+
+### Import Device Links
 
 Supported: modem, device
 
-This command is not yet supported.
+The 'import-scenes' function will take the links defined on each device and
+parse them into a scene which can be saved to the scenes.yaml file.  Please
+read the in [Scene Management](scenes.md)
+
+The command payload is below.  Setting the dry_run flag to true will cause the
+changes to be made to the file, the default false will only report what would
+happen:
+
+  ```
+  { "cmd" : "import_scenes", ["dry_run" : true/false]}
+  ```
+
+  This command can also be run from the command line:
+
+   ```
+   insteon-mqtt config.yaml import-scenes aa.bb.cc
+   ```
+
+### Sync All Device Links
+
+Supported: modem
+
+This function will perform the import-scenes command on all devices.
+
+The command payload is as follows.  Setting the dry_run flag to true will cause
+the changes to be made to the file, the default false will only report what
+would happen:
 
+  ```
+  { "cmd" : "import_scenes_all", ["dry_run" : true/false]}
+  ```
+
+ This command can also be run from the command line:
+
+  ```
+  insteon-mqtt config.yaml import-scenes-all
+  ```
 
 ### Activate all linking mode
 
@@ -391,6 +476,16 @@ will be passed through to the output state change payload.
    { "cmd": "scene", "group" : group, "is_on" : 0/1, ["reason" : "..."] }
    ```
 
+   Supported: modem
+
+   The modem also allows the triggering of scenes from a name defined in a
+   [Scene Management](scenes.md) file as well. To access a scene by its name
+   simply drop the group attribute and add the name attribute such as.
+
+   ```
+   { "cmd": "scene", "name" : "test_scene", "is_on" : 0/1, ["reason" : "..."] }
+   ```
+
 
 ---
 

docs/quick_start.md

@@ -70,14 +70,13 @@ This package assumes that you:
    insteon-mqtt config.yaml start
    ```
 
-5) Join, Pair, and Sync each device in your network.  This can be accomplished
+5) Join, and Pair each device in your network.  This can be accomplished
    using mqtt messages as described in the [Required Device Initialization]
   (mqtt.md#required-device-initialization) section.
 
   ```
   insteon-mqtt config.yaml join aa.bb.cc
   insteon-mqtt config.yaml pair aa.bb.cc
-  insteon-mqtt config.yaml sync aa.bb.cc
   ```
 
 6) Download an Insteon device database for every device.  This may

docs/scenes.md

@@ -0,0 +1,158 @@
+# Scene Management
+The scenes.yaml file and the 'sync' and 'import-scenes' functions provide the
+user with a way to control the links between devices on their Insteon network
+from a configuration file.
+
+## A Brief Description of Links and Scenes
+An Insteon Scene is made up of a controller and one or more responders.  A
+command issued from the controller simultaneously causes each of the responders
+to respond in a pre-defined manner.  For example, pressing a KeypadLinc button
+(the controller) causes a SwitchLinc device (the responder) to turn on.
+
+The Scene is itself made up of a link on the controller for each responder in
+the scene and a link on each responder corresponding to the controller.  If
+one of these links is missing or wrong, the Scene may not work, or may work in
+an odd manner.  Each device has a local database of its controller and responder
+links.
+
+We have attempted to use the terms 'link' and 'scene' in the manner described
+above, but in some cases these terms may have been commingled. This is in
+part because the Insteon specification is itself not very good at keeping these
+terms straight.  Apologies for whatever confusion results.
+
+## Options for Managing Scenes
+### Physically
+Insteon has a process by which you can manually create and delete scenes. We
+won't go into detail, but generally this involves pressing and holding a button
+for a certain period of time on the controller and each responder.  It may or
+may not be helpful to invoke some magic incantation when doing this process.
+Details about how to do this can be found in the instruction manuals that came
+with each device.  What? You don't save those little scraps of paper? Well
+you can also find most of the manuals here as well
+[Insteon Support](https://www.insteon.com/support)
+
+### Using the Command Line
+Insteon-mqtt contains the 'db-add' and 'db-delete' commands which allow the
+user to create and delete scenes from the command line. This can be helpful
+defining more complex scenes using on_levels, when dealing with devices buried
+in walls or behind objects (such as fanlincs), or when you don't want to get
+anymore exercise walking around pressing buttons.
+
+### Using a Scenes.yaml file
+This is primarily what this document is about.  Insteon-mqtt offers the
+user the ability to define scenes in a configuration file and to synchronize
+the device databases on their network to match the defined configuration.
+Insteon-mqtt also allows the user to define scenes using one of the two above
+methods, or using some other means, and to import those scenes into a
+configuration file, either for backup or to allow for modifications.
+
+## Scenes.yaml
+### Defining a Scenes.yaml File
+The scenes.yaml file location has to be defined in the config.yaml file. An
+example of how the definition should be written can be found in the
+[config.yaml](../config.yaml) file.  The scenes.yaml file needs to be defined under
+the insteon key using the scenes key. If a scenes.yaml file is not defined, the
+results of 'import-scenes' cannot be saved to disk.
+
+### The Format of the Scenes.yaml File
+The scenes.yaml file has a simple structure that allows the user to quickly
+define one or more controllers and one or more responders to define a scene.
+
+A sample [scenes.yaml](../scenes.yaml) is included which describes in detail how
+to define a scene in the scenes.yaml file.  You do not need to place anything
+in your scenes.yaml file to use the 'import-scenes' function.  As long as the
+file is defined in your config.yaml file, and the location is writable by
+insteon-mqtt, a scenes.yaml file will be created by the 'import-scenes'
+function.
+
+### A Note about the Modem and Group Numbers
+The modem can control devices using virtual scenes.  This can be helpful for
+turning on an off a keypadlinc button with a light or for turning off a group
+of lights simultaneously.  While, all modem controller entries require a group
+number, you __should not__ specify one.  A group number will be added
+automatically for you when the scenes file is processed.
+
+### The 'Import-Scenes' Function
+NOTE: Because the 'import-scenes' function may make direct writes to your
+scenes.yaml file. It is recommended that you **backup your scenes.yaml file
+before running 'import-scenes'** to be sure no data is lost.
+
+The 'import-scenes' function will take the links defined on each device and
+parse them into a scene which can be saved to the scenes.yaml file.  The
+'import-scenes' function relies on the locally cached version of each device's
+link database.  As such, it is recommended that you **run the 'refresh' command
+on the device before running 'import-scenes'.**
+
+The 'import-scenes' function will attempt to keep the order and all comments
+in the scenes.yaml file when writing to the file.  However, because scenes
+may be combined or split in the process of importing, **it is not always possible
+to maintain comments and ordering in the scenes.yaml file.**
+
+Much like other insteon-mqtt command.  The 'import-scenes' command can be run
+from either the command line or from mqtt.  Help running [commands from the
+command line](quick_start.md), you can also run `insteon-mqtt config.yaml
+import-scenes -h` for help from the command line.  Help running [mqtt
+commands](mqtt.md).  It is important to note, that by default, the command
+will perform a *dry-run* and will only report the changes that would be made
+unless you tell it to write the changes to disk.
+
+### The 'Import-Scenes-All' Function
+The 'import-scenes-all' function will perform the 'import-scenes' function on
+all devices in the network.  The same caveats about 'import-scenes' apply to
+this function as well.
+
+In addition, the 'import-scenes-all' function can take quite a while to
+complete particularly if you have a lot of devices and scenes and/or a slow
+computer. For reference 85 devices on a raspberry pi takes about 20 seconds
+to complete.  This may cause the command line to time out before the command
+completes.  The command should continue to run and complete in the background
+however, you will not see the results printed to the screen.  You can solve this
+by editing the file (../insteon_mqtt/cmd_line/util.py) and changing the line at
+the top from `TIME_OUT = 10` to something like `TIME_OUT = 30`.
+
+You can run `insteon-mqtt config.yaml import-scenes-all -h` for help from the
+command line.  Help running [mqtt commands](mqtt.md).
+
+### The 'Sync' Function
+The 'sync' function will alter the device's link database to match the scenes
+defined in the scenes.yaml file.  This includes adding new links as well as
+deleting un-defined links.  To repeat, **the 'sync' function will delete links
+on the device that are not present in the scenes.yaml config file.**  By
+default, the command will perform a *dry-run* and will only report the changes
+that would be made unless you tell it to write the changes to the device.
+
+The changes will only be made to the device on which this command is called.  So
+if the user creates a new scene the 'sync' function needs to be called on all
+controllers and responders in order for the scene to work properly.
+
+Links created by the 'pair' or 'join' command will not be deleted or added by
+the 'sync' command.
+
+You can run `insteon-mqtt config.yaml sync -h` for help from the command line.
+Help running [mqtt commands](docs/mqtt.md).
+
+#### Unexpected Sync Changes
+The prudent thing to do before performing a sync, is to perform a *dry-run*
+sync and look at all of the changes that will be made.  It can be helpful to
+also run the print_db command on the relevant devices to see what we know about
+the current state of the device database.
+
+Unexpected additions may be as a result of 1) devices that were not refreshed
+such as battery powered devices or 2) small changes in the on_level or
+ramp_rates.  Unexpected deletions may be the result of 1) duplicate entries
+(the entry exists more than once on the device), 2) small changes in the
+on_level or ramp_rate, 3) links from devices that are no longer present on your
+network.
+
+It is also possible that our understanding of the device database is wrong.
+It is not uncommon for corrupt insteon messages to exist, in this case we may
+believe that the device database is different than it actually is.  Try forcing
+a refresh of the device database and running a *dry-run* sync on the device
+again to see if the changes are still necessary.
+
+### The 'Sync-All' Function
+The 'sync-all' function will perform the 'sync' function on all devices in the
+network.  The same caveats about 'sync' apply to this function as well.
+
+You can run `insteon-mqtt config.yaml sync-all -h` for help from the command
+line. Help running [mqtt commands](docs/mqtt.md).