Add docs for auto update client (#1025)

* update docs

* nits

* handle feedback

README.md

@@ -38,6 +38,7 @@ Additional information on how IBC works can be found [here](https://ibc.cosmos.n
 ## Table Of Contents
 - [Basic Usage - Relaying Across Chains](#Basic-Usage---Relaying-Packets-Across-Chains)
 - [Create Path Across Chains](./docs/create-path-across-chain.md)
+- [Advanced Usage](./docs/advanced_usage.md)
 - [Troubleshooting](./docs/troubleshooting.md)
 - [Features](./docs/features.md)
 - [Relayer Terminology](./docs/terminology.md)
@@ -207,8 +208,6 @@ Additional information on how IBC works can be found [here](https://ibc.cosmos.n
      $ rly start 
      ```
 
-    You will need to start a separate shell instance for each path you wish to relay over.
-
     >When running multiple instances of `rly start`, you will need to use the `--debug-addr` flag and provide an address:port. You can also pass an empty string `''`  to turn off this feature or pass `localhost:0` to randomly select a port.
 
     ---

docs/advanced_usage.md

@@ -0,0 +1,56 @@
+# Advanced Usage
+
+## Monitoring
+
+**Prometheus exporter**
+
+If you started `rly` with the default `--debug-addr` argument,
+you can use `http://$IP:7597/metrics` as a target for your prometheus scraper.
+
+**Example metrics**
+
+```
+go_goroutines 29
+...
+go_threads 39
+...
+observed_packets{chain="cosmoshub-4",channel="channel-141",path="hubosmo",port="transfer",type="acknowledge_packet"} 57
+observed_packets{chain="cosmoshub-4",channel="channel-141",path="hubosmo",port="transfer",type="recv_packet"} 103
+observed_packets{chain="cosmoshub-4",channel="channel-141",path="hubosmo",port="transfer",type="send_packet"} 58
+observed_packets{chain="osmosis-1",channel="channel-0",path="hubosmo",port="transfer",type="acknowledge_packet"} 107
+observed_packets{chain="osmosis-1",channel="channel-0",path="hubosmo",port="transfer",type="recv_packet"} 60
+observed_packets{chain="osmosis-1",channel="channel-0",path="hubosmo",port="transfer",type="send_packet"} 102
+...
+relayed_packets{chain="cosmoshub-4",channel="channel-141",path="hubosmo",port="transfer",type="acknowledge_packet"} 31
+relayed_packets{chain="cosmoshub-4",channel="channel-141",path="hubosmo",port="transfer",type="recv_packet"} 65
+relayed_packets{chain="osmosis-1",channel="channel-0",path="hubosmo",port="transfer",type="acknowledge_packet"} 36
+relayed_packets{chain="osmosis-1",channel="channel-0",path="hubosmo",port="transfer",type="recv_packet"} 35
+```
+
+---
+
+## Auto Update Light Client
+
+By default, the Relayer will automatically update clients (`MsgUpdateClient`) if the client has <= 1/3 of its trusting period left. 
+
+> NOTE: The trusting period of the corresponding client is restored with each transaction a relayer relays. In other words, every time a relayer relays a message, it also sends a `MsgUpdateClient` message restarting the time to the clients expiration.*
+
+> This auto-update functionality is specifically useful on low trafficked paths where messages aren't regularly being relayed.
+
+
+You can choose to update clients more regularly by using the `--time-threshold` flag when running the `rly start` command.
+
+Example:
+
+- You are relaying on a path that has a client trusting period of 9 minutes.
+- If no messages are sent for 6 minutes and the client is 3 minutes (1/3) to expiration, the relayer will automatically update the client.
+- If you wish to update the client more frequently, say anytime two minutes have passed without a `MsgUpdateClient` being sent, use flag: `--time-threshold 2m`
+
+Selecting a time-threshold that is greater than 2/3 of the client trusting period will deem itself useless.
+
+\* It is not mandatory for relayers to include the `MsgUpdateClient` when relaying packets, however most, if not all relayers currently do.
+
+---
+
+
+[<-- Create Path Across Chains](create-path-across-chain.md) - [Troubleshooting -->](./troubleshooting.md)

docs/create-path-across-chain.md

@@ -60,4 +60,4 @@ In our "Relaying Packets Across Chains" example, we set up the relayer to relay
 ![banner](./images/github-repo-banner.gif)
  </div>
 
-[<-- Home](../README.md) - [Troubleshooting -->](./troubleshooting.md)
+[<-- Home](../README.md) - [Advanced Usage -->](./advanced_usage.md)

docs/features.md

@@ -11,15 +11,15 @@ The relayer supports the following:
 - sending an UpgradePlan proposal for an IBC breaking upgrade
 - upgrading clients after a counter-party chain has performed an upgrade for IBC breaking changes
 - fetching canonical chain and path metadata from the GitHub repo to quickly bootstrap a relayer instance
+- automatically keep clients alive on low traffic paths
+- scrape metrics via a Prometheus endpoint 
 
 The relayer currently cannot:
 
 - create clients with user chosen parameters (such as UpgradePath)
 - submit IBC client unfreezing proposals
 - monitor and submit misbehavior for clients
-- use IBC light clients other than Tendermint such as Solo Machine
 - connect to chains which don't implement/enable IBC
-- connect to chains using a different IBC implementation (chains not using SDK's `x/ibc` module)
 
 
 [<-- Troubleshooting](./troubleshooting.md) - [Relayer Terminology -->](./terminology.md)

docs/troubleshooting.md

@@ -12,22 +12,18 @@
 
 ---
 
- **Healthy relayer log with no packets to relay should look like:**
-
-   ```log
-2022-03-25T20:11:19.511489Z	info	No packets in queue	{"src_chain_id": "ibc-0", "src_channel_id": "channel-0", "src_port_id": "transfer", "dst_chain_id": "ibc-1", "dst_channel_id": "channel-0", "dst_port_id": "transfer"}
-2022-03-25T20:11:19.514370Z	info	No acknowledgements in queue	{"src_chain_id": "ibc-0", "src_channel_id": "channel-0", "src_port_id": "transfer", "dst_chain_id": "ibc-1", "dst_channel_id": "channel-0", "dst_port_id": "transfer"}
-2022-03-25T20:11:20.517184Z	info	No packets in queue	{"src_chain_id": "ibc-0", "src_channel_id": "channel-0", "src_port_id": "transfer", "dst_chain_id": "ibc-1", "dst_channel_id": "channel-0", "dst_port_id": "transfer"}
-2022-03-25T20:11:20.523035Z	info	No acknowledgements in queue	{"src_chain_id": "ibc-0", "src_channel_id": "channel-0", "src_port_id": "transfer", "dst_chain_id": "ibc-1", "dst_channel_id": "channel-0", "dst_port_id": "transfer"}
-2022-03-25T20:11:21.528712Z	info	No packets in queue	{"src_chain_id": "ibc-0", "src_channel_id": "channel-0", "src_port_id": "transfer", "dst_chain_id": "ibc-1", "dst_channel_id": "channel-0", "dst_port_id": "transfer"}
-2022-03-25T20:11:21.532996Z	info	No acknowledgements in queue	{"src_chain_id": "ibc-0", "src_channel_id": "channel-0", "src_port_id": "transfer", "dst_chain_id": "ibc-1", "dst_channel_id": "channel-0", "dst_port_id": "transfer"}
-2022-03-25T20:11:22.539200Z	info	No packets in queue	{"src_chain_id": "ibc-0", "src_channel_id": "channel-0", "src_port_id": "transfer", "dst_chain_id": "ibc-1", "dst_channel_id": "channel-0", "dst_port_id": "transfer"}
-2022-03-25T20:11:22.543539Z	info	No acknowledgements in queue	{"src_chain_id": "ibc-0", "src_channel_id": "channel-0", "src_port_id": "transfer", "dst_chain_id": "ibc-1", "dst_channel_id": "channel-0", "dst_port_id": "transfer"}
+## **Verify valid `chain`, `client`, and `connection`**
+
+```shell
+$ rly chains list
 ```
 
----
+Output should show all checkboxes:
+```shell
+-> type(cosmos) key(✔) bal(✔) path(✔)
+```
 
-**Verify valid `chain`, `client`, and `connection`**
+## **Verify valid `chain`, `client`, and `connection`**
 
 ```shell
 $ rly paths list
@@ -47,7 +43,7 @@ Your client is the culprit here. Your client may be invalid or expired.
 
 ---
 
-**Inspect Go runtime debug data**
+## **Inspect Go runtime debug data**
 
 If you started `rly` with the default `--debug-addr` argument,
 you can open `http://localhost:7597` in your browser to explore details from the Go runtime.