rpc: Upgrade WebSocketClient to support full client functionality (#646)

* Expand WebSocket functionality This commit is multi-faceted. It: 1. Drastically simplifies much of the subscription-related functionality. 2. Expands the WebSocketClient's capabilities to be able to handle other types of requests other than just subscriptions (i.e. it implements the Client trait). Signed-off-by: Thane Thomson <[email protected]> * The http-client feature also needs the tracing lib Signed-off-by: Thane Thomson <[email protected]> * Make client mutable in example Signed-off-by: Thane Thomson <[email protected]> * Update CHANGELOG Signed-off-by: Thane Thomson <[email protected]> * Update ADR-008 to reflect recent changes Signed-off-by: Thane Thomson <[email protected]> * Fix minor bug in response handling Signed-off-by: Thane Thomson <[email protected]> * Update tests to use new WebSocketClient functionality Signed-off-by: Thane Thomson <[email protected]> * Fix/ignore clippy warnings Signed-off-by: Thane Thomson <[email protected]> * Fix documentation for rpc crate Signed-off-by: Thane Thomson <[email protected]> * Fix broken link in crate docs Signed-off-by: Thane Thomson <[email protected]> * Update RPC repo docs Signed-off-by: Thane Thomson <[email protected]> * Refactor to remove ChannelTx mutability ChannelTx::send now no longer needs to be mutable because the underlying channel has no mutability requirement. This leads to viral changes throughout the interfaces and clients (positive ones). ChannelTx::send also doesn't need to be async, because it relies on an unbounded channel. This also has some viral implications for other methods throughout the RPC client. If, in future, we want to support bounded channels, then we can consider making it async again. But we probably shouldn't make it async if we don't need it to be. Signed-off-by: Thane Thomson <[email protected]> * Client no longer needs to be mutable Signed-off-by: Thane Thomson <[email protected]> * Use explicit borrow_mut to avoid having to obtain subscriptions map twice Signed-off-by: Thane Thomson <[email protected]> * Rename method for clarity Signed-off-by: Thane Thomson <[email protected]>
informalsystems · Nov 26, 2020 · 5105efe · 5105efe
1 parent b618169
commit 5105efe
Show file tree

Hide file tree

Showing 38 changed files with 938 additions and 1,202 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,8 +3,12 @@
 ### IMPROVEMENTS:
 
 - `[light-client]` Only require Tokio when `rpc-client` feature is enabled ([#425])
+- `[rpc]` The `WebSocketClient` now adds support for all remaining RPC requests
+  by way of implementing the `Client` trait ([#646])
 
 [#425]: https://github.com/informalsystems/tendermint-rs/issues/425
+[#646]: https://github.com/informalsystems/tendermint-rs/pull/646
+
 
 ## v0.17.0-rc3
 

diff --git a/docs/architecture/adr-008-event-subscription.md b/docs/architecture/adr-008-event-subscription.md
@@ -218,7 +218,7 @@ WebSocket connection to provide subscription functionality (the
 #[async_trait]
 pub trait SubscriptionClient {
     /// `/subscribe`: subscribe to receive events produced by the given query.
-    async fn subscribe(&mut self, query: String) -> Result<Subscription>;
+    async fn subscribe(&mut self, query: Query) -> Result<Subscription>;
 }
 ```
 
@@ -323,19 +323,23 @@ pub enum EventType {
     ValidatorSetUpdates,
 }
 
-pub struct Condition {
-    key: String,
-    op: Operation,
-}
-
-pub enum Operation {
-    Eq(Operand),
-    Lt(Operand),
-    Lte(Operand),
-    Gt(Operand),
-    Gte(Operand),
-    Contains(Operand),
-    Exists,
+// A condition specifies a key (first parameter) and, depending on the
+// operation, an value which is an operand of some kind.
+pub enum Condition {
+    // Equals
+    Eq(String, Operand),
+    // Less than
+    Lt(String, Operand),
+    // Less than or equal to
+    Lte(String, Operand),
+    // Greater than
+    Gt(String, Operand),
+    // Greater than or equal to
+    Gte(String, Operand),
+    // Contains (to check if a key contains a certain sub-string)
+    Contains(String, String),
+    // Exists (to check if a key exists)
+    Exists(String),
 }
 
 // According to https://docs.tendermint.com/master/rpc/#/Websocket/subscribe,
@@ -346,7 +350,8 @@ pub enum Operation {
 // operand types to the `Operand` enum, as this would improve ergonomics.
 pub enum Operand {
     String(String),
-    Integer(i64),
+    Signed(i64),
+    Unsigned(u64),
     Float(f64),
     Date(chrono::Date),
     DateTime(chrono::DateTime),
@@ -361,7 +366,7 @@ track of all of the queries relating to a particular client.
 ```rust
 pub struct SubscriptionRouter {
     // A map of queries -> (map of subscription IDs -> result event tx channels)
-    subscriptions: HashMap<Query, HashMap<SubscriptionId, ChannelTx<Result<Event>>>>,
+    subscriptions: HashMap<String, HashMap<String, SubscriptionTx>>,
 }
 ```
 
@@ -372,21 +377,61 @@ server [drops subscription IDs from events][tendermint-2949], which is likely if
 we want to conform more strictly to the [JSON-RPC standard for
 notifications][jsonrpc-notifications].
 
-#### Two-Phase Subscribe/Unsubscribe
+### Handling Mixed Events and Responses
+
+Since a full client needs to implement both the `Client` and
+`SubscriptionClient` traits, for certain transports (like a WebSocket
+connection) we could end up receiving a mixture of events from subscriptions
+and responses to RPC requests. To disambiguate these different types of 
+incoming messages, a simple mechanism is proposed for the
+`WebSocketClientDriver` that keeps track of pending requests and only matures
+them once it receives its corresponding response.
 
-Due to the fact that a WebSocket connection lacks request/response semantics,
-when managing multiple subscriptions from a single client we need to implement a
-**two-phase subscription creation/removal process**:
+```rust
+pub struct WebSocketClientDriver {
+    // ...
 
-1. An outgoing, but unconfirmed, subscribe/unsubscribe request is tracked.
-2. The subscribe/unsubscribe request is confirmed or cancelled by a response
-   from the remote WebSocket server.
+    // Commands we've received but have not yet completed, indexed by their ID.
+    // A Terminate command is executed immediately.
+    pending_commands: HashMap<String, DriverCommand>,
+}
 
-The need for this two-phase subscribe/unsubscribe process is more clearly
-illustrated in the following sequence diagram:
+// The different types of requests that the WebSocketClient can send to its
+// driver.
+//
+// Each of SubscribeCommand, UnsubscribeCommand and SimpleRequestCommand keep
+// a response channel that allows for the driver to send a response later on
+// when it receives a relevant one.
+enum DriverCommand {
+    // Initiate a subscription request.
+    Subscribe(SubscribeCommand),
+    // Initiate an unsubscribe request.
+    Unsubscribe(UnsubscribeCommand),
+    // For non-subscription-related requests.
+    SimpleRequest(SimpleRequestCommand),
+    Terminate,
+}
+```
 
-![RPC client two-phase
-subscribe/unsubscribe](./assets/rpc-client-two-phase-subscribe.png)
+IDs of outgoing requests are randomly generated [UUIDv4] strings.
+
+The logic here is as follows:
+
+1. A call is made to `WebSocketClient::subscribe` or
+   `WebSocketClient::perform`.
+2. The client sends the relevant `DriverCommand` to its driver via its internal
+   communication channel.
+3. The driver receives the command, sends the relevant simple or subscription
+   request, and keeps track of the command in its `pending_commands` member
+   along with its ID. This allows the driver to continue handling outgoing 
+   requests and incoming responses in the meantime.
+4. If the driver receives a JSON-RPC message whose ID corresponds to an ID in
+   its `pending_commands` member, it assumes that response is relevant to that
+   command and sends back to the original caller by way of a channel stored in
+   one of the `SubscribeCommand`, `UnsubscribeCommand` or
+   `SimpleRequestCommand` structs. Failures are also communicated through this
+   same mechanism.
+5. The pending command is evicted from the `pending_commands` member.
 
 ## Status
 
@@ -433,4 +478,4 @@ None
 [futures-stream-mod]: https://docs.rs/futures/*/futures/stream/index.html
 [tendermint-2949]: https://github.com/tendermint/tendermint/issues/2949
 [jsonrpc-notifications]: https://www.jsonrpc.org/specification#notification
-
+[UUIDv4]: https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)
diff --git a/docs/architecture/assets/rpc-client-erd.graphml b/docs/architecture/assets/rpc-client-erd.graphml
@@ -94,10 +94,10 @@
     <node id="n7">
       <data key="d6">
         <y:ShapeNode>
-          <y:Geometry height="56.0" width="209.0" x="390.5" y="182.0"/>
+          <y:Geometry height="56.0" width="155.0" x="417.5" y="182.0"/>
           <y:Fill color="#DFC0FF" transparent="false"/>
           <y:BorderStyle color="#000000" raised="false" type="dashed" width="1.0"/>
-          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Helvetica" fontSize="14" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="189.2060546875" x="9.89697265625" xml:space="preserve" y="19.0">TwoPhaseSubscriptionRouter<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
+          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Helvetica" fontSize="14" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="123.0615234375" x="15.96923828125" xml:space="preserve" y="19.0">SubscriptionRouter<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
           <y:Shape type="roundrectangle"/>
         </y:ShapeNode>
       </data>
@@ -216,26 +216,15 @@ Detail)<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:Mo
     <node id="n13">
       <data key="d6">
         <y:ShapeNode>
-          <y:Geometry height="56.0" width="109.0" x="429.5" y="607.0"/>
-          <y:Fill color="#BBDCFC" transparent="false"/>
-          <y:BorderStyle color="#000000" raised="false" type="line" width="1.0"/>
-          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Helvetica" fontSize="14" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="65.482421875" x="21.7587890625" xml:space="preserve" y="19.0">Operation<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
-          <y:Shape type="roundrectangle"/>
-        </y:ShapeNode>
-      </data>
-    </node>
-    <node id="n14">
-      <data key="d6">
-        <y:ShapeNode>
-          <y:Geometry height="56.0" width="109.0" x="616.75" y="607.0"/>
+          <y:Geometry height="56.0" width="109.0" x="440.5" y="607.0"/>
           <y:Fill color="#BBDCFC" transparent="false"/>
           <y:BorderStyle color="#000000" raised="false" type="line" width="1.0"/>
           <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Helvetica" fontSize="14" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="58.482421875" x="25.2587890625" xml:space="preserve" y="19.0">Operand<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
           <y:Shape type="roundrectangle"/>
         </y:ShapeNode>
       </data>
     </node>
-    <node id="n15">
+    <node id="n14">
       <data key="d6">
         <y:ShapeNode>
           <y:Geometry height="56.0" width="155.0" x="1198.0" y="182.0"/>
@@ -305,16 +294,14 @@ Detail)<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:Mo
         </y:PolyLineEdge>
       </data>
     </edge>
-    <edge id="e5" source="n3" target="n7">
+    <edge id="e5" source="n8" target="n7">
       <data key="d10">
         <y:PolyLineEdge>
-          <y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
-            <y:Point x="613.0" y="210.0"/>
-          </y:Path>
+          <y:Path sx="-110.009765625" sy="0.0" tx="0.0" ty="0.0"/>
           <y:LineStyle color="#000000" type="line" width="1.0"/>
           <y:Arrows source="diamond" target="crows_foot_one"/>
-          <y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="28.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="31.33984375" x="-123.43225215534017" xml:space="preserve" y="-51.67025936280061">Has/
-Uses<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="33.36407661646361" distanceToCenter="true" position="left" ratio="0.4515565075001525" segment="0"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
+          <y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="28.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="31.33984375" x="-65.30288833566169" xml:space="preserve" y="8.364076616463592">Has/
+Uses<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="8.36407661646358" distanceToCenter="false" position="left" ratio="0.5954113682210318" segment="-1"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
           <y:BendStyle smoothed="false"/>
         </y:PolyLineEdge>
       </data>
@@ -385,26 +372,15 @@ Uses<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngl
     <edge id="e11" source="n12" target="n13">
       <data key="d10">
         <y:PolyLineEdge>
-          <y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
-          <y:LineStyle color="#000000" type="line" width="1.0"/>
-          <y:Arrows source="diamond" target="crows_foot_one"/>
-          <y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="16.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="25.33984375" x="2.04168701171875" xml:space="preserve" y="-22.0">Has<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="14.0" distanceToCenter="true" position="left" ratio="-2.919921875" segment="-1"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
-          <y:BendStyle smoothed="false"/>
-        </y:PolyLineEdge>
-      </data>
-    </edge>
-    <edge id="e12" source="n13" target="n14">
-      <data key="d10">
-        <y:PolyLineEdge>
-          <y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
+          <y:Path sx="24.75" sy="0.0" tx="0.0" ty="0.0"/>
           <y:LineStyle color="#000000" type="line" width="1.0"/>
           <y:Arrows source="diamond" target="crows_foot_one_optional"/>
-          <y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="16.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="25.33984375" x="1.04168701171875" xml:space="preserve" y="-22.0">Has<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="14.0" distanceToCenter="true" position="left" ratio="-3.919921875" segment="-1"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
+          <y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="16.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="25.33984375" x="1.0521240234375" xml:space="preserve" y="-22.0">Has<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="14.0" distanceToCenter="true" position="left" ratio="-3.919921875" segment="-1"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
           <y:BendStyle smoothed="false"/>
         </y:PolyLineEdge>
       </data>
     </edge>
-    <edge id="e13" source="n5" target="n15">
+    <edge id="e12" source="n5" target="n14">
       <data key="d10">
         <y:PolyLineEdge>
           <y:Path sx="0.0" sy="-11.0" tx="0.0" ty="0.0">
@@ -417,8 +393,7 @@ Uses<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngl
         </y:PolyLineEdge>
       </data>
     </edge>
-    <edge id="e14" source="n3" target="n0">
-      <data key="d9"/>
+    <edge id="e13" source="n3" target="n0">
       <data key="d10">
         <y:PolyLineEdge>
           <y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">

diff --git a/docs/architecture/assets/rpc-client-erd.png b/docs/architecture/assets/rpc-client-erd.png
diff --git a/rpc/Cargo.toml b/rpc/Cargo.toml
@@ -25,7 +25,15 @@ all-features = true
 
 [features]
 default = []
-http-client = [ "async-trait", "futures", "http", "hyper", "tokio/fs", "tokio/macros" ]
+http-client = [
+  "async-trait",
+  "futures",
+  "http",
+  "hyper",
+  "tokio/fs",
+  "tokio/macros",
+  "tracing"
+]
 secp256k1 = [ "tendermint/secp256k1" ]
 websocket-client = [
   "async-trait",

diff --git a/rpc/README.md b/rpc/README.md
@@ -21,25 +21,23 @@ This crate optionally provides access to different types of RPC client
 functionality and different client transports based on which features you
 select when using it.
 
-Two features are provided at present:
+Two features are provided at present.
 
 * `http-client` - Provides `HttpClient`, which is a basic RPC client that
   interacts with remote Tendermint nodes via **JSON-RPC over HTTP**. This
-  client does not provide `Event` subscription functionality. See the 
+  client does not provide `Event` subscription functionality. See the
   [Tendermint RPC] for more details.
-* `websocket-client` - Provides `WebSocketClient`, which currently only 
-  provides `Event` subscription functionality over a WebSocket connection. See
-  the [`/subscribe` endpoint] in the Tendermint RPC for more details. This 
-  client does not yet provide access to the RPC methods provided by the
-  `Client` trait (this is planned for a future release).
+* `websocket-client` - Provides `WebSocketClient`, which provides full
+  client functionality, including general RPC functionality (such as that
+  provided by `HttpClient`) as well as `Event` subscription
+  functionality.
 
 ### Mock Clients
 
 Mock clients are included when either of the `http-client` or
 `websocket-client` features are enabled to aid in testing. This includes
-`MockClient`, which only implements `Client` (no subscription
-functionality), and `MockSubscriptionClient`, which helps you simulate
-subscriptions to events being generated by a Tendermint node.
+`MockClient`, which implements both `Client` and `SubscriptionClient`
+traits.
 
 ### Related