Practices for constructing GRPC clients

[rajat-dlt]

I'm new to navigating Hyperledger Fabric and Grpc connections, and I'd appreciate some guidance on the recommended approach for new development in this environment.

Let's say I have 2-3 peers set up on a Kubernetes cluster, and I've established a Service URL of load balancer (SVC URL) to connect to the gateway.

There are essentially two strategies to consider:

• Establish a GRPC connection for individual requests, closing it afterward.
• Maintain a persistent connection to handle all requests, with the GRPC framework managing reconnections in case of failures.

Initially, I lean towards the latter option. However, in scenarios where a peer goes offline or there's a disruption in the connection, I'm curious about strategies to minimize downtime. I've considered adjusting parameters such as grpc.keepalive_time_ms, grpc.keepalive_timeout_ms, grpc.max_connection_age_ms, and grpc.max_connection_idle_ms for potential improvements.

Unfortunately, I've found limited documentation or guidance on optimizing these parameters. Currently, it seems that without specific values provided, connections may persist indefinitely or not handle reconnection attempts adequately.

What would be the recommended approach to address this? Considering that setting up a GRPC connection entails time, and minimizing latency is crucial, how can we optimize connection management without compromising on system reliability or encountering excessive downtime?

Even in this link, nothing seems to be concluded. #197

[rajat-dlt]

@bestbeforetoday @mbwhite ,
Would you mind assisting me with this matter?
I've been grappling with it for quite a while now.

[bestbeforetoday]

gRPC keep-alive configuration and uses are described in the gRPC documentation.

Depending on whether you want to see and handle transient connection failures in your application code, you might also consider wait-for-ready configuration.

You should always configure appropriate deadlines for invocations, either defaults on the Gateway connect (Go, Node, Java) or per-call (Go, Node, Java), but this becomes even more important if using wait-for-ready otherwise a request to an uncontactable Gateway will block indefinitely.

The bottom of this page from the full-stack-asset-transfer-guide sample has some discussion of fail-over / load balancing options.

[rajat-dlt]

Hi @bestbeforetoday,

Thanks a lot for the sugesstion.
I understand the keep-alive settings can be set using the call options, like:

new grpc.Client("peer0.org1.example.com:7051", tlsCredentials, 
          'grpc.ssl_target_name_override': "peer0.org1.example.com",
          'grpc.keepalive_time_ms': 20000,
          'grpc.keepalive_timeout_ms': 20000,
          'grpc.keepalive_permit_without_calls': 1,
          'grpc.max_connection_age_ms': 60000,
          'grpc.max_connection_idle_ms': 60000,
        });

But how will the wait-for-ready be incorporated, as according to a discussion I read on #197, what if at the time of checking the connection was there and after that, we're facing connection issues?

According to my understanding the code would be something like-

await new Promise<void>((resolve, reject) => {
      this.client.waitForReady(Date.now() + 10000, (error) => {
        if (error) {
          this.logger.error('Client failed to become ready:', error);
          reject(error);
        } else {
          this.logger.info('Client is ready');
          resolve();
        }
      });
    });

const contract = await new  gateway.getNetwork('mychannel').getContract('mycc');

[bestbeforetoday]

I don't think it is necessary do anything this involved. Wait-for-ready simply means that, if the connection is in a transient failure state, the request will block until the connection is re-established instead of failing immediately. If you have set a default timeout / deadline (see example) then the request will still fail when that deadline is reached. If you don't set a default timeout / deadline, a request made when the connection is in a transient failure state could end up waiting indefinitely for the connection to be reestablished.

The default behaviour (with wait-for-ready unset) will still wait for connection to be established if the connection is currently in an idle or connecting state, but will fail immediately on a connection failure.