enhancement request. Please make caliper-fabric adaptor support alternative identity to be used as the client TLS identity for mutualTLS enabled

[aronaks]

Which Caliper version are you using?

latest

Which Node.JS version are you using?

v14.19.0

Which operating system are you using?

macOs Monterey 12.3.1

Please provide some context for your error. For example, when did the error occur? What were you trying to achieve, and how?

I tried to generate load for hp-fabric blockchain SUT with mutualTls enabled. I tried different versions of caliper, namely 0.4.2, 0.5.0 and different versions of fabric adaptor, 1.4, latest-v2.

It is written in the Caliper documentation that mutual TLS is supported. However, I did not find any way to provide Caliper with TLS user private key and certificate neither via network config file nor via connection profile. Maybe I missed something? Please help me to find that out.

What was the observed incorrect behavior?

The observed incorrect behaviour is that caliper-fabric adaptor tries to use invokerIdentity user's private key and certificate for the mutualTLS connection.

You can see on the screenshot below that aliasName is assigned to clientTlsIdentity, but this aliasName is taken from the network certificate organizations.identities.certificates.name and supposed to be used as a invokerIdentity.

fabric-network/src/gateway.ts

So, to conclude, no matter which user and related clientPrivateKey and clientSignedCert I use in organizations.identities.certificates.name, it will end up with the error because that user cannot have two roles at the same time, be a TLS connection user and be an invokerIdentity user.

Please provide the error logs and their surroundings.

2022.04.27-22:02:43.035 info  [caliper] [cli-launch-worker] 	Set SUT type: fabric
2022.04.27-22:02:43.056 info  [caliper] [worker-orchestrator] 	1 workers connected, progressing to worker assignment phase.
2022.04.27-22:02:43.056 info  [caliper] [worker-orchestrator] 	Workers currently unassigned, awaiting index assignment...
2022.04.27-22:02:43.057 info  [caliper] [worker-orchestrator] 	Waiting for 1 workers to be assigned...
2022.04.27-22:02:43.081 info  [caliper] [worker-orchestrator] 	1 workers assigned, progressing to worker initialization phase.
2022.04.27-22:02:43.081 info  [caliper] [worker-orchestrator] 	Waiting for 1 workers to be ready...
2022.04.27-22:02:43.556 info  [caliper] [worker-orchestrator] 	1 workers ready, progressing to test preparation phase.
2022.04.27-22:02:43.556 info  [caliper] [round-orchestrator] 	Started round 1 (first round)
2022.04.27-22:02:43.594 info  [caliper] [worker-message-handler] 	Initializing Worker#0...
2022.04.27-22:02:43.594 info  [caliper] [FabricConnectorFactory] 	Initializing gateway connector compatible with installed SDK: 2.2.11
2022.04.27-22:02:43.594 info  [caliper] [IdentityManager] 	Adding infcauser (admin=false) as infcauser for organization infMSP
2022.04.27-22:02:43.595 info  [caliper] [worker-message-handler] 	Worker#0 initialized
2022.04.27-22:02:43.595 info  [caliper] [worker-message-handler] 	Preparing Worker#0 for Round#0
2022.04.27-22:02:43.595 info  [caliper] [connectors/v2/FabricGateway] 	Connecting user with identity infcauser to a Network Gateway
2022.04.27-22:02:43.750 info  [caliper] [connectors/v2/FabricGateway] 	Successfully connected user with identity infcauser to a Network Gateway
2022.04.27-22:02:43.751 info  [caliper] [connectors/v2/FabricGateway] 	Generating contract map for user infcauser
2022.04.27-22:02:43.806 warn  [caliper] [connectors/v2/FabricGateway] 	Couldn't initialize tnt2 for infcauser. infcauser not available for use on this channel. Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: 8668325376:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 42

2022.04.27-22:02:43.945 info  [caliper] [caliper-worker] 	Worker [0] encountered an error during prepare test phase for round 0: Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: 8668325376:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 42

    at Object.callErrorFromStatus (/Users/aronaks/Projects/INF/inf-caliper-workspace/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/Users/aronaks/Projects/INF/inf-caliper-workspace/node_modules/@grpc/grpc-js/build/src/client.js:189:52)
    at Object.onReceiveStatus (/Users/aronaks/Projects/INF/inf-caliper-workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141)
    at Object.onReceiveStatus (/Users/aronaks/Projects/INF/inf-caliper-workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
    at /Users/aronaks/Projects/INF/inf-caliper-workspace/node_modules/@grpc/grpc-js/build/src/call-stream.js:187:78
    at processTicksAndRejections (internal/process/task_queues.js:77:11)
2022.04.27-22:02:43.946 info  [caliper] [connectors/v2/FabricGateway] 	disconnecting gateway for user infcauser
2022.04.27-22:02:43.946 info  [caliper] [caliper-worker] 	Info: worker 0 prepare test phase for round 0 is completed

Please provide your benchmark configuration file content, if possible.

test:
  name: main
  description: some useful test
  workers:
    number: 1
  rounds:
    - label: first round
      txNumber: 1
      rateControl:
        type: composite-rate
        opts:
          weights: [1, 1, 1, 1]
          rateControllers:
            - type: fixed-rate
              opts:
                tps: 2
            - type: fixed-rate
              opts:
                tps: 3
            - type: fixed-rate
              opts:
                tps: 4
            - type: fixed-rate
              opts:
                tps: 5
      workload:
        module: workload/devLoad.js

Please provide your network configuration file content, if possible.

name: NetworkConfig
version: "2.0.0"

caliper:
  blockchain: fabric
  sutOptions:
    mutualTls: true
  fabric:
    timeout: 5

channels:
  - channelName: tnt2
    contracts:
    - id: identity
      contractID: identity

organizations:
  - mspid: infMSP
    identities:
      certificates:
      - name: 'infcauser'
        clientPrivateKey:
          pem: |-
            -----BEGIN PRIVATE KEY-----
            ...
            -----END PRIVATE KEY-----
        clientSignedCert:
          pem: |-
            -----BEGIN CERTIFICATE-----
            ...
            -----END CERTIFICATE-----
    connectionProfile:
      path: 'networks/fabric/connection-inf.yaml'
      discover: true

Please provide your workload module content, if possible.

'use strict';

const { WorkloadModuleBase } = require('@hyperledger/caliper-core');

class MyWorkload extends WorkloadModuleBase {
    constructor() {
        super();
    }

    async initializeWorkloadModule(workerIndex, totalWorkers, roundIndex, roundArguments, sutAdapter, sutContext) {
        await super.initializeWorkloadModule(workerIndex, totalWorkers, roundIndex, roundArguments, sutAdapter, sutContext);
    }

    async submitTransaction() {

        const requestParams = {
            invokerMspId: 'infMSP',
            channel: 'tnt2',
            contractId: 'identity',
            contractFunction: 'CreateUser',
            contractArguments: [`{"id":"aronaks"}`],
        };

        (async () => {
            try {
              console.log("Are we there before sendRequests?", "YES");

              await this.sutAdapter.sendRequests(requestParams);
            } catch (error) {
              console.log('error', error);
            }
          }
        )();
    }

    async cleanupWorkloadModule() {
        // NOOP
    }
}

function createWorkloadModule() {
    return new MyWorkload();
}

module.exports.createWorkloadModule = createWorkloadModule;

Please provide any additional information you deem relevant to the error.

As a workaround I used the following code below and it works perfectly fine for mutualTLS in my case.

Because clientTlsIdentity is empty, another branch of the if loop will be selected in fabric-network, and TLS data will be taken from defined tlsInfo:
fabric-network/src/gateway.ts

[davidkel]

@aronaks Many thanks for the detailed issue, very helpful.
Until recently (due to new connector not supporting mutual tls) the builds ran with mutualtls enabled and that was working, so I would just like to understand why the build worked for this but fails in your case such that we can ensure in the future that the build sets up an appropriate environment.

I assume the identity you place into options.tlsInfo is a different identity to the one being used to submit transactions ? How are your identities created ? Were there any messages in the peer logs ?

When you do mutualTLS do you have a TLS identity per org or per individual identity ? This would help in defining how we could extend the network configuration file to include information about a TLS identity.

[davidkel]

Initial investigation is that the build uses cryptogen for it's cert creation. This doesn't put any restrictions on the use of a certificate so allows a private_key/certificate identity to be used for TLS as well as an identity which is what the fabric connectors rely on when reusing an identity as a TLS identity for mutual connection.
I've not checked Fabric-CA with how it creates certificates but it's likely to will include usage extensions for the material it generates and external CAs are almost certainly going to do it.

So while caliper does support mutual TLS, it's in a very limited capacility in that identities (ie their certificates in this case) must not restrict their use and also be allowed to be used for TLS comms.

Caliper will need to be enhanced to allow for alternative identities to be supplied to be used as the client TLS identity and probably should support definition at both the Org Level as well as at an individual user level. However I think that support for a TLS identity at an individual user level is not a priority and could be user feedback driven.

[aronaks]

@aronaks Many thanks for the detailed issue, very helpful. Until recently (due to new connector not supporting mutual tls) the builds ran with mutualtls enabled and that was working, so I would just like to understand why the build worked for this but fails in your case such that we can ensure in the future that the build sets up an appropriate environment.

I assume the identity you place into options.tlsInfo is a different identity to the one being used to submit transactions ? How are your identities created ? Were there any messages in the peer logs ?

When you do mutualTLS do you have a TLS identity per org or per individual identity ? This would help in defining how we could extend the network configuration file to include information about a TLS identity.

@davidkel thank you for the answer.

In our case, the identity we place into options.tlsInfo is an identity for TLS auth. It is indeed a different identity to the one being used to submit transactions. The identity for submitting transactions is specified in the network configuration file of caliper (infcauser as you can see above).

In our organisation we have two CAs as it is described here.

We created an identity per end user, and TLS identity per client (caliper in our case).

There were errors like that below on the peer at some point, but I am not sure they are related to the case as we tried different combinations of config:
{"level":"error","ts":1650899534.4987564,"name":"core.comm","caller":"comm/creds.go:109","msg":"Server TLS handshake failed in 306.531µs with error EOF","server":"PeerServer","remote address":"172.21.2.41:40804","stacktrace":"[github.com/hyperledger/fabric/internal/pkg/comm.(*serverCreds).ServerHandshake](http://github.com/hyperledger/fabric/internal/pkg/comm.(*serverCreds).ServerHandshake)\n\t/go/src/github.com/hyperledger/fabric/internal/pkg/comm/creds.go:109\[ngoogle.golang.org/grpc.(*Server).useTransportAuthenticator](http://ngoogle.golang.org/grpc.(*Server).useTransportAuthenticator)\n\t/go/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:635\[ngoogle.golang.org/grpc.(*Server).handleRawConn](http://ngoogle.golang.org/grpc.(*Server).handleRawConn)\n\t/go/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:760\[ngoogle.golang.org/grpc.(*Server).Serve.func3](http://ngoogle.golang.org/grpc.(*Server).Serve.func3)\n\t/go/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:746"}

[davidkel]

What is required here is something like

organizations:
  - mspid: infMSP
    identities:
      certificates:
      - name: 'infcauser'
        clientPrivateKey:
          pem: |-
            -----BEGIN PRIVATE KEY-----
            ...
            -----END PRIVATE KEY-----
        clientSignedCert:
          pem: |-
            -----BEGIN CERTIFICATE-----
            ...
            -----END CERTIFICATE-----
        clientTlsPrivateKey:
          pem: |-
            -----BEGIN PRIVATE KEY-----
            ...
            -----END PRIVATE KEY-----
        clientTlsCert:
          pem: |-
            -----BEGIN CERTIFICATE-----
            ...
            -----END CERTIFICATE-----

we need to also be able to handle the wallet option as well (but that will almost certainly be a naming convention of which ones are tls client certs in the wallet (eg, identities endind with -tls for example)