Multiple Indy Ledger support and State Proof verification

Multiledger.md

@@ -0,0 +1,141 @@
+# Multi-ledger in ACA-Py <!-- omit in toc -->
+
+Ability to use multiple Indy ledgers (both IndySdk and IndyVdr) for resolving a `DID` by the ACA-Py agent. For read requests, checking of multiple ledgers in parallel is done dynamically according to logic detailed in [Read Requests Ledger Selection](#read-requests). For write requests, dynamic allocation of `write_ledger` is not supported. Write ledger can be assigned using `is_write` in the [configuration](#config-properties) or using any of the `--genesis-url`, `--genesis-file`, and `--genesis-transactions` startup (ACA-Py) arguments. If no write ledger is assigned then a `ConfigError` is raised.
+
+More background information including problem statement, design (algorithm) and more can be found [here](https://docs.google.com/document/d/109C_eMsuZnTnYe2OAd02jAts1vC4axwEKIq7_4dnNVA).
+
+## Table of Contents <!-- omit in toc -->
+
+- [Usage](#usage)
+  - [Example config file:](#example-config-file)
+  - [Config properties](#config-properties)
+- [Multi-ledger Admin API](#multi-ledger-admin-api)
+- [Ledger Selection](#ledger-selection)
+  - [Read Requests](#read-requests)
+    - [For checking ledger in parallel](#for-checking-ledger-in-parallel)
+  - [Write Requests](#write-requests)
+- [Impact on other ACA-Py function](#impact-on-other-aca-py-function)
+
+## Usage
+
+Multi-ledger is disabled by default. You can enable support for multiple ledgers using the `--genesis-transactions-list` startup parameter. This parameter accepts a string which is the path to the `YAML` configuration file. For example:
+
+`--genesis-transactions-list ./aries_cloudagent/config/multi_ledger_config.yml`
+
+If `--genesis-transactions-list` is specified, then `--genesis-url, --genesis-file, --genesis-transactions` should not be specified.
+
+### Example config file:
+```
+- id: localVON
+  is_production: false
+  genesis_url: 'http://host.docker.internal:9000/genesis'
+- id: bcorvinTest
+  is_production: true
+  is_write: true
+  genesis_url: 'http://test.bcovrin.vonx.io/genesis'
+```
+
+### Config properties
+For each ledger, the required properties are as following:
+
+- `id`\*: The id (or name) of the ledger, can also be used as the pool name if none provided
+- `is_production`\*: Whether the ledger is a production ledger. This is used by the pool selector algorithm to know which ledger to use for certain interactions (i.e. prefer production ledgers over non-production ledgers)
+
+For connecting to ledger, one of the following needs to be specified:
+
+- `genesis_file`: The path to the genesis file to use for connecting to an Indy ledger.
+- `genesis_transactions`: String of genesis transactions to use for connecting to an Indy ledger.
+- `genesis_url`: The url from which to download the genesis transactions to use for connecting to an Indy ledger.
+
+Optional properties:
+- `pool_name`: name of the indy pool to be opened
+- `keepalive`: how many seconds to keep the ledger open
+- `socks_proxy`
+- `is_write`: Whether the ledger is the write ledger. Only one ledger can be assigned, otherwise a `ConfigError` is raised.
+
+
+## Multi-ledger Admin API
+
+Multi-ledger related actions are grouped under the `ledger` topic in the SwaggerUI or under `/ledger/multiple` path.
+
+- `/ledger/multiple/config`:
+Returns the multiple ledger configuration currently in use
+- `/ledger/multiple/get-write-ledger`:
+Returns the current active/set `write_ledger's` `ledger_id`
+
+## Ledger Selection
+
+### Read Requests
+
+The following process is executed for these functions in ACA-Py:
+1. `get_schema`
+2. `get_credential_definition`
+3. `get_revoc_reg_def`
+4. `get_revoc_reg_entry`
+5. `get_key_for_did`
+6. `get_all_endpoints_for_did`
+7. `get_endpoint_for_did`
+8. `get_nym_role`
+9. `get_revoc_reg_delta`
+
+If multiple ledgers are configured then `IndyLedgerRequestsExecutor` service extracts `DID` from the record identifier and executes the [check](#for-checking-ledger-in-parallel) below, else it returns the `BaseLedger` instance.
+
+#### For checking ledger in parallel
+
+- `lookup_did_in_configured_ledgers` function
+  - If the calling function (above) is in [1-4], then check the `DID` in `cache` for a corresponding applicable `ledger_id`. If found, return the ledger info, else continue.
+  - Otherwise, launch parallel `_get_ledger_by_did` tasks for each of the configured ledgers.
+  - As these tasks get finished, construct `applicable_prod_ledgers` and `applicable_non_prod_ledgers` dictionaries, each with `self_certified` and `non_self_certified` inner dict which are sorted by the original order or index. 
+  - Order/preference for selection: `self_certified` > `production` > `non_production`
+    - Checks `production` ledger where the `DID` is `self_certified`
+    - Checks `non_production` ledger where the `DID` is `self_certified`
+    - Checks `production` ledger where the `DID` is not `self_certified`
+    - Checks `non_production` ledger where the `DID` is not `self_certified`
+  - Return an applicable ledger if found, else raise an exception.
+- `_get_ledger_by_did` function
+  - Build and submit `GET_NYM`
+  - Wait for a response for 10 seconds, if timed out return None
+  - Parse response
+  - Validate state proof
+  - Check if `DID` is self certified
+  - Returns ledger info to `lookup_did_in_configured_ledgers`
+
+### Write Requests
+
+On startup, the first configured applicable ledger is assigned as the `write_ledger` [`BaseLedger`], the selection is dependant on the order (top-down) and whether it is `production` or `non_production`. For instance, considering this [example configuration](#example-config-file), ledger `bcorvinTest` will be set as `write_ledger` as it is the topmost `production` ledger. If no `production` ledgers are included in configuration then the topmost `non_production` ledger is selected.
+
+## Impact on other ACA-Py function
+
+There should be no impact/change in functionality to any ACA-Py protocols.
+
+`IndySdkLedger` was refactored by replacing `wallet: IndySdkWallet` instance variable with `profile: Profile` and accordingly `.aries_cloudagent/indy/credex/verifier`, `.aries_cloudagent/indy/models/pres_preview`, `.aries_cloudagent/indy/sdk/profile.py`, `.aries_cloudagent/indy/sdk/verifier`, `./aries_cloudagent/indy/verifier` were also updated.
+
+Added `build_and_return_get_nym_request` and `submit_get_nym_request` helper functions to `IndySdkLedger` and `IndyVdrLedger`.
+
+Best practice/feedback emerging from `Askar session deadlock` issue and `endorser refactoring` PR was also addressed here by not leaving sessions open unnecessarily and changing `context.session` to `context.profile.session`, etc. 
+
+These changes are made here:
+- `./aries_cloudagent/ledger/routes.py`
+- `./aries_cloudagent/messaging/credential_definitions/routes.py`
+- `./aries_cloudagent/messaging/schemas/routes.py`
+- `./aries_cloudagent/protocols/actionmenu/v1_0/routes.py`
+- `./aries_cloudagent/protocols/actionmenu/v1_0/util.py`
+- `./aries_cloudagent/protocols/basicmessage/v1_0/routes.py`
+- `./aries_cloudagent/protocols/coordinate_mediation/v1_0/handlers/keylist_handler.py`
+- `./aries_cloudagent/protocols/coordinate_mediation/v1_0/routes.py`
+- `./aries_cloudagent/protocols/endorse_transaction/v1_0/routes.py`
+- `./aries_cloudagent/protocols/introduction/v0_1/handlers/invitation_handler.py`
+- `./aries_cloudagent/protocols/introduction/v0_1/routes.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_issue_handler.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_offer_handler.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_proposal_handler.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_request_handler.py`
+- `./aries_cloudagent/protocols/issue_credential/v1_0/routes.py`
+- `./aries_cloudagent/protocols/issue_credential/v2_0/routes.py`
+- `./aries_cloudagent/protocols/present_proof/v1_0/handlers/presentation_handler.py`
+- `./aries_cloudagent/protocols/present_proof/v1_0/handlers/presentation_proposal_handler.py`
+- `./aries_cloudagent/protocols/present_proof/v1_0/handlers/presentation_request_handler.py`
+- `./aries_cloudagent/protocols/present_proof/v1_0/routes.py`
+- `./aries_cloudagent/protocols/trustping/v1_0/routes.py`
+- `./aries_cloudagent/resolver/routes.py`
+- `./aries_cloudagent/revocation/routes.py`

aries_cloudagent/askar/profile.py

@@ -64,23 +64,23 @@ def init_ledger_pool(self):
         if self.settings.get("ledger.disabled"):
             LOGGER.info("Ledger support is disabled")
             return
-
-        pool_name = self.settings.get("ledger.pool_name", "default")
-        keepalive = int(self.settings.get("ledger.keepalive", 5))
-        read_only = bool(self.settings.get("ledger.read_only", False))
-        socks_proxy = self.settings.get("ledger.socks_proxy")
-        if read_only:
-            LOGGER.error("Note: setting ledger to read-only mode")
-        genesis_transactions = self.settings.get("ledger.genesis_transactions")
-        cache = self.context.injector.inject_or(BaseCache)
-        self.ledger_pool = IndyVdrLedgerPool(
-            pool_name,
-            keepalive=keepalive,
-            cache=cache,
-            genesis_transactions=genesis_transactions,
-            read_only=read_only,
-            socks_proxy=socks_proxy,
-        )
+        if self.settings.get("ledger.genesis_transactions"):
+            pool_name = self.settings.get("ledger.pool_name", "default")
+            keepalive = int(self.settings.get("ledger.keepalive", 5))
+            read_only = bool(self.settings.get("ledger.read_only", False))
+            socks_proxy = self.settings.get("ledger.socks_proxy")
+            if read_only:
+                LOGGER.error("Note: setting ledger to read-only mode")
+            genesis_transactions = self.settings.get("ledger.genesis_transactions")
+            cache = self.context.injector.inject_or(BaseCache)
+            self.ledger_pool = IndyVdrLedgerPool(
+                pool_name,
+                keepalive=keepalive,
+                cache=cache,
+                genesis_transactions=genesis_transactions,
+                read_only=read_only,
+                socks_proxy=socks_proxy,
+            )
 
     def bind_providers(self):
         """Initialize the profile-level instance providers."""
@@ -118,6 +118,7 @@ def bind_providers(self):
             injector.bind_provider(
                 BaseLedger, ClassProvider(IndyVdrLedger, self.ledger_pool, ref(self))
             )
+        if self.ledger_pool or self.settings.get("ledger.ledger_config_list"):
             injector.bind_provider(
                 IndyVerifier,
                 ClassProvider(

aries_cloudagent/askar/tests/test_profile.py

@@ -26,6 +26,7 @@ async def test_remove_success(self, AskarOpenStore):
         context.settings = {
             "multitenant.wallet_type": "askar-profile",
             "wallet.askar_profile": profile_id,
+            "ledger.genesis_transactions": mock.MagicMock(),
         }
         askar_profile = AskarProfile(openStore, context)
         remove_profile_stub = asyncio.Future()

aries_cloudagent/commands/provision.py

@@ -7,7 +7,11 @@
 from ..config import argparse as arg
 from ..config.default_context import DefaultContextBuilder
 from ..config.base import BaseError
-from ..config.ledger import get_genesis_transactions, ledger_config
+from ..config.ledger import (
+    get_genesis_transactions,
+    ledger_config,
+    load_multiple_genesis_transactions_from_config,
+)
 from ..config.util import common_config
 from ..config.wallet import wallet_config
 from ..protocols.coordinate_mediation.mediation_invite_store import (
@@ -36,7 +40,14 @@ async def provision(settings: dict):
     context = await context_builder.build_context()
 
     try:
-        await get_genesis_transactions(context.settings)
+        if context.settings.get("ledger.ledger_config_list"):
+            await load_multiple_genesis_transactions_from_config(context.settings)
+        if (
+            context.settings.get("ledger.genesis_transactions")
+            or context.settings.get("ledger.genesis_file")
+            or context.settings.get("ledger.genesis_url")
+        ):
+            await get_genesis_transactions(context.settings)
 
         root_profile, public_did = await wallet_config(context, provision=True)
 

aries_cloudagent/config/argparse.py

@@ -748,24 +748,49 @@ def add_arguments(self, parser: ArgumentParser):
                 "connect to the public (outside of corporate network) ledger pool"
             ),
         )
+        parser.add_argument(
+            "--genesis-transactions-list",
+            type=str,
+            required=False,
+            dest="genesis_transactions_list",
+            metavar="<genesis-transactions-list>",
+            env_var="ACAPY_GENESIS_TRANSACTIONS_LIST",
+            help=(
+                "Load YAML configuration for connecting to multiple"
+                " HyperLedger Indy ledgers."
+            ),
+        )
 
     def get_settings(self, args: Namespace) -> dict:
         """Extract ledger settings."""
         settings = {}
         if args.no_ledger:
             settings["ledger.disabled"] = True
         else:
+            configured = False
             if args.genesis_url:
                 settings["ledger.genesis_url"] = args.genesis_url
+                configured = True
             elif args.genesis_file:
                 settings["ledger.genesis_file"] = args.genesis_file
+                configured = True
             elif args.genesis_transactions:
                 settings["ledger.genesis_transactions"] = args.genesis_transactions
-            else:
+                configured = True
+            if args.genesis_transactions_list:
+                with open(args.genesis_transactions_list, "r") as stream:
+                    txn_config_list = yaml.safe_load(stream)
+                    ledger_config_list = []
+                    for txn_config in txn_config_list:
+                        ledger_config_list.append(txn_config)
+                    settings["ledger.ledger_config_list"] = ledger_config_list
+                    configured = True
+            if not configured:
                 raise ArgsParseError(
-                    "One of --genesis-url --genesis-file or --genesis-transactions "
-                    "must be specified (unless --no-ledger is specified to "
-                    "explicitly configure aca-py to run with no ledger)."
+                    "One of --genesis-url --genesis-file, --genesis-transactions "
+                    "or --genesis-transactions-list must be specified (unless "
+                    "--no-ledger is specified to explicitly configure aca-py to"
+                    " run with no ledger)."
                 )
             if args.ledger_pool_name:
                 settings["ledger.pool_name"] = args.ledger_pool_name

aries_cloudagent/config/ledger.py

@@ -4,6 +4,7 @@
 import logging
 import re
 import sys
+import uuid
 
 import markdown
 import prompt_toolkit
@@ -56,6 +57,67 @@ async def get_genesis_transactions(settings: Settings) -> str:
     return txns
 
 
+async def load_multiple_genesis_transactions_from_config(settings: Settings):
+    """Fetch genesis transactions for multiple ledger configuration."""
+
+    ledger_config_list = settings.get("ledger.ledger_config_list")
+    ledger_txns_list = []
+    write_ledger_set = False
+    for config in ledger_config_list:
+        txns = None
+        if "genesis_transactions" in config:
+            txns = config.get("genesis_transactions")
+        if not txns:
+            if "genesis_url" in config:
+                txns = await fetch_genesis_transactions(config.get("genesis_url"))
+            elif "genesis_file" in config:
+                try:
+                    genesis_path = config.get("genesis_file")
+                    LOGGER.info(
+                        "Reading ledger genesis transactions from: %s", genesis_path
+                    )
+                    with open(genesis_path, "r") as genesis_file:
+                        txns = genesis_file.read()
+                except IOError as e:
+                    raise ConfigError(
+                        "Error reading ledger genesis transactions"
+                    ) from e
+        is_write_ledger = (
+            False if config.get("is_write") is None else config.get("is_write")
+        )
+        ledger_id = config.get("id") or str(uuid.uuid4())
+        if is_write_ledger and write_ledger_set:
+            raise ConfigError("Only a single ledger can be is_write")
+        elif is_write_ledger:
+            write_ledger_set = True
+        ledger_txns_list.append(
+            {
+                "id": ledger_id,
+                "is_production": (
+                    True
+                    if config.get("is_production") is None
+                    else config.get("is_production")
+                ),
+                "is_write": is_write_ledger,
+                "genesis_transactions": txns,
+                "keepalive": int(config.get("keepalive", 5)),
+                "read_only": bool(config.get("read_only", False)),
+                "socks_proxy": config.get("socks_proxy"),
+                "pool_name": config.get("pool_name", ledger_id),
+            }
+        )
+    if not write_ledger_set and not (
+        settings.get("ledger.genesis_transactions")
+        or settings.get("ledger.genesis_file")
+        or settings.get("ledger.genesis_url")
+    ):
+        raise ConfigError(
+            "No is_write ledger set and no genesis_url,"
+            " genesis_file and genesis_transactions provided."
+        )
+    settings["ledger.ledger_config_list"] = ledger_txns_list
+
+
 async def ledger_config(
     profile: Profile, public_did: str, provision: bool = False
 ) -> bool:

aries_cloudagent/config/tests/test-ledger-args.yaml

@@ -0,0 +1,32 @@
+- id: sovrinMain
+  is_production: true
+  genesis_transactions:
+    reqSignature: {}
+    txn:
+      data:
+        data:
+          alias: Node1
+          blskey: >-
+            4N8aUNHSgjQVgkpm8nhNEfDf6txHznoYREg9kirmJrkivgL4oSEimFF6nsQ6M41QvhM2Z33nves5vfSn9n1UwNFJBYtWVnHYMATn76vLuL3zU88KyeAYcHfsih3He6UHcXDxcaecHVz6jhCYz1P2UZn2bDVruL5wXpehgBfBaLKm3Ba
+          blskey_pop: >-
+            RahHYiCvoNCtPTrVtP7nMC5eTYrsUA8WjXbdhNc8debh1agE9bGiJxWBXYNFbnJXoXhWFMvyqhqhRoq737YQemH5ik9oL7R4NTTCz2LEZhkgLJzB3QRQqJyBNyv7acbdHrAT8nQ9UkLbaVL9NBpnWXBTw4LEMePaSHEw66RzPNdAX1
+          client_ip: 192.168.65.3
+          client_port: 9702
+          node_ip: 192.168.65.3
+          node_port: 9701
+          services:
+            - VALIDATOR
+        dest: Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv
+      metadata:
+        from: Th7MpTaRZVRYnPiabds81Y
+      type: '0'
+    txnMetadata:
+      seqNo: 1
+      txnId: fea82e10e894419fe2bea7d96296a6d46f50f93f9eeda954ec461b2ed2950b62
+    ver: '1'
+- id: sovrinStaging
+  is_production: true
+  genesis_file: /home/indy/ledger/sandbox/pool_transactions_genesis
+- id: sovrinTest
+  is_production: false
+  genesis_url: 'http://localhost:9000/genesis'