Endorser doc updates and some bug fixes

Endorser.md

@@ -1,7 +1,5 @@
 # Transaction Endorser Support
 
-Note that the ACA-Py transaction support is in the process of code refactor and cleanup.  The following documents the current state, but is subject to change.
-
 ACA-Py supports an [Endorser Protocol](https://github.com/hyperledger/aries-rfcs/pull/586), that allows an un-privileged agent (an "Author") to request another agent (the "Endorser") to sign their transactions so they can write these transactions to the ledger.  This is required on Indy ledgers, where new agents will typically be granted only "Author" privileges.
 
 Transaction Endorsement is built into the protocols for Schema, Credential Definition and Revocation, and endorsements can be explicitely requested, or ACA-Py can be configured to automate the endorsement workflow.
@@ -61,3 +59,44 @@ Endorsement:
                         For Authors, specify whether to automatically promote a DID to the wallet public DID after writing to the ledger.
 ```
 
+## How Aca-py Handles Endorsements
+
+Internally, the Endorsement functionality is implemented as a protocol, and is implemented consistently with other protocols:
+
+- a [routes.py](https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/protocols/endorse_transaction/v1_0/routes.py) file exposes the admin endpoints
+- [handler files](https://github.com/hyperledger/aries-cloudagent-python/tree/main/aries_cloudagent/protocols/endorse_transaction/v1_0/handlers) implement responses to any received Endorse protocol messages
+- a [manager.py](https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/protocols/endorse_transaction/v1_0/manager.py) file implements common functionality that is called from both the routes.py and handler classes (as well as from other classes that need to interact with Endorser functionality)
+
+The Endorser makes use of the [Event Bus](https://github.com/hyperledger/aries-cloudagent-python/blob/main/CHANGELOG.md#july-14-2021) (links to the PR which links to a hackmd doc) to notify other protocols of any Endorser events of interest.  For example, after a Credential Definition endorsement is received, the TransactionManager writes the endorsed transaction to the ledger and uses the Event Bus to notify the Credential Defintition manager that it can do any required post-processing (such as writing the cred def record to the wallet, initiating the revocation registry, etc.).
+
+The overall architecture can be illustrated as:
+
+![Class Diagram](./docs/assets/endorser-design.png)
+
+### Create Credential Definition and Revocation Registry
+
+An example of an Endorser flow is as follows, showing how a credential definition endorsement is received and processed, and optionally kicks off the revocation registry process:
+
+![Sequence Diagram](./docs/assets/endorse-cred-def.png)
+
+You can see that there is a standard endorser flow happening each time there is a ledger write (illustrated in the "Endorser" process).
+
+At the end of each endorse sequence, the TransactionManager sends a notification via the EventBus so that any dependant processing can continue.  Each Router is responsible for listening and responding to these notifications if necessary.
+
+For example:
+
+- Once the credential definition is created, a revocation registry must be created (for revocable cred defs)
+- Once the revocation registry is created, a revocation entry must be created
+- Potentially, the cred def status could be updated once the revocation entry is completed
+
+Using the EventBus decouples the event sequence.  Any functions triggered by an event notification are typically also available directly via Admin endpoints.
+
+### Create DID and Promote to Public
+
+... and an example of creating a DID and promoting it to public (and creating an ATTRIB for the endpoint:
+
+![Sequence Diagram](./docs/assets/endorse-public-did.png)
+
+You can see the same endorsement processes in this sequence.
+
+Once the DID is written, the DID can (optionally) be promoted to the public DID, which will also invoke an ATTRIB transaction to write the endpoint.

aries_cloudagent/ledger/indy.py

@@ -793,7 +793,7 @@ async def update_endpoint_for_did(
                     )
                     resp = await self._submit(
                         request_json,
-                        True,
+                        sign=True,
                         sign_did=public_info,
                         write_ledger=write_ledger,
                     )
@@ -802,6 +802,7 @@ async def update_endpoint_for_did(
 
             await self._submit(request_json, True, True)
             return True
+
         return False
 
     async def register_nym(

aries_cloudagent/protocols/endorse_transaction/v1_0/manager.py

@@ -21,7 +21,10 @@
 from ....storage.error import StorageError, StorageNotFoundError
 from ....transport.inbound.receipt import MessageReceipt
 from ....wallet.base import BaseWallet
-from ....wallet.util import notify_endorse_did_event
+from ....wallet.util import (
+    notify_endorse_did_event,
+    notify_endorse_did_attrib_event,
+)
 
 from .messages.cancel_transaction import CancelTransaction
 from .messages.endorsed_transaction_response import EndorsedTransactionResponse
@@ -794,6 +797,11 @@ async def endorsed_txn_post_processing(
             did = ledger_response["result"]["txn"]["data"]["dest"]
             await notify_endorse_did_event(self._profile, did, meta_data)
 
+        elif ledger_response["result"]["txn"]["type"] == "100":
+            # write DID ATTRIB to ledger
+            did = ledger_response["result"]["txn"]["data"]["dest"]
+            await notify_endorse_did_attrib_event(self._profile, did, meta_data)
+
         else:
             # TODO unknown ledger transaction type, just ignore for now ...
             pass

aries_cloudagent/wallet/indy.py

@@ -1,6 +1,7 @@
 """Indy implementation of BaseWallet interface."""
 
 import json
+import logging
 
 from typing import List, Sequence, Tuple, Union
 
@@ -36,6 +37,8 @@
 from .util import b58_to_bytes, bytes_to_b58, bytes_to_b64
 
 
+LOGGER = logging.getLogger(__name__)
+
 RECORD_TYPE_CONFIG = "config"
 RECORD_NAME_PUBLIC_DID = "default_public_did"
 

aries_cloudagent/wallet/routes.py

@@ -17,6 +17,7 @@
 from ..ledger.error import LedgerConfigError, LedgerError
 from ..messaging.models.base import BaseModelError
 from ..messaging.models.openapi import OpenAPISchema
+from ..messaging.responder import BaseResponder
 from ..messaging.valid import (
     DID_POSTURE,
     ENDPOINT,
@@ -442,6 +443,17 @@ async def wallet_set_public_did(request: web.BaseRequest):
     connection_id = request.query.get("conn_id")
     attrib_def = None
 
+    # check if we need to endorse
+    if is_author_role(context.profile):
+        # authors cannot write to the ledger
+        write_ledger = False
+        create_transaction_for_endorser = True
+        if not connection_id:
+            # author has not provided a connection id, so determine which to use
+            connection_id = await get_endorser_connection_id(context.profile)
+            if not connection_id:
+                raise web.HTTPBadRequest(reason="No endorser connection found")
+
     wallet = session.inject_or(BaseWallet)
     if not wallet:
         raise web.HTTPForbidden(reason="No wallet available")
@@ -542,7 +554,6 @@ async def promote_wallet_public_did(
             connection_id = await get_endorser_connection_id(context.profile)
         if not connection_id:
             raise web.HTTPBadRequest(reason="No endorser connection found")
-
     if not write_ledger:
         try:
             async with profile.session() as session:
@@ -594,11 +605,6 @@ async def promote_wallet_public_did(
                     routing_keys=routing_keys,
                 )
 
-        # Commented the below lines as the function set_did_endpoint
-        # was calling update_endpoint_for_did of ledger
-        # async with ledger:
-        #     await ledger.update_endpoint_for_did(info.did, endpoint)
-
         # Route the public DID
         route_manager = profile.inject(RouteManager)
         await route_manager.route_public_did(profile, info.verkey)
@@ -814,14 +820,60 @@ async def on_register_nym_event(profile: Profile, event: Event):
         did = event.payload["did"]
         connection_id = event.payload.get("connection_id")
         try:
-            await promote_wallet_public_did(
+            info, attrib_def = await promote_wallet_public_did(
                 profile, profile.context, profile.session, did, connection_id
             )
-        except Exception:
+        except Exception as err:
+            # log the error, but continue
+            LOGGER.exception(
+                "Error promoting to public DID: %s",
+                err,
+            )
+            return
+
+        transaction_mgr = TransactionManager(profile)
+        try:
+            transaction = await transaction_mgr.create_record(
+                messages_attach=attrib_def["signed_txn"], connection_id=connection_id
+            )
+        except StorageError as err:
             # log the error, but continue
             LOGGER.exception(
                 "Error accepting endorser invitation/configuring endorser connection: %s",
+                err,
             )
+            return
+
+        # if auto-request, send the request to the endorser
+        if profile.settings.get_value("endorser.auto_request"):
+            try:
+                transaction, transaction_request = await transaction_mgr.create_request(
+                    transaction=transaction,
+                    # TODO see if we need to parameterize these params
+                    # expires_time=expires_time,
+                    # endorser_write_txn=endorser_write_txn,
+                )
+            except (StorageError, TransactionManagerError) as err:
+                # log the error, but continue
+                LOGGER.exception(
+                    "Error creating endorser transaction request: %s",
+                    err,
+                )
+
+            # TODO not sure how to get outbound_handler in an event ...
+            # await outbound_handler(transaction_request, connection_id=connection_id)
+            responder = profile.inject_or(BaseResponder)
+            if responder:
+                await responder.send(
+                    transaction_request,
+                    connection_id=connection_id,
+                )
+            else:
+                LOGGER.warning(
+                    "Configuration has no BaseResponder: cannot update "
+                    "ATTRIB record on DID: %s",
+                    did,
+                )
 
 
 async def register(app: web.Application):

aries_cloudagent/wallet/util.py

@@ -102,7 +102,9 @@ def abbr_verkey(full_verkey: str, did: str = None) -> str:
 
 
 DID_EVENT_PREFIX = "acapy::ENDORSE_DID::"
+DID_ATTRIB_EVENT_PREFIX = "acapy::ENDORSE_DID_ATTRIB::"
 EVENT_LISTENER_PATTERN = re.compile(f"^{DID_EVENT_PREFIX}(.*)?$")
+ATTRIB_EVENT_LISTENER_PATTERN = re.compile(f"^{DID_ATTRIB_EVENT_PREFIX}(.*)?$")
 
 
 async def notify_endorse_did_event(profile: Profile, did: str, meta_data: dict):
@@ -111,3 +113,11 @@ async def notify_endorse_did_event(profile: Profile, did: str, meta_data: dict):
         DID_EVENT_PREFIX + did,
         meta_data,
     )
+
+
+async def notify_endorse_did_attrib_event(profile: Profile, did: str, meta_data: dict):
+    """Send notification for a DID ATTRIB post-process event."""
+    await profile.notify(
+        DID_ATTRIB_EVENT_PREFIX + did,
+        meta_data,
+    )

demo/features/0453-issue-credential.feature

@@ -1,3 +1,4 @@
+@RFC0453
 Feature: RFC 0453 Aries agent issue credential
 
   @T003-RFC0453 @GHA

demo/features/0586-sign-transaction.feature

@@ -1,3 +1,4 @@
+@RFC0586
 Feature: RFC 0586 Aries sign (endorse) transactions functions
 
    @T001-RFC0586

demo/features/steps/0586-sign-transaction.py

@@ -34,11 +34,18 @@ def step_impl(context, agent_name, did_role):
     )
 
     # make the new did the wallet's public did
-    created_did = agent_container_POST(
+    published_did = agent_container_POST(
         agent["agent"],
         "/wallet/did/public",
         params={"did": created_did["result"]["did"]},
     )
+    if "result" in published_did:
+        # published right away!
+        pass
+    elif "txn" in published_did:
+        # we are an author and need to go through the endorser process
+        # assume everything works!
+        async_sleep(3.0)
 
     if not "public_dids" in context:
         context.public_dids = {}

demo/runners/support/agent.py

@@ -600,8 +600,10 @@ async def register_or_switch_wallet(
                 if self.endorser_role and self.endorser_role == "author":
                     if endorser_agent:
                         await self.admin_POST("/wallet/did/public?did=" + self.did)
+                        await asyncio.sleep(3.0)
                 else:
                     await self.admin_POST("/wallet/did/public?did=" + self.did)
+                    await asyncio.sleep(3.0)
             elif cred_type == CRED_FORMAT_JSON_LD:
                 # create did of appropriate type
                 data = {"method": DID_METHOD_KEY, "options": {"key_type": KEY_TYPE_BLS}}

docs/assets/endorse-cred-def.puml

@@ -0,0 +1,75 @@
+@startuml
+' List of actors for our use case
+actor Admin
+participant CredDefRoutes
+participant RevocationRoutes
+participant IndyRevocation
+participant Ledger
+participant TransactionManager
+participant EventBus
+participant OutboundHandler
+participant EndorsedTxnHandler
+boundary OtherAgent
+
+' Sequence for writing a new credential definition
+Admin --> CredDefRoutes: POST /credential-definitions
+group Endorse transaction process
+CredDefRoutes --> Ledger: create_and_send_credential_definition()
+CredDefRoutes --> TransactionManager: create_record()
+CredDefRoutes --> TransactionManager: create_request()
+CredDefRoutes --> OutboundHandler: send_outbound_msg()
+OutboundHandler --> OtherAgent: send_msg()
+OtherAgent --> OtherAgent: endorse_msg()
+EndorsedTxnHandler <-- OtherAgent: send_msg()
+TransactionManager <-- EndorsedTxnHandler: receive_endorse_response()
+TransactionManager <-- EndorsedTxnHandler: complete_transaction()
+Ledger <-- TransactionManager: txn_submit()
+TransactionManager --> TransactionManager: endorsed_txn_post_processing()
+TransactionManager --> EventBus: notify_cred_def_event()
+end
+
+' Create the revocation registry once the credential definition is written
+CredDefRoutes <-- EventBus: on_cred_def_event()
+CredDefRoutes --> IndyRevocation: init_issuer_registry()
+IndyRevocation --> EventBus: notify_revocation_reg_init_event()
+RevocationRoutes <-- EventBus: on_revocation_registry_init_event()
+RevocationRoutes --> RevocationRoutes: generate_tails()
+group Endorse transaction process
+RevocationRoutes --> Ledger:send_revoc_reg_def()
+RevocationRoutes --> TransactionManager: create_record()
+RevocationRoutes --> TransactionManager: create_request()
+RevocationRoutes --> OutboundHandler: send_outbound_msg()
+OutboundHandler --> OtherAgent: send_msg()
+OtherAgent --> OtherAgent: endorse_msg()
+EndorsedTxnHandler <-- OtherAgent: send_msg()
+TransactionManager <-- EndorsedTxnHandler: receive_endorse_response()
+TransactionManager <-- EndorsedTxnHandler: complete_transaction()
+Ledger <-- TransactionManager: txn_submit()
+TransactionManager --> TransactionManager: endorsed_txn_post_processing()
+TransactionManager --> EventBus: notify_revocation_reg_endorsed_event()
+end
+
+' Now create the revocation entry (accumulator)
+RevocationRoutes <-- EventBus: on_revocation_registry_endorsed_event()
+RevocationRoutes --> RevocationRoutes: upload_tails()
+RevocationRoutes --> EventBus: notify_revocation_entry_event()
+RevocationRoutes <-- EventBus: on_revocation_entry_event()
+group Endorse transaction process
+RevocationRoutes --> IndyRevocation: send_entry()
+IndyRevocation --> Ledger: send_entry()
+RevocationRoutes --> TransactionManager: create_record()
+RevocationRoutes --> TransactionManager: create_request()
+RevocationRoutes --> OutboundHandler: send_outbound_msg()
+OutboundHandler --> OtherAgent: send_msg()
+OtherAgent --> OtherAgent: endorse_msg()
+EndorsedTxnHandler <-- OtherAgent: send_msg()
+TransactionManager <-- EndorsedTxnHandler: receive_endorse_response()
+TransactionManager <-- EndorsedTxnHandler: complete_transaction()
+Ledger <-- TransactionManager: txn_submit()
+TransactionManager --> TransactionManager: endorsed_txn_post_processing()
+
+' Notify that the revocation entry is completed (no one listens to this notification yet)
+TransactionManager --> EventBus: notify_revocation_entry_endorsed_event()
+end
+
+@enduml