From 338851c3a1e18be418ab604f0a8b054670a29b09 Mon Sep 17 00:00:00 2001 From: Stephen Curran Date: Thu, 16 Jun 2022 11:55:55 -0700 Subject: [PATCH 1/3] Add troubleshooting document, include initial examples - ledger connection, out-of-sync RevReg Signed-off-by: Stephen Curran --- DevReadMe.md | 1 + README.md | 13 ++++++ Troubleshooting.md | 100 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 114 insertions(+) create mode 100644 Troubleshooting.md diff --git a/DevReadMe.md b/DevReadMe.md index 53e8511fcb..e37f1c624a 100644 --- a/DevReadMe.md +++ b/DevReadMe.md @@ -18,6 +18,7 @@ See the [README](README.md) for details about this repository and information ab - [Developing](#developing) - [Prerequisites](#prerequisites) - [Running Locally](#running-locally) + - [Logging](#logging) - [Running Tests](#running-tests) - [Running Aries Agent Test Harness Tests](#running-aries-agent-test-harness-tests) - [Development Workflow](#development-workflow) diff --git a/README.md b/README.md index 39d61eb421..936847fbc5 100644 --- a/README.md +++ b/README.md @@ -78,6 +78,19 @@ An ACA-Py instance puts together an OpenAPI-documented REST interface based on t Technical note: the administrative API exposed by the agent for the controller to use must be protected with an API key (using the --admin-api-key command line arg) or deliberately left unsecured using the --admin-insecure-mode command line arg. The latter should not be used other than in development if the API is not otherwise secured. +## Troubleshooting + +There are a number of resources for getting help with ACA-Py and troubleshooting +any problems you might run into. The [Troubleshooting](Troubleshooting.md) +document contains some guidance about issues that have been experienced in the +past. Feel free to submit PRs to supplement the troubleshooting document! +Searching the [ACA-Py GitHub +issues](https://github.com/hyperledger/aries-cloudagent-python/issues) will +often uncover challenges that others have experienced, often with answers to +solving those challenges. As well, there is the "aries-cloudagent-python" +channel on the Hyperledger Discord chat server ([invitation +here](https://discord.gg/hyperledger)). + ## Credit The initial implementation of ACA-Py was developed by the Government of British Columbia’s Digital Trust Team in Canada. To learn more about what’s happening with decentralized identity and digital trust in British Columbia, a new website will be launching and the link will be made available here. diff --git a/Troubleshooting.md b/Troubleshooting.md new file mode 100644 index 0000000000..776b2d63fa --- /dev/null +++ b/Troubleshooting.md @@ -0,0 +1,100 @@ +# Troubleshooting Aries Cloud Agent Python + +## Table of Contents + +- [Unable to Connect to Ledger](#unable-to-connect-to-ledger) + - [Local ledger running?](#local-ledger-running) + - [Any Firewalls](#any-firewalls) +- [Damaged, Unpublishable Revocation Registry](#damaged-unpublishable-revocation-registry) + +## Unable to Connect to Ledger + +The most common issue hit by first time users is getting an error on startup "unable to connect to ledger". Here are a list of things to check when you see that error. + +### Local ledger running? + +Unless you specify via startup parameters or environment variables that you are using a public Hyperledger Indy ledger, ACA-Py assumes that you are running a local ledger -- an instance of [von-network](https://github.com/bcgov/von-network). +If that is the cause -- have you started your local ledger, and did it startup properly. Things to check: + +- Any errors in the startup of von-network? +- Is the von-network webserver (usually at `https:/localhost:9000`) accessible? If so, can you click on and see the Genesis File? +- Do you even need a local ledger? If not, you can use a public sandbox ledger, + such as the [Dev Greenlight ledger](), likely by just prefacing your ACA-Py + command with `LEDGER_URL=http://dev.greenlight.bcovrin.vonx.io`. For example, + when running the Alice-Faber demo in the [demo](demo) folder, you can run (for + example), the Faber agent using the command: + `LEDGER_URL=http://dev.greenlight.bcovrin.vonx.io ./run_demo faber` + +### Any Firewalls + +Do you have any firewalls in play that might be blocking the ports that are used by the ledger, notably 9701-9708? To access a ledger +the ACA-Py instance must be able to get to those ports of the ledger, regardless if the ledger is local or remote. + +## Damaged, Unpublishable Revocation Registry + +We have discovered that in the ACA-Py AnonCreds implementation, it is possible +to get into a state where the publishing of updates to a Revocation Registry +(RevReg) is impossible. This can happen where ACA-Py starts to publish an update +to the RevReg, but the write transaction to the Hyperledger Indy ledger fails +for some reason. When a credential revocation is published, aca-py (via indy-sdk +or askar/credx) updates the revocation state in the wallet as well as on the +ledger. The revocation state is dependant on whatever the previous revocation +state is/was, so if the ledger and wallet are mis-matched the publish will fail. +(Andrew/s PR # 1804 (merged) should mitigate but probably won't completely +eliminate this from happening). + +For example, in case we've seen, the write RevRegEntry transaction failed at the +ledger because there was a problem with accepting the TAA (Transaction Author +Agreement). Once the error occurred, the RevReg state held by the ACA-Py agent, +and the RevReg state on the ledger were different. Even after the ability to +write to the ledger was restored, the RevReg could still not be published +because of the differences in the RevReg state. Such a situation can now be +corrected, as follows: + +To address this issue, some new endpoints were added to ACA-Py in Release 0.7.4, +as follows: + +- GET /revocation/registry//issued - counts of the number of issued/revoked + within a registry +- GET /revocation/registry//issued/details - details of all credentials + issued/revoked within a registry +- GET /revocation/registry//issued/indy_recs - calculated rev_reg_delta from + the ledger + - This is used to compare ledger revoked vs wallet revoked credentials, which + is essentially the state of the RevReg on the ledger and in ACA-Py. Where + there is a difference, we have an error. +- PUT /revocation/registry//fix-revocation-entry-state - publish an update + to the RevReg state on the ledger to bring it into alignment with what is in + the ACA-Py instance. + - There is a boolean parameter (`apply_ledger_update`) to control whether the + ledger entry actually gets published so, if you are so inclined, you can + call the endpoint to see what the transaction would be, before you actually + try to do a ledger update. This will return: + - `rev_reg_delta` - same as the ".../indy_recs" endpoint + - `accum_calculated` - transaction to write to ledger + - `accum_fixed` - If `apply_ledger_update`, the transaction actually written + to the ledger + +Note that there is (currently) a backlog item to prevent the wallet and ledger +from getting out of sync (e.g. don't update the ACA-Py RevReg state if the +ledger write fails), but even after that change is made, having this ability +will be retained for use if needed. + +We originally ran into this due to the TAA acceptance getting lost when +switching to multi-ledger (as described +[here](https://github.com/hyperledger/aries-cloudagent-python/blob/main/Multiledger.md#a-special-warning-for-taa-acceptance). +Note that this is one reason how this "out of sync" scenario can occur, but +there may be others. + +We add an integration test that demonstrates/tests this issue [here](https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/features/taa-txn-author-acceptance.feature#L67). + +To run the scenario either manually or using the integration tests, you can do the following: + +- Start von-network in TAA mode: + - `./manage start --taa-sample --logs` +- Start the tails server as usual: + - `./manage start --logs` +- To run the scenario manually, start faber and let the agent know it needs to TAA-accept before doing any ledger writes: + - `./run_demo faber --revocation --taa-accept`, and then you can run through all the transactions using the Swagger page. +- To run the scenario via an integration test, run: + - `./run_bdd -t @taa_required` From b69cc56e90b900b49cc8f719b8956a47a1a81e80 Mon Sep 17 00:00:00 2001 From: Stephen Curran Date: Thu, 16 Jun 2022 12:07:33 -0700 Subject: [PATCH 2/3] Add an intro to the troubleshooting document. Signed-off-by: Stephen Curran --- Troubleshooting.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Troubleshooting.md b/Troubleshooting.md index 776b2d63fa..3a526473c0 100644 --- a/Troubleshooting.md +++ b/Troubleshooting.md @@ -1,5 +1,14 @@ # Troubleshooting Aries Cloud Agent Python +This document contains some troubleshooting information that contributors to the +community think may be helpful. Most of the content here assumes the reader has +gotten started with ACA-Py and has arrived here because of an issue that came up +in their use of ACA-Py. + +Contributions (via pull request) to this document are welcome. Topics added here +will mostly come from reported issues that contributors think would be helpful +to the larger community. + ## Table of Contents - [Unable to Connect to Ledger](#unable-to-connect-to-ledger) From 115b2e75a99a19a18e914b0b071181386731b93a Mon Sep 17 00:00:00 2001 From: Stephen Curran Date: Tue, 21 Jun 2022 10:05:51 -0700 Subject: [PATCH 3/3] Updated endpoints containing id to be backtick quoted Signed-off-by: Stephen Curran --- Troubleshooting.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Troubleshooting.md b/Troubleshooting.md index 3a526473c0..15660e6b45 100644 --- a/Troubleshooting.md +++ b/Troubleshooting.md @@ -63,16 +63,16 @@ corrected, as follows: To address this issue, some new endpoints were added to ACA-Py in Release 0.7.4, as follows: -- GET /revocation/registry//issued - counts of the number of issued/revoked +- GET `/revocation/registry//issued` - counts of the number of issued/revoked within a registry -- GET /revocation/registry//issued/details - details of all credentials +- GET `/revocation/registry//issued/details` - details of all credentials issued/revoked within a registry -- GET /revocation/registry//issued/indy_recs - calculated rev_reg_delta from +- GET `/revocation/registry//issued/indy_recs` - calculated rev_reg_delta from the ledger - This is used to compare ledger revoked vs wallet revoked credentials, which is essentially the state of the RevReg on the ledger and in ACA-Py. Where there is a difference, we have an error. -- PUT /revocation/registry//fix-revocation-entry-state - publish an update +- PUT `/revocation/registry//fix-revocation-entry-state` - publish an update to the RevReg state on the ledger to bring it into alignment with what is in the ACA-Py instance. - There is a boolean parameter (`apply_ledger_update`) to control whether the