Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use default wallet type askar for alice/faber demo and bdd tests #1761

Merged
merged 16 commits into from
Jun 30, 2022

Conversation

ianco
Copy link
Contributor

@ianco ianco commented May 5, 2022

Signed-off-by: Ian Costanzo [email protected]

swcurran
swcurran previously approved these changes May 5, 2022
@ianco
Copy link
Contributor Author

ianco commented May 5, 2022

Two of the integration tests are failing, revocation, @T002.1-RFC0586 and @T003.1-RFC0586

The step that is failing is:

      And "Bob" authors a credential definition transaction with <Schema_name>
      And "Bob" has written the credential definition for <Schema_name> to the ledger
      And "Bob" has written the revocation registry definition to the ledger
      And "Bob" has written the revocation registry entry transaction to the ledger
      And "Acme" has an issued <Schema_name> credential <Credential_data> from "Bob"
      And "Bob" revokes the credential without publishing the entry
>>>>> this step fails >>>>>      And "Bob" authors a revocation registry entry publishing transaction
      Then "Acme" can verify the credential from "Bob" was revoked

Looking into it ...

@ianco
Copy link
Contributor Author

ianco commented May 5, 2022

Two of the integration tests are failing, revocation, @T002.1-RFC0586 and @T003.1-RFC0586

The /revocation/publish-revocations endpoint hangs when trying to update the ledger. Specifically the call to did_info = await wallet.get_local_did(issuer_did) hangs in indy_vdr.py here:

https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/ledger/indy_vdr.py#L1345

Some kind of database locking issue ... @andrewwhitehead ?

@swcurran
Copy link
Contributor

swcurran commented May 5, 2022

That sounds pretty important :-).

@ianco
Copy link
Contributor Author

ianco commented May 5, 2022

I think what's happening is this chunk of code here:

https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/revocation/manager.py#L180

... creates and uses a transaction for the duration of the code block, however this line here:

https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/revocation/manager.py#L202

await issuer_rr_rec.send_entry(self._profile)

... doesn't propagate the transaction, so then when the ledger update is done here:

https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/revocation/models/issuer_rev_reg_record.py#L289

... and then within the ledger code it tries to lookup the DID in the wallet:

https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/ledger/indy_vdr.py#L1345

... it's not using the transaction that is already open, so I think that makes it block.

@ianco
Copy link
Contributor Author

ianco commented May 5, 2022

I notice there are some updates in progress to askar so not sure if that will address this issue or we need to fix this issue in aca-py code? I'll leave this for now pending feedback from @andrewwhitehead

@andrewwhitehead
Copy link
Contributor

I don't think the latest changes would impact this. There shouldn't generally be an issue unless it's running out of available pool connections.

@ianco
Copy link
Contributor Author

ianco commented May 6, 2022

It's definitely hitting a database lock. When I move this line await issuer_rr_rec.send_entry(self._profile) (https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/revocation/manager.py#L202) after the txn.commit() everything works fine.

There are database rows getting updated during the transaction and locking the table, and the DID lookup is happening on a different db connection (I assume) and not able to fetch the row from the (locked) table.

@ianco
Copy link
Contributor Author

ianco commented May 9, 2022

The code works with postgres but not with sqlite

Publishing revocation updates works when you auto-publish when the credential is revoked, but not when you later publish revocations (batched)

@codecov-commenter
Copy link

codecov-commenter commented Jun 3, 2022

Codecov Report

Merging #1761 (958be35) into main (8038584) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1761   +/-   ##
=======================================
  Coverage   93.72%   93.72%           
=======================================
  Files         536      536           
  Lines       34015    34015           
=======================================
  Hits        31881    31881           
  Misses       2134     2134           

@swcurran
Copy link
Contributor

swcurran commented Jun 3, 2022

Any ideas on getting past this? Would be really good to have askar as the default going forward for ACA-Py.

@ianco
Copy link
Contributor Author

ianco commented Jun 3, 2022

Any ideas on getting past this? Would be really good to have askar as the default going forward for ACA-Py.

@andrewwhitehead mentioned he could look at this once the other revocation PR is merged. I think we were talking about adding a new state for the individual credential revocation, but now that we have a method to fix the problem I don't think it's as big of an issue of the ledger gets out of sync ...

@swcurran swcurran added the 0.7.4 label Jun 14, 2022
@andrewwhitehead
Copy link
Contributor

Actually I think there's a few more references to update: in docker-compose.yml, alice-local.sh, faber-local.sh, and runners/support/utils.py (check_requires method)

@shaangill025

This comment was marked as outdated.

@andrewwhitehead
Copy link
Contributor

@shaangill025 I think that these errors are due to the agent shutting down prior to all of the messages being sent (a connection is still in progress), and they don't seem to affect the result of the test:

Aries | 2022-06-17 22:53:11,051 aries_cloudagent.transport.outbound.manager ERROR >>> Error when posting to: http://192.168.65.3:8022/webhooks/topic/connections/; Error: (<class 'aiohttp.client_exceptions.ClientOSError'>, ClientOSError(None, 'Can not write request body for http://192.168.65.3:8022/webhooks/topic/connections/'), <traceback object at 0x7f153a1c3888>); Re-queue failed message ...
Aries | 2022-06-17 22:53:11,155 aries_cloudagent.transport.outbound.manager ERROR >>> Error when posting to: http://192.168.65.3:8022/webhooks/topic/connections/; Error: (<class 'aiohttp.client_exceptions.ClientOSError'>, ClientOSError(32, 'Broken pipe'), <traceback object at 0x7f153a16b4c8>); Re-queue failed message ...
Aries |
Aries | Shutting down
Aries | 2022-06-17 22:53:11,531 aries_cloudagent.transport.outbound.manager ERROR Exception in outbound queue processing:
Aries | concurrent.futures._base.CancelledError: Task was cancelled

It looks like the TAA-related tests should be filtered with -t ~@taa_required.

@shaangill025
Copy link
Contributor

Actually I think there's a few more references to update: in docker-compose.yml, alice-local.sh, faber-local.sh, and runners/support/utils.py (check_requires method)

@ianco Based upon Andrew's comment, I opened the following PR [default-askar branch]:
https://github.com/ianco/aries-cloudagent-python/pull/2

Copy link
Contributor

@swcurran swcurran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like an integration test is still failing.

@ianco
Copy link
Contributor Author

ianco commented Jun 23, 2022

Looks like an integration test is still failing.

Hmm runs successfully on my local, I'll take a look ...

@ianco
Copy link
Contributor Author

ianco commented Jun 24, 2022

Adjusted tests, as one (@T004-RFC0453) was failing in GHA even though it was passing locally. (I think an issue with resources in GHA, I ran into this once before)

Also incldued 2 new tests in GHA that were failing locally with askar (but worked with indy):

BDD_EXTRA_AGENT_ARGS="{\"wallet-type\":\"indy\"}" ./run_bdd -t @T003-RFC0454.1,@T003-RFC0454.2

... works, but:

BDD_EXTRA_AGENT_ARGS="{\"wallet-type\":\"askar\"}" ./run_bdd -t @T003-RFC0454.1,@T003-RFC0454.2

fails.

@ianco
Copy link
Contributor Author

ianco commented Jun 24, 2022

Connect duration: 0.11s
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:06 +0000] "POST /webhooks/topic/out_of_band/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:06 +0000] "POST /webhooks/topic/connections/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
Aries      | Connected
Aries      | Check for endorser role ...
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:06 +0000] "POST /webhooks/topic/connections/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:06 +0000] "POST /webhooks/topic/connections/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
Aries      | Connected
Aries      | Check for endorser role ...
    When "Faber" sends a request for proof presentation DL_age_over_19_v2_with_health_id to "Bob"                                                      # features/steps/0454-present-proof.py:32

#20 Request proof of degree from alice

  >>> asking for proof for request: {'name': 'Proof of stuff', 'version': '0.1.0', 'requested_attributes': {'address_attrs': {'name': 'address', 'non_revoked': {'to': 1656068946}, 'restrictions': [{'schema_name': 'Schema_DriversLicense_v2', 'schema_version': '1.0.1'}]}, 'health_attrs': {'name': 'health_id_num', 'non_revoked': {'to': 1656068946}, 'restrictions': [{'schema_name': 'Schema_Health_ID', 'schema_version': '1.0.0'}]}}, 'requested_predicates': {'age': {'name': 'age', 'p_type': '>', 'p_value': 19, 'non_revoked': {'to': 1656068946}, 'restrictions': [{'schema_name': 'Schema_DriversLicense_v2', 'schema_version': '1.0.1'}]}}}
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:06 +0000] "POST /webhooks/topic/present_proof_v2_0/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
Aries      | Presentation: state = request-sent, pres_ex_id = 4adbf297-f71c-4ceb-9c34-f24a5b87dee8
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:06 +0000] "POST /webhooks/topic/connections/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
    Then "Faber" has the proof verification fail                                                                                                       # features/steps/0454-present-proof.py:62
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:06 +0000] "POST /webhooks/topic/connections/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:06 +0000] "POST /webhooks/topic/present_proof_v2_0/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
Aries      | Presentation: state = request-received, pres_ex_id = b7324b30-ddd5-4d1a-b1b3-bc797618650d

#24 Query for credentials in the wallet that satisfy the proof request

#25 Generate the proof

#26 Send the proof to X: {"indy": {"requested_predicates": {"age": {"cred_id": "bda1a1d2-0762-4630-9627-31893e4bba2f"}}, "requested_attributes": {"address_attrs": {"cred_id": "bda1a1d2-0762-4630-9627-31893e4bba2f", "revealed": true}, "health_attrs": {"cred_id": "0766bac0-a348-4a53-8d48-14870c8f1191", "revealed": true}}, "self_attested_attributes": {}}}
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:06 +0000] "POST /webhooks/topic/present_proof_v2_0/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
Aries      | Presentation: state = abandoned, pres_ex_id = b7324b30-ddd5-4d1a-b1b3-bc797618650d

Presentation exchange abandoned
Aries      | Problem report message: No revocation states provided for credential '0766bac0-a348-4a53-8d48-14870c8f1191'with rev_reg_id 'None'.
Aries      | 2022-06-24 11:09:06,956 aries_cloudagent.admin.server ERROR Handler error with exception: No revocation states provided for credential '0766bac0-a348-4a53-8d48-14870c8f1191'with rev_reg_id 'None'.
Aries      | Traceback (most recent call last):
Aries      |   File "/home/indy/aries_cloudagent/protocols/present_proof/v2_0/routes.py", line 1113, in present_proof_send_presentation
Aries      |     comment=comment,
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<AriesAgent.handle_present_proof_v2_0() done, defined at /home/indy/demo/runners/agent_container.py:387> exception=Exception("Error: 400: No revocation states provided for credential '0766bac0-a348-4a53-8d48-14870c8f1191'with rev_reg_id 'None'.",)>
Traceback (most recent call last):
  File "/home/indy/demo/runners/support/agent.py", line 848, in admin_request
    resp.raise_for_status()
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 1009, in raise_for_status
    headers=self.headers,
aiohttp.client_exceptions.ClientResponseError: 400, message="No revocation states provided for credential '0766bac0-a348-4a53-8d48-14870c8f1191'with rev_reg_id 'None'.", url=URL('http://host.docker.internal:8051/present-proof-2.0/records/b7324b30-ddd5-4d1a-b1b3-bc797618650d/send-presentation')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/indy/demo/runners/agent_container.py", line 515, in handle_present_proof_v2_0
    request,
  File "/home/indy/demo/runners/support/agent.py", line 949, in admin_POST
    "POST", path, data, text, params, headers=headers
  File "/home/indy/demo/runners/support/agent.py", line 851, in admin_request
    raise Exception(f"Error: {resp_text}") from e
Exception: Error: 400: No revocation states provided for credential '0766bac0-a348-4a53-8d48-14870c8f1191'with rev_reg_id 'None'.
Aries      |   File "/home/indy/aries_cloudagent/protocols/present_proof/v2_0/manager.py", line 268, in create_pres
Aries      |     request_data,
Aries      |   File "/home/indy/aries_cloudagent/protocols/present_proof/v2_0/formats/indy/handler.py", line 165, in create_pres
Aries      |     requested_credentials=requested_credentials,
Aries      |   File "/home/indy/aries_cloudagent/protocols/present_proof/indy/pres_exch_handler.py", line 213, in return_presentation
Aries      |     revocation_states,
Aries      |   File "/home/indy/aries_cloudagent/indy/credx/holder.py", line 522, in create_presentation
Aries      |     rev_state=get_rev_state(cred_id, timestamp) if timestamp else None,
Aries      |   File "/home/indy/aries_cloudagent/indy/credx/holder.py", line 485, in get_rev_state
Aries      |     f"No revocation states provided for credential '{cred_id}'"
Aries      | aries_cloudagent.indy.holder.IndyHolderError: No revocation states provided for credential '0766bac0-a348-4a53-8d48-14870c8f1191'with rev_reg_id 'None'
Aries      | 
Aries      | The above exception was the direct cause of the following exception:
Aries      | 
Aries      | Traceback (most recent call last):
INFO:aiohttp.access:172.17.0.1 [24/Jun/2022:11:09:07 +0000] "POST /webhooks/topic/present_proof_v2_0/ HTTP/1.1" 200 149 "-" "Python/3.6 aiohttp/3.8.1"
Aries      | Presentation: state = abandoned, pres_ex_id = 4adbf297-f71c-4ceb-9c34-f24a5b87dee8

Presentation exchange abandoned
Aries      | Problem report message: abandoned: abandoned
Aries      |   File "/home/indy/aries_cloudagent/admin/server.py", line 170, in ready_middleware
Aries      |     return await handler(request)
Aries      |   File "/home/indy/aries_cloudagent/admin/server.py", line 207, in debug_middleware
Aries      |     return await handler(request)
Aries      |   File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/aiohttp_apispec/middlewares.py", line 45, in validation_middleware
Aries      |     return await handler(request)
Aries      |   File "/home/indy/aries_cloudagent/admin/server.py", line 395, in setup_context
Aries      |     return await task
Aries      |   File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/asyncio/tasks.py", line 250, in _wakeup
Aries      |     future.result()
Aries      |   File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/asyncio/tasks.py", line 180, in _step
Aries      |     result = coro.send(None)
Aries      |   File "/home/indy/aries_cloudagent/protocols/present_proof/v2_0/routes.py", line 1132, in present_proof_send_presentation
Aries      |     outbound_handler,
Aries      |   File "/home/indy/aries_cloudagent/protocols/present_proof/v2_0/__init__.py", line 63, in report_problem
Aries      |     raise http_error_class(reason=err.roll_up) from err
Aries      | aiohttp.web_exceptions.HTTPBadRequest: No revocation states provided for credential '0766bac0-a348-4a53-8d48-14870c8f1191'with rev_reg_id 'None'.

@andrewwhitehead ?

@swcurran
Copy link
Contributor

This is the most troubling issue for me related to the 0.7.4 release. @andrewwhitehead - any progress on this? AFAIK -- the schema, creddef and revreg gets created, the credential issued, the proof request with revocation sent and then the Prover (Bob) fails the proof generation process, but only with Askar. At that level, how can we get differences between Indy and Askar? Is that an indication of some low-level problem?

Copy link
Contributor

@swcurran swcurran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Integration tests now working - ready to merge. Good stuff!~

@swcurran swcurran merged commit 1673245 into openwallet-foundation:main Jun 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants