feat: support connectionless exchange

[TimoGlastra]

Adds support for connectionnless exchange using ICv1, ICv2, PPv1 and PPv2 protocols for both roles (so issuer, holder and verifier).

Can be initiated by creating or receiving an oob invitation without handshake protocols. If a connection already exists for the did in the oob invitation service, the connection will be used instead (as specified in OOB RFC).

Fixes #1636

Some notes on the implementation:

• Added an OobRecord to keep track of the oob state. This is used for both connectionfull and connectionless exchanges and will be removed once the interaction is done
• The oob record will also emit events at the moment about reuse of connections etc... To not make breaking changes I kept the old OOB reuse webhook events in place. May be good to remove these in a future PR
• Connectionless is not integrated with mediation yet (when creating an invitation). If this is desired, maybe someone else can take a look at this
• I disabled multi use when attachments were present in the invitation as this would never have worked (there's no support for multi-use requests, offers AFAIK?)
• Adds a cleanup of stale connections if a connection is being reused by sending a direct response to an attachment message in the oob invitation (I think connections weren't being cleaned up in this case)

Keeping it in draft until I know for sure all CI tests have passed (can't run all tests on my M1 mac)

[swcurran]

@cvarjao --- FYI about this PR.

[codecov-commenter]

Codecov Report

Merging #1710 (1dd931b) into main (00d97b3) will decrease coverage by 1.09%.
The diff coverage is 95.31%.

@@            Coverage Diff             @@
##             main    #1710      +/-   ##
==========================================
- Coverage   95.21%   94.11%   -1.10%     
==========================================
  Files         530      533       +3     
  Lines       33235    33639     +404     
==========================================
+ Hits        31644    31660      +16     
- Misses       1591     1979     +388     

[TimoGlastra]

I just realized (when integrating with AATH) this PR does introduce a breaking change.... Would like some input on how to resolve.

In the current main branch of ACA-Py the /out-of-band/receive-invitation/ will return a connection record. This was possible because connectionless wasn't supported.

Now the endpoint returns an out of band record, which optionally contains a connection_id (if not using connectionless). So how should I go about handling this case? I see a few options:

1. Return connection record when using a connection, oob record when not using a connection. This is somewhat not breaking because connectionless is a new feature. (this is my least prefered approach as it makes working with the api endpoint more complex
2. Return the oob record and introduce a breaking change. Implementations relying on the currently returned connection record should take the connection_id from the oob record and resolve the connection.
3. ??

My preference would go to 2, but that does mean it's not fit for a 0.7.4 release. However 1. is technically also a breaking change.

[TimoGlastra]

Any updates on this? :)

[swcurran]

@TimoGlastra -- can you address the merge conflicts?

@shaangill025 and @ianco -- can the two of you take a look at this one?

Perhaps at ACA-Pug tomorrow, y'all can discuss if this is an 0.7.4 PR.

[ianco]

I just realized (when integrating with AATH) this PR does introduce a breaking change.... Would like some input on how to resolve.

Discussed at today's aca-pug. The consensus is that this is not a "breaking change" if it only affects connectionless.

I.e. if an implementation is not using connectionless, then there are no breaking changes introduced. If they are using connectionless, then there re changes required.

I'm happy with this solution:

Return the oob record and introduce a breaking change. Implementations relying on the currently returned connection record should take the connection_id from the oob record and resolve the connection.

[ianco]

@TimoGlastra looks like there's a conflict

[ianco]

@TimoGlastra if you can cleanup the conflicts I think we can get this reviewed/merged

[TimoGlastra]

Conflicts have been resolved! Sorry this one slipped my mind

[ianco]

Some AATH tests are failing, for example ./manage run -d acapy-main @T001-RFC0025

(acapy-main built off of this branch)

  @T001-RFC0025 @AcceptanceTest @DIDExchangeConnection @Transport_Http @Transport_Ws @normal
  Scenario Outline: Create DIDExchange connection between two agents with overlapping transports -- @5.2 DIDExchange connection with one agent using http for inbound and the other using ws and both agents supporting ws and http outbound  # features/0025-didcomm-transports.feature:47
    Given we have "2" agents                                                                                                                                                                                                                  # features/steps/0160-connection.py:22 0.000s
      | name | role      |
      | Acme | responder |
      | Bob  | requester |
    And "Acme" is running with parameters "{"inbound_transports": ["ws"], "outbound_transports": ["ws", "http"] }"                                                                                                                            # features/steps/0044-mime-types.py:12 2.557s
    And "Bob" is running with parameters "{"inbound_transports": ["http"], "outbound_transports": ["ws", "http"] }"                                                                                                                           # features/steps/0044-mime-types.py:12 3.070s
    When "Acme" and "Bob" create a new connection                                                                                                                                                                                             # features/steps/0160-connection.py:368 0.175s
      Assertion Failed: FAILED SUB-STEP: When "Acme" and "Bob" create a new didexchange connection
      Substep info: Assertion Failed: FAILED SUB-STEP: And "Bob" receives the invitation
      Substep info: Assertion Failed: resp_status 500 is not 200; 'done'
      Traceback (of failed substep):
        File "/usr/local/lib/python3.7/site-packages/behave/model.py", line 1329, in run
          match.run(runner.context)
        File "/usr/local/lib/python3.7/site-packages/behave/matchers.py", line 98, in run
          self.func(context, *args, **kwargs)
        File "features/steps/0023-did-exchange.py", line 226, in step_impl
          assert resp_status == 200, f"resp_status {resp_status} is not 200; {resp_text}"
      
      Traceback (of failed substep):
        File "/usr/local/lib/python3.7/site-packages/behave/model.py", line 1329, in run
          match.run(runner.context)
        File "/usr/local/lib/python3.7/site-packages/behave/matchers.py", line 98, in run
          self.func(context, *args, **kwargs)
        File "features/steps/0023-did-exchange.py", line 37, in step_impl
          """
        File "/usr/local/lib/python3.7/site-packages/behave/runner.py", line 404, in execute_steps
          assert False, message

There are no errors or stack traces showing up in the agent logs though ...

[TimoGlastra]

@ianco did you try with the draft PR in AATH? openwallet-foundation/owl-agent-test-harness#474

I had to change some things in AATH to make it work, but it should work fine on that branch. If you have, then I'll take another look and see if some things broke in the meantime

[ianco]

did you try with the draft PR in AATH? hyperledger/aries-agent-test-harness#474

I'll try this today @TimoGlastra thanks

[ianco]

I'm still getting one AATH failure with ./manage run -d acapy-main -t @T001-RFC0025

(including the draft PR mentioned above by @TimoGlastra )

  @T001-RFC0025 @AcceptanceTest @DIDExchangeConnection @Transport_Ws @Transport_NoHttpOutbound @normal
  Scenario Outline: Create DIDExchange connection between two agents with overlapping transports -- @2.1 DIDExchange connection with both agents using WS for inbound and outbound transport  # features/0025-didcomm-transports.feature:28
    Given we have "2" agents                                                                                                                                                                  # features/steps/0160-connection.py:22 0.001s
      | name | role      |
      | Acme | responder |
      | Bob  | requester |
    And "Acme" is running with parameters "{"inbound_transports": ["ws"], "outbound_transports": ["ws"] }"                                                                                    # features/steps/0044-mime-types.py:12 2.049s
    And "Bob" is running with parameters "{"inbound_transports": ["ws"], "outbound_transports": ["ws"] }"                                                                                     # features/steps/0044-mime-types.py:12 3.032s
    When "Acme" and "Bob" create a new connection                                                                                                                                             # features/steps/0160-connection.py:360 10.740s
      Assertion Failed: FAILED SUB-STEP: When "Acme" and "Bob" create a new didexchange connection
      Substep info: Assertion Failed: FAILED SUB-STEP: And "Acme" receives the request
      Substep info: Assertion Failed: Could not retreive responders connection_id
      Traceback (of failed substep):
        File "/usr/local/lib/python3.7/site-packages/behave/model.py", line 1329, in run
          match.run(runner.context)
        File "/usr/local/lib/python3.7/site-packages/behave/matchers.py", line 98, in run
          self.func(context, *args, **kwargs)
        File "features/steps/0023-did-exchange.py", line 313, in step_impl
          assert False, f"Could not retreive responders connection_id"
      
      Traceback (of failed substep):
        File "/usr/local/lib/python3.7/site-packages/behave/model.py", line 1329, in run
          match.run(runner.context)
        File "/usr/local/lib/python3.7/site-packages/behave/matchers.py", line 98, in run
          self.func(context, *args, **kwargs)
        File "features/steps/0023-did-exchange.py", line 37, in step_impl
          """
        File "/usr/local/lib/python3.7/site-packages/behave/runner.py", line 404, in execute_steps
          assert False, message

    Then the invitation serviceEndpoint should use the "["ws"]" protocol scheme                                                                                                               # None

[ianco]

Overall looks good, except the AATH issue I mentioned. Do we need to synchronize merging the AATH PR before we can merge this one?

[ianco]

I'm still getting one AATH failure with ./manage run -d acapy-main -t @T001-RFC0025

(including the draft PR mentioned above by @TimoGlastra )

... actually I get this failure even running against the main branches, do prob not related to this PR

[shaangill025]

Possible typo, shouldn't it be something like Inbound message is not allowed for thread

[ianco]

Depends on AATH PR: openwallet-foundation/owl-agent-test-harness#474

[swcurran]

Remember to close #1636 when this is merged.