Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: ucan endpoint supports agent message #187

Merged
merged 5 commits into from
Apr 12, 2023

Conversation

vasco-santos
Copy link
Contributor

@vasco-santos vasco-santos commented Apr 11, 2023

As agreed, first iteration adds support for POST /ucan into the API. upload-api is still using old world, therefore we need to keep both ucanto dependencies for this first iteration. Code changes aim to just drop most of the functions in place in favour of the added ones once in next PR we update ucanto in upload-api.

Decisions:

  • new bucket structures based on Update ucanlog to [email protected] #186 (comment)
  • we install "@ucanto/core-next": "npm:@ucanto/core@^7.0.1" + "@ucanto/transport-legacy": "npm:@ucanto/transport@^5.1.1"
    • initial goal here was to install new ones and keep olds with suffix -legacy to easily drop them in next PR. Sadly, given other ucanto packages depend on the 5th line of core, I could not work around having "@ucanto/core-legacy": "npm:@ucanto/core@^5.1.0" because dependencies would struggle to find correct one...
    • path forward is still relatively easy given ucanto/core is barely used
  • tried to avoid naming workflow everywhere, moving to agentMessage to uniform with ucanto

Closes #186

@seed-deploy seed-deploy bot temporarily deployed to pr187 April 11, 2023 14:21 Inactive
@vasco-santos vasco-santos force-pushed the feat/ucan-endpoint-supports-agent-message branch from 4210567 to d415b12 Compare April 11, 2023 14:32
@seed-deploy
Copy link

seed-deploy bot commented Apr 11, 2023

View stack outputs
  • pr187-w3infra-CarparkStack

    Name Value
    BucketName carpark-pr187-0
    Region us-east-2
  • pr187-w3infra-SatnavStack

    Name Value
    BucketName satnav-pr187-0
    Region us-east-2
  • pr187-w3infra-UploadApiStack

    Name Value
    ApiEndpoint https://2ifz5e23b4.execute-api.us-east-2.amazonaws.com
    CustomDomain https://pr187.up.web3.storage
  • pr187-w3infra-BusStack

  • pr187-w3infra-ReplicatorStack

  • pr187-w3infra-UcanInvocationStack

  • pr187-w3infra-UploadDbStack

@seed-deploy seed-deploy bot temporarily deployed to pr187 April 11, 2023 14:34 Inactive
@vasco-santos vasco-santos force-pushed the feat/ucan-endpoint-supports-agent-message branch from d415b12 to 0f11106 Compare April 11, 2023 14:46
@seed-deploy seed-deploy bot temporarily deployed to pr187 April 11, 2023 14:46 Inactive
@vasco-santos vasco-santos marked this pull request as ready for review April 11, 2023 14:57
@vasco-santos vasco-santos requested review from Gozala and gobengo April 11, 2023 14:58
@vasco-santos vasco-santos added this to the w3up phase 4 milestone Apr 11, 2023
@vasco-santos vasco-santos self-assigned this Apr 11, 2023
Copy link
Contributor

@Gozala Gozala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thanks for getting it all done so quickly! It all looks good although I made some minor suggestions here and there.

I do however find naming misleading however, because through out agentMessage is used to refer to the CAR file containing an AgentMessage block in a root. I would suggest update naming to call it agentArchive or agentMessageArchive instead to disambiguate CAR containing the message from the message block itself.

docs/ucan-invocation-stream.md Outdated Show resolved Hide resolved
docs/ucan-invocation-stream.md Outdated Show resolved Hide resolved
@@ -29,11 +30,11 @@ Note that at the time of writing Event Archival flow is still to be implemented.

UCAN Invocation Stack contains 3 buckets so that it can keep an audit of the entire system, while allowing this information to be queried in multiple fashions.

Firstly, the **`workflow-store` bucket** stores the entire encoded file containing one or more invocations to be executed. It is stored as received from UCAN services interacting with UCAN Invocation Stream. It is keyed as `${workflow.cid}/${workflow.cid}` and its value is likely in CAR format. However, CID codec should tell if it is something else.
Firstly, the **`workflow-store` bucket** stores the entire encoded agent message files containing invocations to be executed, and/or created receipts for ran invocations. It is stored as received from UCAN services interacting with UCAN Invocation Stream. It is keyed as `${agentMessage.cid}/${agentMessage.cid}` and its value is likely in CAR format. However, CID codec should tell if it is something else.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is misleading, because AgentMessage is a root block inside the incoming / outgoing CAR and therefor has it's own CID, yet I think here we mean CAR CID not the block cid.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah you are right. Wanted to be generic to enable non CAR encodings in the future. Should we keep the workflow.cid naming then?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per suggestion in review text, I changed this into agentMessageArchive.cid. Let me know if it is more clear this way

upload-api/buckets/invocation-store.js Show resolved Hide resolved
upload-api/test/helpers/ucan.js Outdated Show resolved Hide resolved
upload-api/ucan-invocation.js Show resolved Hide resolved
const agentMessageWithInvocationCid = await ctx.invocationBucket.getInLink(invocationCid)

if (!agentMessageWithInvocationCid) {
throw new NoCarFoundForGivenReceiptError()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we want to error here and more importantly:

  1. Who is supposed to handle this error ?
  2. What can they do when it occurs ?

It seems like we could run into this due to concurrency when handler returns quickly, because receipt may make it sooner then invocation. Overall seems to me that it would be better to not crash if we do not have an invocation, but rather store receipt and report such instances so someone can take a look and figure out what's up.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, if we do not have the invocation we won't be able to get the needed information into the UCAN LOG. We could have a way around putting this to a queue and do it async (even though it might be the case that we never receive this and only a receipt for some reason).

Not sure I follow the concurrency issue though. We currently send the invocation before executing it (and block on caller) until POST /ucan succeeds. Once POST succeeds, invocation is stored and this code path will work without errors for receipts.

The error case is mostly for when we receive receipts for unknown invocations. When we implement invocation spec, I assume everything will be equal? We store invocation before executing it.

With the above said, we return error for caller because they are sending us a receipt for an invocation that was not first reported. What they can do is sending invocation that is going to be executed first

upload-api/ucan-invocation.js Outdated Show resolved Hide resolved
upload-api/ucan-invocation.js Outdated Show resolved Hide resolved
upload-api/ucan-invocation.js Outdated Show resolved Hide resolved
@seed-deploy seed-deploy bot temporarily deployed to pr187 April 12, 2023 09:37 Inactive
@seed-deploy seed-deploy bot temporarily deployed to pr187 April 12, 2023 13:40 Inactive
@seed-deploy seed-deploy bot temporarily deployed to pr187 April 12, 2023 13:49 Inactive
@vasco-santos
Copy link
Contributor Author

@Gozala addressed all your review, looks like we might need to still align on 1 point:

Given PR is approved, I will move forward and merge this in, so that we can start working on access-api and upload-api upgrade. Happy to do any needed change in upcoming PR based on your answer there

@vasco-santos vasco-santos merged commit 8532785 into main Apr 12, 2023
@vasco-santos vasco-santos deleted the feat/ucan-endpoint-supports-agent-message branch April 12, 2023 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update ucanlog to [email protected]
2 participants