fully migrate off D1 #711
Comments
travis
added a commit
to storacha/w3infra
that referenced
this issue
Apr 20, 2023
This is a first step toward storacha/w3up#711 as documented in https://github.com/web3-storage/w3up/pull/744/files?short_path=4e012d9#diff-4e012d97ece45b455806fabbe1f93ce11af56213b9137463a427e1c5d4d8bda6 This is an implementation spike of storing delegations in DynamoDB rather than D1. Because we already use DynamoDB in the upload-api, it made sense to reuse the configuration and testing infrastructure we've already built there. I copied a number of interfaces from https://github.com/web3-storage/w3up/tree/main/packages/access-api to get this done, and will migrate back to them if that ends up making sense. As this implies, the path forward for this is somewhat unclear and I need feedback and consensus from the team to keep this moving. We have a couple options to move forward: 1) Port the code in this PR over to the `access-api` repository and run it inside a cloudflare worker. This will require us to port some of the configuration and testing utilities in this repository to that repository and will leave us with a situation where compute in Cloudflare is reading and writing to DynamoDB in AWS. It will also require us to change the S3 logic in this PR to operate against R2, but that may Just Work thanks to theoretical API compatibility. This will also result in AWS-related logic being split between this repository and w3up, and we will probably want to extract common utilities to a separate package at some point in the future. 2) Port the delegations-related service endpoints in `w3up/access-api` over to this repository and get them running in an AWS Lambda. In this scenario we could either use S3 as implemented in this PR or continue to store delegations in R2. This option would move us in the direction of abstracting all operational details out of the w3up repository, following the pattern already used by w3up's `upload-api` package, which is imported by `upload-api` in this package. In this option I think it probably makes sense to migrate existing delegations from R2 to S3, but could be wrong on that front. My conversations with @Bengo suggested that the team may be mostly aligned behind option (2), especially because it seems like it moves us in the direction of isolating operational details of our services to this repository and turning the w3ui `*-api` packages into abstract implementations of the service providers we implement. Thoughts?
travis
added a commit
to storacha/w3infra
that referenced
this issue
May 16, 2023
This is a first step toward storacha/w3up#711 as documented in https://github.com/web3-storage/w3up/pull/744/files?short_path=4e012d9#diff-4e012d97ece45b455806fabbe1f93ce11af56213b9137463a427e1c5d4d8bda6 This is an implementation spike of storing delegations in DynamoDB rather than D1. Because we already use DynamoDB in the upload-api, it made sense to reuse the configuration and testing infrastructure we've already built there. I copied a number of interfaces from https://github.com/web3-storage/w3up/tree/main/packages/access-api to get this done, and will migrate back to them if that ends up making sense. As this implies, the path forward for this is somewhat unclear and I need feedback and consensus from the team to keep this moving. We have a couple options to move forward: 1) Port the code in this PR over to the `access-api` repository and run it inside a cloudflare worker. This will require us to port some of the configuration and testing utilities in this repository to that repository and will leave us with a situation where compute in Cloudflare is reading and writing to DynamoDB in AWS. It will also require us to change the S3 logic in this PR to operate against R2, but that may Just Work thanks to theoretical API compatibility. This will also result in AWS-related logic being split between this repository and w3up, and we will probably want to extract common utilities to a separate package at some point in the future. 2) Port the delegations-related service endpoints in `w3up/access-api` over to this repository and get them running in an AWS Lambda. In this scenario we could either use S3 as implemented in this PR or continue to store delegations in R2. This option would move us in the direction of abstracting all operational details out of the w3up repository, following the pattern already used by w3up's `upload-api` package, which is imported by `upload-api` in this package. In this option I think it probably makes sense to migrate existing delegations from R2 to S3, but could be wrong on that front. My conversations with @Bengo suggested that the team may be mostly aligned behind option (2), especially because it seems like it moves us in the direction of isolating operational details of our services to this repository and turning the w3ui `*-api` packages into abstract implementations of the service providers we implement. Thoughts?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Work:
The text was updated successfully, but these errors were encountered: