feat: upgrade storefront api with filecoin offer from bucket event instead of directly compute #343

vasco-santos · 2024-03-18T12:45:19Z

This PR intends to upgrade storefront package. This enables filecoin/offer invocations lead to messages to the submit queue, where message consumer will compute Piece for the content. The main goal of this is to have the client to compute Filecoin Pieces and offer them, instead of pieces being directly computed from bucket event and written into the DB (skipping the submit validation).

Note that:

given we will need to deal with client upgrades and old clients still running for the time being, I opted in this PR to still keep a version of the bucket event in place. However, instead of it running computation of piece, it also triggers filecoin/offer so that we can keep old clients running where filecoin/offer is not invoked. This of course means that we may need to run Piece computation twice for the old path. However, looking at our AWS bill, we paid 200 USD in February for lambda execution of whole w3infra. This assumes then that we will not incur a lot of additional cost while making migration much simpler
datastore currently implements a composed store with S3+R2, which allows us to move on an iterative approach where we can deploy first step without client upgrade, while we can iterate on writes to anywhere with the overall picture.

Needs:

feat: api waits for trigger filecoin pipeline from the client w3up#1332

vasco-santos · 2024-03-26T13:03:02Z

filecoin/test/helpers/resources.js

@@ -140,7 +140,7 @@ export const createSQS = async (opts = {}) => {
  const port = opts.port || 9324

  const queue = await pRetry(() =>
-    new Container('softwaremill/elasticmq')
+    new Container('softwaremill/elasticmq:1.5.4')


looks like latest queue docker image is bad 😱

This was failing with a random error, fixed it for now to previous version

…stead of compute

seed-deploy · 2024-03-26T18:01:19Z

View stack outputs

pr343-w3infra-BillingDbStack

Name Value

customerTableName pr343-w3infra-customer

spaceDiffTableName pr343-w3infra-space-diff

spaceSnapshotTableName pr343-w3infra-space-snapshot

usageTable pr343-w3infra-usage

pr343-w3infra-BillingStack

Name	Value
ApiEndpoint	https://re7u89upp4.execute-api.us-east-2.amazonaws.com
billingCronHandlerURL	https://nyv75cgjfp5zajm53xokuju4ze0mwbgs.lambda-url.us-east-2.on.aws/
CustomDomain	https://pr343.billing.web3.storage

pr343-w3infra-CarparkStack

Name Value

BucketName carpark-pr343-0

Region us-east-2
pr343-w3infra-RoundaboutStack

Name Value

ApiEndpoint https://ndcs221yjl.execute-api.us-east-2.amazonaws.com

CustomDomain https://pr343.roundabout.web3.storage
pr343-w3infra-SatnavStack

Name Value

BucketName satnav-pr343-0

Region us-east-2
pr343-w3infra-UploadApiStack

Name Value

ApiEndpoint https://9us0jsb30k.execute-api.us-east-2.amazonaws.com

CustomDomain https://pr343.up.web3.storage
pr343-w3infra-BusStack
pr343-w3infra-FilecoinStack
pr343-w3infra-ReplicatorStack
pr343-w3infra-UcanFirehoseStack
pr343-w3infra-UcanInvocationStack
pr343-w3infra-UploadDbStack

vasco-santos · 2024-03-26T18:11:58Z

filecoin/functions/handle-piece-insert-to-content-claim.js


+  const { ok, error } = await storefrontEvents.handlePieceInsertToEquivalencyClaim(context, record)


we now use the API 🎉

vasco-santos · 2024-03-26T18:12:37Z

filecoin/functions/handle-filecoin-submit-message.js

  const context = {
-    pieceStore: createPieceTable(AWS_REGION, pieceTableName)
+    pieceStore: createPieceTable(AWS_REGION, pieceTableName),
+    dataStore: composeDataStoresWithOrderedStream(


we want to read from S3 first, fallback to R2 if not there (prep for new world)

Nit: this would read more naturally if it was something like

cons s3 = createDataStore(AWS_REGION, s3BucketName) const r2 = createDataStore(R2_REGION, r2BucketName, { endpoint: r2BucketEndpoint, credentials: { accessKeyId: r2BucketAccessKeyId, secretAccessKey: r2BucketSecretAccessKey, }, }) const store = S3.or(r2)

good call, we actually already use this pattern on upload-api https://github.com/web3-storage/w3infra/blob/main/upload-api/functions/ucan-invocation-router.js#L185C52-L186C1 and that is why I kept. But probably we could get more declarative on a follow up

vasco-santos · 2024-03-26T18:14:10Z

filecoin/store/data.js

+    // Only used for testing storing a CAR
+    // until we hook up claims to look for data
+    put: async (bytes) => {
+      const hash = await sha256.digest(bytes)
+      const root = CID.create(1, raw.code, hash)
+
+      const { writer, out } = CarWriter.create(root)
+      writer.put({ cid: root, bytes })
+      writer.close()
+
+      const chunks = []
+      for await (const chunk of out) {
+        chunks.push(chunk)
+      }
+      const blob = new Blob(chunks)
+      const cid = await CAR.codec.link(new Uint8Array(await blob.arrayBuffer()))
+
+      const putCmd = new PutObjectCommand({
+        Bucket: bucketName,
+        Key: `${cid.toString()}/${cid.toString()}.car`,
+        Body: bytes


we only need this for the tests, so that we can put something on the store. Maybe I should change the API to have an augmented store just on the tests interface, but not the actual service store. WDYT?

vasco-santos · 2024-03-26T18:14:49Z

filecoin/test/helpers/resources.js

@@ -140,7 +140,7 @@ export const createSQS = async (opts = {}) => {
  const port = opts.port || 9324

  const queue = await pRetry(() =>
-    new Container('softwaremill/elasticmq')
+    new Container('softwaremill/elasticmq:1.5.4')


This was failing with a random error, fixed it for now to previous version

vasco-santos · 2024-03-26T18:15:27Z

package.json

@@ -13,7 +13,7 @@
    "lint": "tsc && eslint '**/*.js'",
    "clean": "rm -rf dist node_modules package-lock.json ./*/{.cache,dist,node_modules}",
    "test": "npm test -w billing -w upload-api -w carpark -w replicator -w satnav -w roundabout -w filecoin",
-    "test-integration": "ava --verbose --serial --timeout=600s test/*.test.js",
+    "test-integration": "ava --verbose --serial --timeout=660s test/*.test.js",


not sure if needed, but now tests look a bit slower with extra queue

Gozala · 2024-04-02T07:26:55Z

filecoin/store/data.js

+    if (result === undefined) {
+      throw new Error('no result received')
+    }
+    return result


Nit: result should probably aggregate errors as opposed to just returning the last error.

true, given this is a temporary thing I am tempted to keep it, given otherwise would be difficult to align the types

Gozala · 2024-04-02T07:30:08Z

filecoin/functions/handle-filecoin-submit-message.js

  const context = {
-    pieceStore: createPieceTable(AWS_REGION, pieceTableName)
+    pieceStore: createPieceTable(AWS_REGION, pieceTableName),
+    dataStore: composeDataStoresWithOrderedStream(


Nit: this would read more naturally if it was something like

cons s3 = createDataStore(AWS_REGION, s3BucketName) const r2 = createDataStore(R2_REGION, r2BucketName, { endpoint: r2BucketEndpoint, credentials: { accessKeyId: r2BucketAccessKeyId, secretAccessKey: r2BucketSecretAccessKey, }, }) const store = S3.or(r2)

This is not needed since #1332 got released and used in w3infra storacha/w3infra#343 and has been deployed in production without issues for the last few weeks BREAKING CHANGE: not possible to skip submit queue on storefront service anymore

seed-deploy bot had a problem deploying to pr343 March 18, 2024 12:45 Failure

vasco-santos marked this pull request as draft March 18, 2024 12:45

vasco-santos mentioned this pull request Mar 18, 2024

[parent] filecoin/offer flow storacha/project-tracking#18

Closed

4 tasks

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from bbda033 to 1f3b1f3 Compare March 26, 2024 12:07

seed-deploy bot had a problem deploying to pr343 March 26, 2024 12:07 Failure

vasco-santos marked this pull request as ready for review March 26, 2024 12:08

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch 2 times, most recently from a18f008 to 31a834f Compare March 26, 2024 12:19

seed-deploy bot had a problem deploying to pr343 March 26, 2024 12:19 Failure

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from 31a834f to e9c6d1c Compare March 26, 2024 12:44

seed-deploy bot had a problem deploying to pr343 March 26, 2024 12:44 Failure

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from e9c6d1c to f85dccb Compare March 26, 2024 13:02

seed-deploy bot had a problem deploying to pr343 March 26, 2024 13:02 Failure

vasco-santos commented Mar 26, 2024

View reviewed changes

vasco-santos requested a review from alanshaw March 26, 2024 13:19

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from f85dccb to 0ab61f5 Compare March 26, 2024 13:53

seed-deploy bot had a problem deploying to pr343 March 26, 2024 13:53 Failure

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from 0ab61f5 to 4b7fe90 Compare March 26, 2024 14:54

seed-deploy bot had a problem deploying to pr343 March 26, 2024 14:54 Failure

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from 4b7fe90 to 6343e78 Compare March 26, 2024 15:14

seed-deploy bot had a problem deploying to pr343 March 26, 2024 15:14 Failure

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from 6343e78 to 01724a5 Compare March 26, 2024 15:28

seed-deploy bot had a problem deploying to pr343 March 26, 2024 15:28 Failure

seed-deploy bot had a problem deploying to pr343 March 26, 2024 15:53 Failure

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from 01724a5 to 45776db Compare March 26, 2024 16:22

seed-deploy bot had a problem deploying to pr343 March 26, 2024 16:22 Failure

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from 45776db to e40cd0f Compare March 26, 2024 16:49

seed-deploy bot had a problem deploying to pr343 March 26, 2024 16:49 Failure

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from e40cd0f to e33fa3e Compare March 26, 2024 17:22

seed-deploy bot had a problem deploying to pr343 March 26, 2024 17:22 Failure

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from e33fa3e to 0906e7a Compare March 26, 2024 17:37

seed-deploy bot had a problem deploying to pr343 March 26, 2024 17:37 Failure

feat: upgrade storefront api with filecoin offer from bucket event in…

cb735d2

…stead of compute

vasco-santos force-pushed the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch from 0906e7a to cb735d2 Compare March 26, 2024 17:49

seed-deploy bot temporarily deployed to pr343 March 26, 2024 17:49 Inactive

vasco-santos commented Mar 26, 2024

View reviewed changes

vasco-santos requested a review from Gozala March 28, 2024 17:59

vasco-santos mentioned this pull request Mar 28, 2024

[parent] write to destinations that don't support bucket events storacha/project-tracking#6

Closed

10 tasks

Gozala approved these changes Apr 2, 2024

View reviewed changes

vasco-santos merged commit 3fd05a8 into main Apr 2, 2024
3 checks passed

vasco-santos deleted the feat/upgrade-storefront-api-with-filecoin-offer-from-bucket-event-instead-of-compute branch April 2, 2024 11:01

This was referenced Apr 15, 2024

fix!: drop filecoin storefront skip submit queue option storacha/w3up#1371

Merged

infra for filecoin offer storacha/project-tracking#52

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: upgrade storefront api with filecoin offer from bucket event instead of directly compute #343

feat: upgrade storefront api with filecoin offer from bucket event instead of directly compute #343

vasco-santos commented Mar 18, 2024 •

edited

Loading

vasco-santos Mar 26, 2024

vasco-santos Mar 26, 2024

seed-deploy bot commented Mar 26, 2024

vasco-santos Mar 26, 2024

vasco-santos Mar 26, 2024

Gozala Apr 2, 2024

vasco-santos Apr 2, 2024

vasco-santos Mar 26, 2024

vasco-santos Mar 26, 2024

vasco-santos Mar 26, 2024

Gozala Apr 2, 2024

vasco-santos Apr 2, 2024

Gozala Apr 2, 2024


		const { ok, error } = await storefrontEvents.handlePieceInsertToEquivalencyClaim(context, record)

feat: upgrade storefront api with filecoin offer from bucket event instead of directly compute #343

feat: upgrade storefront api with filecoin offer from bucket event instead of directly compute #343

Conversation

vasco-santos commented Mar 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seed-deploy bot commented Mar 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos commented Mar 18, 2024 •

edited

Loading