store access-api delegation bytes in R2 not D1

[gobengo]

Motivation:

• the bytes column on delegations table should go away because
  • cloudflare D1 has a relatively small (~100Mb) db storage size
  • we shouldn't put unbounded amount of bytes anywhere in the d1 database
  • delegation bytes can be arbitrarily large
  • so in order to lower risk we'll fill up the D1 data store, we shouldn't put delegation bytes in the d1 database
• related to: make plan for not hitting 100MB cloudflare D1 limit #475

Solutions Considered

• move delegation bytes out of D1 and into a non-d1 cloudflare data store
  • pros
    • cheap to try to remove the biggest data from d1 - bytes
    • changes max amount of delegations stored from O(100MB/(avgByteLength * n)) to O(100Mb/n)
  • cons
    • reading delegations will be slower. Before the read just had to query D1, but now it will query D1 for delegation CIDs, and then read the bytes for those CIDs from R2
    • migrating this data will be more complicated. When it's in d1, we can use d1 migrations and sql to manage data migrations like e.g. adding a delegations expiry index. If the underlying delegation bytes are not in D1, we will need to make our own data migration system that spans the data stores we do use.
      • in practice today, the d1 migrations tools are alpha, but we will no longer benefit from those tools if/when cloudflare improves them
    • non-bytes data will still be in D1 and subject to size limitation
• avoid using d1 entirely
  • this may make sense in the long run, e.g. using a datastore with a much bigger size limit, but I don't think we can do this quickly today without looking beyond cloudflare's data storage products (e.g. to AWS like DyanmoDB or Aurora)
  • pros
    • avoid D1 and its limits entirely
  • cons
    • would involve modifying models/DBs/functionality in access-api other than just delegations
    • complexity of using cloudflare workers with non-cloudflare data store
      • e.g. what would localdev use?
• store delegation bytes using web3.storage store/add protocol
  • insufficient because the delegations may be non-public, but all things stored in web3.storage are retrievable by CID alone

data store needs for delegation bytes

• stored delegations are non-public and we only serve them to authorized invocations e.g. access/claim invocations
• want read-after-write strong consistency
  • e.g. for access/delegate + immediate access/claim
• keys will be CIDs
• value never needs to be mutated once written
• deletes of delegation CIDs can be processed eventually

should we store bytes in cloudflare KV or R2?

tl;dr - looks like R2 provides what's needed at the lowest cost

• KV
  • price
    • $0.50/GB*mo storage
    • per request depends
  • unique value proposition
    • low latency + caching - I don't think we need this here (esp at 3x the per-GB cost)
  • is good for https://developers.cloudflare.com/workers/platform/storage-objects/#kv
    • high volumes of reads or repeated reads - delegations bytes aren't really this
    • per-object TTL - we don't need this
• R2

  • is good for https://developers.cloudflare.com/workers/platform/storage-objects/#r2

  • prices

    • $0.015/GB*mo storage
    • per request depends

  • unique benefits

    • free/cheap egress

  • limits

    • object size 5TB - delegations should be < 1MB on average. Workers request limit is 100MB.
    • object metadata size 2MB

[gobengo]

(via @Gozala)
Long term: consider whether r2 key format should be similar to elastic ipfs s3 key format
{car.cid}/{root.cid}.{fileExtension}