Public testnet feature completeness

[sanderpick]

We can begin an audit when these features are complete.

Credits

• Buy credit with HOKU
  • Pre-buy
  • Buy “inline” with a transaction
• Approve credit for others
  • With and without TTL
  • With and without limit
• Revoke credit for others
• Use credit to pay for transaction gas
• All available from SDK and solidity contracts

Blobs

• Add blobs from other actors, e.g., a bucket
• Add blobs paid for with credit
  • With and without TTL
  • Underlying blob includes recovery metadata from entanglement
• Add same blob repeatedly from with same subscriber and different actors and only pay for storage once
• Delete blobs from other actors, e.g., a bucket
• Get blob info
• Read blob bytes by hash from another actor
  • Full
  • Partial bytes
• All available from solidity contracts (note, adding raw blobs from SDK is not available by design, but should be from contracts so that people can build their own bucket-like / other contracts)

Buckets

• Create buckets
• List your buckets
• Store objects paid for with credit
  • Under a key
  • With and without TTL
  • With metadata
    • With content-type
  • Objects are encrypted with client encryption key
• Download objects
  • From SDK
  • Using the Object API, which can render content-type
  • Using byte ranges
• Query objects in a bucket
• Delete objects in a bucket
• All available from SDK and solidity contracts
• All available over S3 API
• Read object bytes by key from another actor
  • Full
  • Partial bytes

Timehubs

• Create/list timehubs
• Crud ops (push, get, count, peaks, root)
• Store timestamp alongside stored value
• Restrict API to accepting CIDs only
• SDK/CLI changes
• Access control

Validation

• Stake HOKU
• Pledge storage capacity
• Earn rewards from staking
• Earn rewards from storing data
• Unstake
• Unpledge storage capacity
• Only store portion of blob data required by disperser

Misc.

• Spin local network with N nodes without Docker from hoku CLI
• Chain explorer based on Blockscout
  • Deployed, managed, metrics, etc
• Contract based token faucet
• Registrar service on top of token faucet
• Faucet UI with GH login
• Pick and set Hoku top-level subnet ChainID

[sanderpick]

@carsonfarmer some questions for you that are probably answered already, just trying to consolidate here:

Re: "Unpledge storage capacity". I remember us having a long convo about this but forgot what we decided. When a validator just up and leaves, the bits it was storing just naturally backfill into the other validators when that data is requested, right? Anything else to think about here around "unpledging" storage capacity, ie., time locks, slashing, etc.?

Re: "Only store portion of blob data required by disperser", does mid Nov. sounds reasonable for this?

[sanderpick]

@brunocalza I remember you saying you did the encryption POC on top of the S3 api for speed... however, I think the final product needs to be over Buckets generically. Does that still make sense to you?

[sanderpick]

@dtbuchholz remind me, in the Credits wrapper you did, you did not include Blob related methods, right? I think it would be nice to have those as well so that devs can leverage raw blobs in a solidity contract to create their own bucket-like contracts, ie, list of blobs. Does that sound doable?

[brunocalza]

@brunocalza I remember you saying you did the encryption POC on top of the S3 api for speed... however, I think the final product needs to be over Buckets generically. Does that still make sense to you?

yeah, it makes sense. do you think most of that logic should reside in the sdk? if so, that should not take long

[carsonfarmer]

Re: "Only store portion of blob data required by disperser", does mid Nov. sounds reasonable for this?

I think so? In this case, we're saying each peer should only have to store the portion of the data they are responsible for IIRC. So peers can drop any data for which they are not responsible. Once way to easily handle this, would be that peers only "pin" the data they are required to store, and the rest can be garbage cleaned?

[carsonfarmer]

Re: "Unpledge storage capacity". I remember us having a long convo about this but forgot what we decided. When a validator just up and leaves, the bits it was storing just naturally backfill into the other validators when that data is requested, right? Anything else to think about here around "unpledging" storage capacity, ie., time locks, slashing, etc.?

Yes I think this is actually a pretty big one. We might want to a) push this off somewhat, and b) have a separate project to track it.

Basically, if a peer wants to unpledge, there is going to have to be an unbonding period, and if we can't adjust the subnet due to over-capacity, there will have to be slashing (which we don't have yet). There are well-defined ways to deciding who should take over responsibility for their blobs using the rendezvous hashing "clusters". Actually, nothing needs to happen really, except that we'll want to "move" the data to other peers to ensure the level of redundancy we are targeting, but subsequent queries based on rendezvous hashing will just point to the (new) correct peer.

[sanderpick]

yeah, it makes sense. do you think most of that logic should reside in the sdk? if so, that should not take long

yep, sounds right to me 👍 i created an issue

[sanderpick]

Re: "Only store portion of blob data required by disperser", does mid Nov. sounds reasonable for this?

I think so? In this case, we're saying each peer should only have to store the portion of the data they are responsible for IIRC. So peers can drop any data for which they are not responsible. Once way to easily handle this, would be that peers only "pin" the data they are required to store, and the rest can be garbage cleaned?

This is only three weeks away though. I don't think we've even started the disperser work, right?

[dtbuchholz]

@dtbuchholz remind me, in the Credits wrapper you did, you did not include Blob related methods, right? I think it would be nice to have those as well so that devs can leverage raw blobs in a solidity contract to create their own bucket-like contracts, ie, list of blobs. Does that sound doable?

@sanderpick correct. the credits wrapper exposes only the hoku storage ... and hoku credit ... CLI commands. i.e., i'd need to implement these: add_blob, get_blob, get_blob_status, get_pending_blobs, and delete_blob. the read-only methods are definitely doable, shouldn't take much work to add those.

the work I'm doing in the buckets wrapper obviously has some overlap with the blobs actor, so yes, the mutating methods should also be doable since of the blob's add_blob is the subset of the bucket's add_object.

[sanderpick]

oh right, so you'll have to add the blob write-side methods with your existing work.

i wouldn't worry about:

• get blob status
• get pending blobs

[sanderpick]

@carsonfarmer does your thumb down above mean we should not expect it by mid Nov.?

[carsonfarmer]

Oh sorry, I didn't mean to use a thumbs down. I meant to use a 👍 to indicate an awk. I think we should have a better idea of timing after our call tomorrow.

[dtbuchholz]

note: i just added the issue noted above for the blobs actor having a solidity wrapper for all of its methods: https://github.com/orgs/hokunet/projects/1/views/1?filterQuery=-status%3ADONE++-phase%3APOC+dtb&pane=issue&itemId=85432592&issue=hokunet%7Ccontracts%7C30

i.e., the item under the Blobs section:

All available from solidity contracts (note, adding raw blobs from SDK is not available by design, but should be from contracts so that people can build their own bucket-like / other contracts)

[sanderpick]

I striked some items that are out-of-scope. I didn't delete them for posterity.

[sanderpick]

@stbrody @carsonfarmer I think it would be a good idea to put targets here describing what we're comfortable with re: public testnet. ie,

• X number of txns per sec
• blob upload / download speed
• others?

Maybe this isn't the best place for it... but we should at least take some guess around and model the network load we expect to see when we go "public".

[stbrody]

@sanderpick can this ticket be closed? Please note we now have a whole initiative for Public Testnet Readiness, but I believe everything that was tracked by this ticket has been completed.