BEP-364: Primary Storage Provider acts as the upload agent for object creation and update on Greenfield

[alexgao001]

  BEP: 364
  Title: Primary Storage Provider acts as the upload agent for object creation and update on Greenfield
  Status: Draft
  Type: Standards
  Created: 2024-03-16

BEP-364: Primary Storage Provider acts as the upload agent for object creation and update on Greenfield

• BEP-364: Primary Storage Provider acts as the upload agent for object creation and update on Greenfield
  • 1. Summary
  • 2. Motivation
  • 3. Status
  • 4. Specification
  • 5. License

1. Summary

This BEP introduces a new method where the primary storage provider (SP) acts as the upload agent for creating objects in Greenfield. It allows users to send objects directly to the primary SP without sending transactions on chain. The primary SP will then create the object on-chain on behalf of the user, eliminating the requirement for users to switch over to the Greenfield network and incur gas fees when they create objects on Greenfield.

2. Motivation

The current object creation process requests users to send a MsgCreateObject transaction to the Greenfield chain. After the object is successfully created on chain, users would then send the object's payload data to the primary SP for data storage. Once the primary SP confirms the object’s integrity and distributes the EC chunks to secondary SPs, a MsgSealObject transaction will be sent to the Greenfield chain to finalize the object, enabling regular access to the object afterward.

This approach ensures the safety of object creation. However, when integrating with Greenfield, developers often face the following challenges:

Developers must conduct erasure coding (EC) processing and calculate checksums for objects on the client side. This increases integration complexity and resource usage on the client, particularly affecting user experience when handling large objects on web platforms.

When a user uploads an object on Greenfield, they are required to initiate a transaction to create the object. This process presents several challenges:
Users must switch to the Greenfield network before uploading objects.
Users need to possess BNB on the Greenfield network.
Each object upload requires a transaction confirmation, resulting in multiple wallet pop-ups for signatures when uploading many objects. Greenfield provides a batch upload solution using temporary accounts, but this method complicates integration.

The process for uploading objects is too complex, needing many interactions with SP and the Greenfield chain, making integration harder. Some project stakeholders may find integration exceptionally challenging if they can't use the official SDK directly due to version problems.

In light of the challenges encountered during the previous object creation process in Greenfield, we are planning to introduce a new method of object creation while retaining the existing approach.

The new solution enables users to send objects to the Primary SP directly using SP's API, eliminating the need for users to perform any additional steps. This significantly streamlines the complexity of object creations for users.

3. Status

This BEP is in progress.

4. Specification

4.1 APIs

4.1.1 SP APIs

SP currently exposes two APIs for users to upload object after they created/updated the object on-chain:

• for small objects, refer to https://docs.bnbchain.org/greenfield-docs/docs/api/storage-provider-rest/put_object/
• resumable upload for large objects, refer to https://docs.bnbchain.org/greenfield-docs/docs/api/storage-provider-rest/resumable_put_object/

for delegators who want to upload objects to SP, APIs are almost the same and users just need to specify it is a delegated request using the query parameter “delegate”, if it is for object update, specify the query parameter “is_update”.

ParameterName	Type	Required	Description
delegate	string	false	specify whether need SP to delegate create object onchain. Either “” or not provided
is_update	string	false	specify whether to delegate to update a existing object. Either true or false

SDK

There are encapsulated interfaces provided by greenfiled-go-sdk to fulfill users' needs.

DelegatePutObject

The DelegatePutObject interface enables users to send objects to the primary SP, allowing the primary SP to create objects on the chain on behalf of the user. It supports resumable uploads, by automatically fetching previously uploaded segments to allow for seamless continuation, improving upload efficiency.

DelegatePutObject(ctx context.Context, bucketName, objectName string, objectSize int64, reader io.Reader, opts types.PutObjectOptions) error

DelegateUpdateObjectContent

The DelegateUpdateObjectContent interface enables users to send updated objects to the primary SP directly without interaction with the Greenfield chain to update their existing object, allowing the primary SP to update objects on the chain on behalf of the user.

DelegateUpdateObjectContent(ctx context.Context, bucketName, objectName string, objectSize int64, reader io.Reader, opts types.PutObjectOptions) error

4.1.2 Greenfield Chain New Messages

Greenfield introduces several new messages for SP acts as the upload agent feature

MsgDelegateCreateObject

The MsgDelegateCreateObject message enables the primary SP linked to a bucket to create objects on behalf of users authorized to do so within the bucket. It includes below attributes:

• Operator: the account address of the SP
• Creator: the account address of the object creator
• BucketName: the name of the bucket
• ObjectName: the name of object
• PayloadSize: size of the object's payload
• ContentType: the format of the object which should be a standard MIME type.
• Visibility: visibility means the object is private or public
• ExpectChecksums: a list of hashes which was generated by the redundancy algorithm. It is optional.
• RedundancyType: can be ec or replica

The MsgDelegateUpdateObjectContent message enables the primary SP linked to a bucket to update objects on behalf of users authorized to do so within the bucket. It includes below attributes:

• Operator: the account address of the SP
• Updater: the account address of the object updater
• BucketName: the name of the bucket
• ObjectName: the name of the object
• PayloadSize: the size of the object's payload
• ContentType: the format of the object which should be a standard MIME type
• ExpectChecksums: a list of hashes which was generated by the redundancy algorithm. It is optional

MsgSealObjectV2

In our previous implementation, users were responsible for calculating the checksum themselves and specifying it during object creation. However, in this new method, when the SP creates an object, it does not have the complete object information. It has to wait until the SP seals the object to fill this information into the object info. To address this, we need to upgrade the MsgSealObject message to MsgSealObjectV2 to enhance this functionality.

MsgToggleSPAsDelegatedAgent

It is used by bucket owners to enable/disable the delegated create/update feature for a specific bucket.

• Operator: account address of the operator, only the bucket owner can send the tx
• BucketName: the name of the bucket

4.2 New workflow for object creation/update on Greenfield

The delegate create/update object workflows is outlined below:

4.2.1 Users

Users can simply use the APIs introduced in section 4.1, to upload objects to the primary SP of the bucket, without incurring any transaction fees or having to switch to the Greenfield network.

Users are required to sign the message. This step ensures SPs can verify the user's identity.
If users use go-sdk/js-sdk to send the “create/update object” request to SPs, the wallet signature will be added to the request automatically.
If users use a web-based Dapp, the Dapp needs to integrate Greenfield off-chain-auth mechanism, which has a built-in EDDSA signing process and makes sure that the request sent to SPs are attached with users’ signature. About EDDSA authentication, see BEP-346

4.2.2 SP

SP needs to do the following tasks:

1. Ensure the user has permission to create/update the object in the bucket. Send the MsgDelegateCreateObject/MsgUpdateObject transaction to Greenfield to create/update the object that represents the user, and wait for the object ID if creating a new object.
2. After generating the object ID on-chain, store the object piece and its metadata in S3 and the database promptly.
3. After the object is fully uploaded, the integrity hash for all segments will be calculated and stored in the database by the primary SP.
4. The primary SP should execute erasure coding (EC), calculate integrity hashes for each EC chunk, store them in the database, and replicate all EC chunks to the secondary SPs.
5. The secondary SP verifies the integrity hash of the received piece with the one sent by the primary SP, then signs to seal the object.
6. The primary SP collects signatures from secondary SPs, seals the object on-chain, and adds the integrity hash to objectinfo.

4.3 Security

This type of proxy for creating objects provides users with great convenience, but it also introduces potential security issues:

1. The primary SP may have the potential to tamper with objects. In such cases, users can download the objects for comparison. The easiest method is to use the md5sum tool for verification.
2. The primary SP may maliciously substitute objects not specified by the user for upload. In such cases, since the transactions creating the objects are all recorded on the blockchain, users can easily trace and find evidence of this situation.

Once users detect malicious behavior from the primary SP, they have the following options:

1. Users can send the tx ToggleSPAsDelegatedAgent to prevent the Primary SP from creating files further, users can create object on their own.
2. By initiating the MigrateBucket operation, users can switch the primary SP and select a trusted SP to act as the new primary SP for the Bucket.

5. License

The content is licensed under CC0.