Add README for Decentralized Backend Storage

[alexcos20]

In order to add multiple decentralized storages, we need a backend that will:

• allow users to upload content
• handle payments
• push the content to a decentralized storage
• return the storage object that is going to be used in the ddo

This will greatly improve the UX flow for publishers.

Arhitecture:

• DBS:

  • exposes API used by frontend (see below)
  • exposes API for microservices (private network)
  • proxies requests to microservices
  • handles IPFS temporary storage

• 1 -N Storage MicroServices (each microservice can handle one storage type)

DBS public API Endpoints:

Info

Description: Information about supported storage types & payments

Path: /

Arguments: None

Returns:

[
    {
        "type": "filecoin",
        "description":  "File storage on FileCoin",
        "payment":
            [ 
                {
                    "chainId": 1,
                    "acceptedTokens": 
                        [ 
                            "OCEAN": "0xOCEAN_on_MAINNET",
                            "DAI": "0xDAI_ON_MAINNET"
                        ]
                },
                {
                    "chainId": "polygon_chain_id",
                    "acceptedTokens": 
                        [ 
                            "OCEAN": "0xOCEAN_on_POLYGON",
                            "DAI": "0xDAI_ON_POLYGON"
                        ]
                }
            ]
    },
    {
        "type": "arweave",
        "description":  "File storage on Arweave",
        "payment":
            [ 
                {
                    "chainId": 1,
                    "acceptedTokens": 
                        [ 
                            "OCEAN": "0xOCEAN_on_MAINNET",
                            "DAI": "0xDAI_ON_MAINNET"
                        ]
                },
                {
                    "chainId": "arweave_chain_id",
                    "acceptedTokens": 
                        [ 
                            "ARWEAVE": "0xARWEAVEtoken_on_arweaveChain"
                        ]
                }
            ]
    }
]    

GetQuote

Description: Gets a quote in order to store some files on a specific storage

Path: POST /getQuote

Arguments:

{
    "type": "filecoin",
    "files": [
                {"length":2343545},
                {"length":2343545},
            ],
    "duration": 4353545453
    "payment": {
        "chainId": 1,
        "tokenAddress": "0xOCEAN_on_MAINNET"
    },
    "userAddress": "0x456"
}

Where:

• type: type of storage desired
• files : array with files length
• duration: how long to store this files (in seconds)
• payment.chainId: chainId that will be used to make the payment
• payment.token: token that will be used to make the payment
• userAddress: user who will pay

Returns:

{
    "tokenAmount": 500,
    "approveAddress": "0x123",
    "chainId": 1,
    "tokenAddress": "0xOCEAN_on_MAINNET",
    "quoteId": "xxxx"
}  

Where:

- tokenAmount - tokenAmount that needs to be approved for "approveAddress" on tokenAddress  (backend server will do a transferFrom to get the payment)
- chainId - chainId used for payment
- quoteId  - backend server will generate a quoteId, used to upload, getStatus and getLink

Upload

Description: Upload files, according to the quote request

Path: POST /upload?quoteId=xxxx&nonce=1&signature=0xXXXXX

Input:

• quoteId
• nonce (timestamp) (has to be higher then previous stored nonce for this user)
• signature: user signed hash of SHA256(quoteID+nonce)

Files are sent using multipart body.

Returns: 200 OK if succeded
DBS will upload files to IPFS Cluster and then it will make a request to the microservice, using the file CIDS.

GetStatus

Description: Gets status for a job

Path: POST /getStatus?quoteId=xxx

Returns:

{
    "status": 0,
}  

Where:

Status	Status Description
0	No such quote
1-99	Waiting for files to be uploaded by the user
100-199	Processing payment
200-299	Processing payment failure modes
300-399	Uploading files to storage
400	Upload done
401-499	Upload failure modes

GetLink

Description: Gets DDO files object for a job

Path: POST /getLink?quoteId=xxx&nonce=1&signature=0xXXXXX

Input:

• quoteId
• nonce (timestamp) (has to be higher then previous stored nonce for this user)
• signature: user signed hash of SHA256(quoteID+nonce)

Returns:

[
    {
       "type": "filecoin",
       "CID": "xxxx",
       "dealIDs":["x" , "x2"]
    }
]

(see https://docs.oceanprotocol.com/core-concepts/did-ddo#files for how each microservice is formatting the response, based on storage type)

DBS private API Endpoints (used by microservices), using on a different port

Register

Description: Register a new microservice which handles a storage

Path: POST /register

Input:

{
    "type": "filecoin",
    "description":  "File storage on FileCoin",
    "url": "http://microservice.url",
    "payment":
            [ 
                {
                    "chainId": 1,
                    "acceptedTokens": 
                        [ 
                            "OCEAN": "0xOCEAN_on_MAINNET",
                            "DAI": "0xDAI_ON_MAINNET"
                        ]
                },
                {
                    "chainId": "polygon_chain_id",
                    "acceptedTokens": 
                        [ 
                            "OCEAN": "0xOCEAN_on_POLYGON",
                            "DAI": "0xDAI_ON_POLYGON"
                        ]
                }
            ]
}  

Returns: 200 OK if accepted

Each microservice should call this endpoint every 10 minutes, otherwise the storage type will be removed from the main list

Sample storage flow:

title Filecoin upload flow

Frontend ->DBS: fetch storage types & payment options
DBS -> Frontend: storage types & payment options
Frontend -> DBS: Get quote for storing 1Mb file on Filecoin
DBS -> Filecoin microservice: Get quote for storing 1Mb file on Filecoin
Filecoin microservice -> DBS : Payment details & quoteId
DBS -> Frontend : Payment details & quoteId


Frontend -> DBS: Upload files
DBS -> IPFS: Write files
DBS -> Filecoin microservice: Store data (upload endpoint)
Filecoin microservice -> IPFS: Read files
Filecoin microservice -> Filecoin microservice: store files on Filecoin

Frontend -> DBS:  Get Status
DBS-> Filecoin microservice :  Get Status
Filecoin microservice -> DBS: Upload done
DBS -> Frontend : Upload done

Frontend -> DBS:  Get Link
DBS-> Filecoin microservice :  Get Link
Filecoin microservice -> DBS: Filecoin object
DBS -> Frontend : Filecoin object

[andreaarmanni]

@alexcos20 Do we have more details on the Websockets and StorJ microservice I can share with the applicants? Similar to what you did with FIlecoin and Arweave

[jrynkiew]

I plan to submit my proposal for an API for file storage, should I fork this repo and submit my proposal via a pull request?

[andreaarmanni]

I plan to submit my proposal for an API for file storage, should I fork this repo and submit my proposal via a pull request?

Kindly submit your proposal via oceanprotocol.com/shipyard through this form

[happy-dev]

I listed my questions here: https://discord.com/channels/612953348487905282/805826044455944212/1022085688293855273

Not sure where is the right place to have them.
Don't hesitate to guide me a bit

[kremalicious]

Quick note that the API design in here can be improved, as using verbs to describe resources is always a bit critical and confusing like POST /get....

I suggest those endpoints, where xxx as the quoteId is always part of the path as it signals a specific resource of those collections:

• GET /
• GET /files/xxx?nonce=1&signature=0xXXXXX
• GET /status/xxx (and the response should have the status text included too, not just a cryptic number)
• POST /quote (using POST here for body arguments only)
• POST /upload/xxx?nonce=1&signature=0xXXXXX

Then I guess this works as a start but I fail to see what exactly is decentralized about this project, and we create a huge maintenance effort as the APIs in the background constantly change. We create another "provider" with this so just like with existing provider, if this single endpoint goes down, nobody can access their files. This is the opposite of decentralized.

Are all frontends supposed to get the final files through this API? Or can they get the files by hitting the storage providers directly?

Also, what's the difference to https://filebase.com? Their approach of creating a unified interface no matter where the file sits sounds like a better approach when creating a centralized solution. And also good example that this functionality can create its own business, yet we try to handle this as yet another feature of Ocean Protocol.

And finally, backend initiating token transfers sounds like an anti-pattern. Why not have user create the transaction so they actually see in wallet how much value is transferred? Why the extra step with the approve? And whoever runs the backend won't have to pay gas fees, which seems to be the current design in here.

[alexcos20]

I listed my questions here: https://discord.com/channels/612953348487905282/805826044455944212/1022085688293855273

Not sure where is the right place to have them. Don't hesitate to guide me a bit

check discord for answers

[alexcos20]

Quick note that the API design in here can be improved, as using verbs to describe resources is always a bit critical and confusing like POST /get....

I suggest those endpoints, where xxx as the quoteId is always part of the path as it signals a specific resource of those collections:

• GET /
• GET /files/xxx?nonce=1&signature=0xXXXXX
• GET /status/xxx (and the response should have the status text included too, not just a cryptic number)
• POST /quote (using POST here for body arguments only)
• POST /upload/xxx?nonce=1&signature=0xXXXXX

Agreed, I will update them

Then I guess this works as a start but I fail to see what exactly is decentralized about this project, and we create a huge maintenance effort as the APIs in the background constantly change. We create another "provider" with this so just like with existing provider, if this single endpoint goes down, nobody can access their files. This is the opposite of decentralized.

Not really. DBS is used as unified upload API. This is fine, because from market/ocean.js/ocean.py we need a single API to work with different storage solutions (and not have a bunch of classes, api's, plugins, etc..)

Are all frontends supposed to get the final files through this API? Or can they get the files by hitting the storage providers directly?

Frontends, once a file was upload, by hitting the "getLink" endpoint, they should get something like:

• for arweave storage

[
    {
    "type": "arweave",
    "transactionId": "a4qJoQZa1poIv5guEzkfgZYSAD0uYm7Vw4zm_tCswVQ",
  }
] 

• for ipfs:

[
    {
    "type": "ipfs",
    "hash": "a4qJoQZa1poIv5guEzkfgZYSAD0uYm7Vw4zm_tCswVQ",
  }
] 

• for filecoin:

[
    {
    "type": "filecoin",
    "cid": "a4qJoQZa1poIv5guEzkfgZYSAD0uYm7Vw4zm_tCswVQ",
  }
] 

(exactly what is described in https://docs.oceanprotocol.com/core-concepts/did-ddo#files)
So even if DBS crashes, there is no impact on the assets

[MantisClone]

EDIT: I think it's likely that the first iteration will only accept payment on EVM-compatible chains. This is possible on Arweave using Bundlr.

A few minor comments about the response from GET /:

                {
                    "chainId": "arweave_chain_id",
                    "acceptedTokens": 
                        [ 
                            "ARWEAVE": "0xARWEAVEtoken_on_arweaveChain"
                        ]
                }

Arweave doesn't have a chain ID in the same way EVM-compatible chains do. The Arweave chain identifies itself as "arweave.N.1". Reference: https://twitter.com/samecwilliams/status/1547013045716766721?s=20&t=JnBIQmax7Y3OOo2coxEzSA

Also, AR tokens are native to Arweave like how ETH is native to Ethereum so it doesn't have an address.

[flyworker]

We add a chainlink oracle to record the cross-chain storage proof in a smart contract, so each storage file becomes more reliable and storage on multiple filecoin nodes.

[kitty2121]

Hi All, Kate here from Functionland! fx.land . is it too late to submit a backend integration. i understand the funding may not be available, we still keen! Which pool of funding would you suggest might help support an OCEAN <> Fx Blox integration. Thank in advance frens [email protected]

[balessan]

@alexcos20 I am starting the work on the DBS as Startin'blox CTO and under the guidance of @happy-dev who actually built the proposal.

I was wondering if you could add me as member of the oceanprotocol organisation here or at least on this project so I could push the beginning of my work, or would you prefer that I clone and open a pull-request only when I have something complete ?

Do you have access to the open-source GitHub program, including some CI Tooling and the rest. Have you been using the CI/CD features on other projects I could check ? I could propose something but we would also need to discuss deployment/infrastructure.

I first started on our gitlab instance with only the setup of a django-based package here: https://git.startinblox.com/applications/oceandbs/ocean-dbs/

Maybe the issue is not the right place to discuss that ?

[balessan]

Exploding this issue into multiple ones, for a better tracking of questions and progress.

[LoznianuAnamaria]

Note: These are the initial specs for the DBS. Next step-> transform it into a readme and add it in the repo

[calina-c]

A couple of mentions on the sample responses:
For all instances of accepted tokens, the delimiters are [] but it seems like a dictionary is intended (i.e. {})
For get Link, the response seems to be a list, but is it possible to have more than one file per quote? If so, is there another format for the upload, where we send files instead of a single file?