Ocean Uploader Arweave Microservice

A microservice for uploading files to Arweave. To be used with Ocean Uploader Backend.

• 🔌 Endpoints
  • 🏷️ getQuote
  • ⬆️ upload
  • 🔄 getStatus
  • 🔗 getLink
  • 📜 getHistory
• ✅ Register
• ▶️ Run
• 🧪 Test
  • ✋ Manual tests
  • 🤖 Automated tests
  • 📦 Expensive, large upload test
  • 🖥️ Example Curl Commands
• 🤝 Support
• 🗺 Roadmap
• 🤖 Contributing
• 📜 License
• 🔗 Associated Projects

Getting Started

Clone the repository and navigate to the project directory:

git clone https://github.com/oceanprotocol/uploader_arweave.git
cd arweave-upload

Make sure that you have a compatible version of node.js installed (as specified in .nvmrc). We recommend running:

nvm use

Install the necessary dependencies:

npm install

Start the server using:

npm start

Endpoints

getQuote

Description: Gets a quote in order to store some files

Path: POST /getQuote

Arguments:

{
  "type": "arweave",
  "files": [{ "length": 2343545 }, { "length": 2343545 }],
  "duration": 4353545453,
  "payment": {
    "chainId": 1,
    "tokenAddress": "0xWETH_on_ETHERUEM"
  },
  "userAddress": "0x456"
}

Where:

• type: type of storage desired
• files : array with files length
• duration: how long to store this files (in seconds) (irrelevent for Arweave storage)
• payment.chainId: chainId that will be used to make the payment
• payment.token: token that will be used to make the payment
• userAddress: address from which payment is pulled

Returns:

{
  "tokenAmount": 500,
  "approveAddress": "0x123",
  "chainId": 1,
  "tokenAddress": "0xWETH_on_MAINNET",
  "quoteId": "xxxx"
}

Where:

• tokenAmount: tokenAmount that needs to be approved
• approveAddress: The address of the microservice that needs to be approved (microservice will do a transferFrom to get the payment)
• chainId: chainId used for payment
• tokenAddress: token that will be used to make the payment
• quoteId: backend server will generate a quoteId

upload

Description: Upload some files

Path: POST /upload

Input:

{
  "quoteId": "23",
  "nonce": 12345.12345,
  "signature": "0x2222",
  "files": ["ipfs://xxxx", "ipfs://yyyy"]
}

Microservice will upload files to Arweave and it will take the payment

Returns: 200 OK if all the pre-checks pass. Upload occurs asynchronously. Call getStatus to monitor status.

getStatus

Description: Gets status for a job

Path: POST /getStatus?quoteId=xxx

Returns:

{
  "status": 0
}

Where:

Status	Status Description
0	No such quote
1-99	Waiting for files to be uploaded by the user
100-199	Processing payment
200-299	Processing payment failure modes
300-399	Uploading files to storage
400	Upload done
401-499	Upload failure modes

getLink

Description: Gets DDO files object for a job

Path: POST /getLink?quoteId=xxx&nonce=1&signature=0xXXXXX

Input:

{
  "quoteId": "23",
  "nonce": 12345.12345,
  "signature": "0x2222"
}

Where:

• quoteId: quoteId generated by backend server
• nonce: (timestamp) (has to be higher then previous stored nonce for this user)
• signature: user signed hash of SHA256(quoteID+nonce)

Returns:

[
  {
    "type": "arweave",
    "transactionHash": "xxxx"
  }
]

getHistory

Description: Gets history quotes for a certain user

Path: GET /getHistory?userAddress=xxx&nonce=1&signature=0xXXXXX

Input:

{
  "userAddress": "0x1234",
  "nonce": 12345.12345,
  "signature": "0x2222"
}

Where:

• userAddress: wallet address
• nonce: (timestamp) (has to be higher then previous stored nonce for this user)
• signature: user signed hash of SHA256(''+nonce)

Returns:

[
  {
    "type": "arweave",
    "quoteId": "23",
    "userAddress": "0x111",
    "status": 400,
    "chainId": 80001,
    "tokenAddress": "0x222",
    "tokenAmount": "999999999",
    "approveAddress": "0x1234",
    "transactionHash": "xxxx"
  }
]

Register

Every 10 minutes (configurable), Arweave microservice should register itself to Uploader Backend, using the register endpoint. DBS_URI will be defined as env.

POST DBS_URI/register

{
  "type": "arweave",
  "description": "File storage on Arweave",
  "url": "http://microservice.url",
  "payment": [
    {
      "chainId": 1,
      "acceptedTokens": [
        { "OCEAN": "0xWETH_on_ETHEREUM" },
        { "DAI": "0xOCEAN_ON_ETHEREUM" }
      ]
    },
    {
      "chainId": 137,
      "acceptedTokens": [
        { "OCEAN": "0xWMATIC_on_POLYGON" },
        { "DAI": "0xOCEAN_ON_POLYGON" }
      ]
    }
  ]
}

Run

export ACCEPTED_PAYMENTS=ethereum,matic
export NODE_RPC_URIS=default,default
export BUNDLR_URI="https://node1.bundlr.network"
#export BUNDLR_URI="https://devnet.bundlr.network" # Use Budnlr devnet when interacting with testnets
export PORT=8081
export PRIVATE_KEY="0000000000000000000000000000000000000000000000000000000000000000"
export SQLITE_DB_PATH=/path/to/db/file
export REGISTRATION_INTERVAL=30000 # default: 30000 ms
export DBS_URI="https://localhost" # "DEBUG" to skip registration
export SELF_URI="https://localhost"
export IPFS_GATEWAY="https://cloudflare-ipfs.com/ipfs/" # should have trailing slash
export ARWEAVE_GATEWAY="https://arweave.net/" # should have trailing slash
export MAX_UPLOAD_SIZE=1099511627776 # in bytes, 0 means unlimited
export BUNDLR_BATCH_SIZE=1 # default: 1 (chunk at a time)
export BUNDLR_CHUNK_SIZE=524288 # default: 524288 (512 kB)
export BUNDLR_PRICE_BUFFER=10 # percent, default: 10
export GAS_PRICE_BUFFER=10 # percent, default: 10

npm start

Test

Testing requires two accounts:

PRIVATE_KEY is the server account TEST_PRIVATE_KEY is the client account

The server needs an operational reserve of native gas fee tokens, enough to cover one full upload procedure (transferFrom, unwrap (withdraw), send, wrap (deposit), transfer).

The client needs sufficient wrapped token to cover upload fee + reimburse the server's gas fees.

Manual tests

export TEST_PRIVATE_KEY="0000000000000000000000000000000000000000000000000000000000000000"
export ENABLE_EXPENSIVE_TESTS="false"

npm test

Automated tests

This repo uses Github Actions to run the tests.

The tests use public testnets: Goerli and Mumbai

Warning Don't run multiple instances of the tests at the same time. It will cause race conditions with token approvals.

• Tests: https://github.com/MantisClone/arweave-upload/blob/main/test
• Config: https://github.com/MantisClone/arweave-upload/blob/main/.github/workflows/ci.yml

Expensive, large upload test

The large upload test is run periodically (quarterly) via Github Actions. It requires over 4 WMATIC in the client account (TEST_PRIVATE_KEY).

• Config: https://github.com/MantisClone/arweave-upload/blob/main/.github/workflows/ci_expensive.yml

Example Curl Commands

curl -d '{ "type":"arweave", "userAddress": "0x0000000000000000000000000000000000000000", "files": [{"length": 1048576}, {"length": 256}], "payment": {"chainId": 137, "tokenAddress": "0x0000000000000000000000000000000000000000"} }' -X POST -H 'Content-Type: application/json' http://localhost:8081/getQuote

curl -d '{ "quoteId":"60f7d48ccd08653b2ef2edfe4bbe4620", "signature": "0x0000000000000000000000000000000000000000", "files": ["https://example.com/", "ipfs://xxx"], "nonce": 0 }' -X POST -H 'Content-Type: application/json' http://localhost:8081/upload

curl -d '{ "type":"arweave", "userAddress": "0x0000000000000000000000000000000000000000", "files": [{"length": 1256}, {"length": 5969}], "payment": {"chainId": 80001, "tokenAddress": "0x0000000000000000000000000000000000001010"} }' -X POST -H 'Content-Type: application/json' http://localhost:8081/getQuote
curl -d '{ "quoteId":"40acc6937e1bd98631f47e7cbda72920", "signature": "0x0000000000000000000000000000000000000000", "files": ["https://example.com/", "https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png"], "nonce": 0 }' -X POST -H 'Content-Type: application/json' http://localhost:8081/upload
curl 'http://localhost:8081/getStatus?quoteId=40acc6937e1bd98631f47e7cbda72920'
curl 'http://localhost:8081/getLink?quoteId=40acc6937e1bd98631f47e7cbda72920&signature=0x0000000000000000000000000000000000000000&nonce=0'

Support

Please open issues on github if you need support of have any questions.

Roadmap

Stay tuned for more integrations and services. Follow the issues on github to see the latest development plans.

Contributing

This project is fully open-source, backed by the OCEAN community and is open for contributions.

License

Released under the Apache 2.0 License.

Associated Projects

• Uploader Filecoin microservice
• Ocean Uploader Backend