🌟 Parity to IPFS

[daviddias]

Update: I rewrote this first post to make it more useful for tracking the developments of this.

Goal: Add a 'Storage Backend' for js-ipfs that backed by Parity, the Rust implementation of Ethereum. This will enable js-ipfs nodes to fetch content from the Ethereum Network as if it was just a big bucket with all the blocks from the chain.

TASKS

• rust-cid
• Implement an http endpoint in Parity that lets us retrieve the blocks
• Implement the datastore backend that mounts on top of parity
• Try it out, a lot :)

[kumavis]

from parity mh-ipfs branch

cargo build --release
./target/release/parity

while thats running, you can make requests by cid

curl 'http://localhost:5001/api/v0/block/get?arg=z45oqTRzb9a5xyvx5RbfSXH1K5jibyZ4AxnXyYReuLw7KU5veYw'

[kumavis]

i made this util to help with this https://github.com/kumavis/eth-ipld

[daviddias]

@kumavis let's pick up on this again, now that we have datastore bubbling up all the way :)

[kumavis]

whee

[kumavis]

seems simple enough

https://github.com/ipfs/js-datastore-level
https://github.com/ipfs/interface-datastore

how do you mount multiple stores? and can you mount one as read-only?

[kumavis]

also - do you have a preferred isomorphic xhr module? otherwise ill use what i usually do
EDIT: I guess the http module works fine for browserify, not sure about webpack

[daviddias]

We shim stream-http in js-ipfs-api https://github.com/ipfs/js-ipfs-api/blob/master/package.json#L10

The code for reaching to the block store, given that it exposes the same API as go-ipfs does for blocks, is exactly what js-ipfs-api does https://github.com/ipfs/js-ipfs-api/blob/master/src/api/block.js

On read-only, it means that essentially when it calls write, you say 'All good folks', but in reality does nothing.

Right now, ipfs-repo is mounting a single datastore for the blockstore (blocks folder) https://github.com/ipfs/js-ipfs-repo/blob/master/src/index.js#L136-L139, which is them used throughout blockstore code: https://github.com/ipfs/js-ipfs-repo/blob/master/src/blockstore.js#L34-L137

Tasks:

• Ability to mount more than one datastore for blockstore (make it an array essentially)
• Make ./blockstore check if there is an array of this.store and issue the get/put to all of them.

Note: blockstore will pass a base32 encoded CID to your parity-datastore, which you then have to decode to send through the js-ipfs-api block.get.

[kumavis]

would be nice if js-ipfs-repo's data store could be specified externally somehow
seems hardcoded in with a lot of other config https://github.com/ipfs/js-ipfs-repo/blob/master/src/index.js#L136-L139

[daviddias]

Agreed, right now the decision is being made inside the ipfs-repo module by signalling through the browser property in the package.json, here:

https://github.com/ipfs/js-ipfs-repo/blob/master/package.json#L8

I would rather have those options passed in, so that we can hot swap it per application basis, just like libp2p bundles.

[kumavis]

would be nice if we could boot the http api by passing in an instance of an ipfs node, instead of it needing to boot its own

[kumavis]

for now, i think this will work for ipfs-repo's datastore but NotAGoodIdea™

node.on('ready', () => {
  console.log('node ready!')
  // hacky late add to MountStore
  console.log('Adding mount to Parity')
  repo.store.mounts.push({
    prefix: new Key('/'),
    datastore: new RemoteIpfsDataStore({host: 'localhost', port: '5002', protocol: 'http'})
  })
  setupHttpApi()
})

[kumavis]

hmmm seems bitswap is somehow not using my late-mounted datastore

[kumavis]

ipfs-inactive/js-ipfs-http-client#550 ( not blocking, using a string as a workaround )

[kumavis]

ipfs-api uses POST for all requests, parity only implements GET for block.get

API spec seems to indicate it should be a GET
https://github.com/ipfs/http-api-spec/blob/master/apiary.apib

[kumavis]

PR for the http method fix here ipfs-inactive/js-ipfs-http-client#551

[kumavis]

Remaining IPFS todo's

• fix js-ipfs-api http method request-api allow the http method to be specified ipfs-inactive/js-ipfs-http-client#551
• publish datastore-ipfs-http-api (is there a less horrible name to use?) https://github.com/kumavis/js-datastore-ipfs-http-api

optional:

• fix js-ipfs-api cid serialization block.get - use cid.toBaseEncodedString ipfs-inactive/js-ipfs-http-client#550
• change the http-api server api to accept an ipfs instance instead of a ipfs repo location
• change the ipfs-repo datastore config to be easier to configure at the app level

attn @diasdavid

[kumavis]

i have it working and deployed (via patched deps) https://github.com/kumavis/ipfs-eth-bridge

[daviddias]

Checked the boxes that are already done

publish datastore-ipfs-http-api (is there a less horrible name to use?)

Yeah, just call itjs-datastore-ipfs, it is an IPFS adapter for the datastore. Kind of a mobius strip datastore, but that is what it is :) Just make it clear in the Readme

---

@kumavis did you manage to figure out the bug of the other day?

[kumavis]

because of the GET/POST mismatch, ended up making a "read-only-hooked datastore" module, used here: https://github.com/kumavis/ipfs-eth-bridge/blob/master/index.js#L24-L39

[daviddias]

@kumavis What are the next steps?

[b1u3h4t]

how can I find the block data in IPFS format when i use parity with --ipfs-api --ipfs-api-hosts all?
It also return the error block not found, anyone can help me?

[5chdn]

Currently, you can only get block headers, block lists, uncles, transactions, and the state-trie node.

See https://github.com/paritytech/parity/wiki/IPFS

[b1u3h4t]

Do i need an ipfs instance running? if not, how the eth data is encode to ipfs data format?

[5chdn]

You need a Parity node running with IPFS enabled. There is no need for an additional IPFS node unless you want to access other content/APIs via IPFS.

[b1u3h4t]

Is the eth data not saved in ipfs format? Is there no ipfs store indeed? I can't get any data directory about ipfs info, but when i start an additional ipfs node, i can find data in ~/.ipfs/ directory, so where is partiy's ipfs data store?

[daviddias]

It would be amazing to have a package on npm that downloads the Parity client and exposes the datastore with it to access the Eth network so that we can create a quick example on how to set it up. Does this sound like something you can handle in the near future, @kumavis @hermanjunge?

[ghost]

That's right @diasdavid. This is one of our main pain points for the ethereum browser light client currently. I am working on extracting the functions I need from go-ethereum to store the K-V data into a Redis DB cluster.

It would be nice to have a slim parity client version that synchronizes to the network, to store the data into a database we can easily mount as an IPFS client datastore.

For now, we have this docker compose package to do the trick, is a js-ipfs client that mounts the parity HTTP IPFS, that way we can access eth dag nodes one by one. The ideal, of course, is unleash more capabilities.

https://github.com/kumavis/ipfs-eth-bridge/blob/master/mustekala/docker-compose.yml

[jualy007]

Great work

[achingbrain]

js-ipfs is being deprecated in favor of Helia. You can #4336 and read the migration guide.

Helia accepts implementations of interface-blockstore and interface-datastore in a similar way to js-IPFS so this would still be possible, someone just needs to finish the work here and publish the modules!

Closing as not planned but I would love to see this in a usable state still.