feat: import/export a DAG from/to a CAR file

[mikeal]

Across IPLD and Filecoin we’re starting to really zero in on CAR files being the preferred import/export format for DAGs.

It would be great if IPFS had an easy way to export the full graph for any CID to a CAR file and to load a CAR file’s blocks into its local blockstore.

It’s not very hard. I was able to write a proof of concept in less than an hour for the exporter https://github.com/mikeal/export-ipld-graph

[achingbrain]

This might be better as a separate command? Given its size it'd be better not to add more things to IPFS.

$ cat ./path-to-input.car | read-car | ipfs add

$ ipfs get $CID | write-car > ./path-to-output.car

In the API a carSource module similar to globSource or urlSource could be created?

for await (const entry of ipfs.add(carSource(stream))) {
  console.log(entry)
}

Then for outputting, use something like:

await pipe(
  ipfs.get(carCid),
  carWriter(),
  fileWriter('/path/to/output.car')
)

[mikeal]

The car file contains all the blocks for the entire graph, so you have to traverse through all of the links and it’s not quite as simple as this. It could possibly still be a separate command but you’d need to output in a multi-part structure that distinguishes the blocks, which is basically a car file.

The car file library shouldn’t be too large, the POC for export may look large because the default codecs and hash functions used in the new Block API, but js-ipfs wouldn’t need to use that and already includes relevant codecs and hashing functions.

[rvagg]

proposal / WIP @ #2953 to match what go-ipfs is getting

[ribasushi]

Just a poke on this issue: what we are shipping in go-ipfs 0.5 is the final version. It is missing only:

• Import of zero-root .car files (discussion: Present codebase does not allow opening root-less .car files ipld/go-car#26 (comment))
• Progress events on import (no functional changes)

The test fixture and the tests themselves are complete.

/cc @achingbrain @mikeal

[mikeal]

Kicking this again because it’s actually very important.

This needs to ship in js-ipfs because it covers a pretty important compatibility story between IPFS and Filecoin. We’ve made more improvements to the JS library for the CAR format and we can put some more time into this PR if we can get some indication that it might be accepted.

[lidel]

I'm adding this to stewards maintenance backlog.
We need to add support for ipfs.dag.export|import to match go-ipfs implementation, but even bigger need is for js-ipfs-http-client, to work with /api/v0/dag/export|import to enable thin client setups and unblock reliable DAG backups for collabs like Pinata (cc @obo20).

[obo20]

Just to be clear, this isn't an immediate blocker for us. We found some workarounds that are enabling us to directly hit the CLI from nodeJS and make things work that way.

From an ease of use standpoint, having ipfs.dag.export available as a GET request on the gateway endpoint would probably be more useful immediately.

[rvagg]

#2953 uses an outdated stack, which will need to be updated to remove @ipld/block and swap datastore-car for @ipld/car. The work that's landing now, along with @achingbrain's https://github.com/ipld/js-blockcodec-to-ipld-format, makes it easier to bridge the two worlds so it should be a bit more straightforward to implement. IIRC there was a challenge around test cases, mirroring the go-ipfs impl and locking in correctness is difficult without being able to use sharness.

For the http-api, they're basically streaming operations, agnostic of the data (i.e. it's just Uint8Arrays, even if they contain CAR data), so it's roughly just a matter of copying something like the /add pattern (but simpler) for writes and /cat for reads, isn't it?

[rvagg]

Since this is on the JS maintenance board and I'm on the roster next week, this could be something in my wheelhouse to chew in to, since #2953 is most of the way there. But it seems like the priority of this in js-ipfs itself is questionable compared to other work so I'll defer to stewards. We have CAR export from the gateway now, which takes the load off I imagine.