flattened multipart message and new ipfsAPI().add API

[daviddias]

The latest 0.4.0 version has an update on what it is expecting when it comes to multipart messages which breaks how ipfs add -r. multipart messages have been historically a pain to handle, parse and construct. A new and cleaner way to do it is simply to flat the multipart messages (avoiding the nesting).

In order to let the IPFS HTTP API to know if we are sending a file, directory or symlink, ipfsAPI client will send the following "Content-Type":

• application/x-directory when it is a directory
• application/x-symlink when it is a symlink
• application/octet-stream when it is a file

The .add interface is going to be updated to:

.add(buffer)                   (browser + node.js)
.add(stream)                   (browser + node.js)
.add(path)                     (node.js)
.add(array of paths)           (node.js)
.add(path, {recursive: true})  (node.js)
.add([                         (browser + node.js)
  { “/a/b”: <stream or buffer> } 
  { “/a/e/f”: <stream or buffer> }, 
  { “/a/eeee/“: true},            -> Sinalize it is a directory
  { “/a/im-a-symlink”: “/a/“}     -> Creates a symlink
])

This calls will create always a flatened multipart message

@whyrusleeping @dignifiedquire

[daviddias]

• let's keep the dev of this on the feat/files-api branch since they are both 0.4.0 changes (+ it is what making tests fail with 0.4.0 now)

[dignifiedquire]

Changed

.add([
  {path: 'my/file.txt', content: <stream or buffer or string>}
  {path: 'my/folder', dir: true}
])

[fbaiodias]

While investigating how symlinks are handled on ipfs add -r, I compared with go-ipfs and the feat/files-api and noticed the following:

• go-ipfs sends symlinks with Content-Type: application/symlink and the target file path in the body
• js-ipfs-api by default follows symlinks and sends the targets contents (file symlink, directory symlink and file inside directory symlink)
• js-ipfs-api when used with followSymlinks: false is broken and sends directories symlinks with Content-Type: application/x-directory, still follows them but sends its content files also with Content-Type: application/x-directory and no body

(You can see the wireshark captures here: https://gist.github.com/xicombd/174f3af4ad7e341f2f56)

Should we keep the follow symlinks functionality on by default on js-ipfs-api or replicate go-ipfs cli?

I'll try to fix the followSymlinks: false bugs in the meanwhile :)

@diasdavid @dignifiedquire

[chpio]

what's about a stream of file objects (file objects as specified by @dignifiedquire)? this would enable file streaming. atm all files are loaded into the memory and then passed to the ipfs api. but with streams (and back pressure magicz) only a limited amount of files are needed to be in ram.

[chpio]

import ipfsApi from 'ipfs-api';
import streamMap from 'through2-map';

const ipfs = ipfsApi({...});

const addStream = ipfs.createAddStream(); // with an implicit {recursive: true}

// then use it with eg gulp/vinyl file objects:
gulp.src('bla/**/*')
  .pipe(...)
  .pipe(streamMap.obj(v => ({path: v.relative, content: v.contents}))) // convert vinyl to "file object"
  .pipe(addStream);

i dont think we could put this functionality into the current ipfs.add function and pass the stream to it, cause we couldnt exactly determine the stream type. also it would be more "node like" (fs.createWriteStream).

---

Passing an array of file streams, aka:

ipfs.add([{
  {path: 'foo', content: fs.createReadStream('foo')},
  {path: 'foo2', content: fs.createReadStream('foo2')},
}]);

has a few disadvantages:

• can easily reach the open file limit (ulimit)
• not optimal in all use cases (when file objects can be streamed but the file content can not, eg. the file content is coming from another fancy system which emits buffers, but can be paused)

[dignifiedquire]

What exactly do we want to do here, the refactoring which was mentioned at the beginning of the issue was done and released.

[daviddias]

I think we are good in this one :)