go-ipfs retrieves content from js-ipfs

[flyingzumwalt]

Acceptance Criteria

• a go-ipfs node can consistently & reliably retrieve content from js-ipfs nodes

This requires getting a stream multiplexer functional in both languages.

Background

go-ipfs and js-ipfs do not share a working stream multiplexer. The ones that work go-go are not implemented in javascript and the ones that work js-js are not implemented in go.

[whyrusleeping]

Updates here. After adding in support to go-multiplex for half closed streams (which are, in my opinion, unnecessary) we have discovered that the js implementation of multiplex also does not support half closed streams.

We're working on fixing that

[daviddias]

As soon as these last test passes -- daviddias/multiplex#2 --, we should be golden!

[daviddias]

@whyrusleeping did you get the chance to bubble up the changes yesterday?

[whyrusleeping]

ipfs/kubo#3695

working on it. Tests arent passing for some reason, will look at them as soon as i can. In the meantime you should be able to build off of that branch and test

[daviddias]

Getting a:

# JS side
LIBP2P_MUXER=multiplex jsipfs daemon
# ...
» jsipfs id
{
  "id": "QmWxqpnDn1ZFcaJQ8nEpAkcTz9xR8foaNXnwpY9THi34e7",
  "publicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQChWiNNuHw1cCabhbtvSaCxH0KcY2ueipkasRkHYkJ6QfrpCO8mvK0QGbLMpK9YlzZ+Yc1tKUwydVwxNLu7msqau6kKCAgpOJ1II3U9ct4Mx59w4SGY8m5bWZXgTmqx6Tw03CzZl206/sPnRq+tzxsGdizQo+rZQtYr8fiHlF8G4YDy0QLO2gCbn5sxRvza/TlJeDpR1Nu1nlXWJJkQaxfNCsih3HxtXlmE2IfzFW+HZaSRiqh7Hxrn01UWcnY+O91Mhr3WQkyaRch09fBBS7vcQigTu+m+anhNVNrRiVGtn2uB4RV6CYSb3XCAmP76BpRa8lAN9Ew/EjgK0xh4azgtAgMBAAE=",
  "addresses": [
    "/ip4/127.0.0.1/tcp/14002/ipfs/QmWxqpnDn1ZFcaJQ8nEpAkcTz9xR8foaNXnwpY9THi34e7",
    "/ip4/127.0.0.1/tcp/14003/ws/ipfs/QmWxqpnDn1ZFcaJQ8nEpAkcTz9xR8foaNXnwpY9THi34e7",
    "/ip4/192.168.2.215/tcp/14002/ipfs/QmWxqpnDn1ZFcaJQ8nEpAkcTz9xR8foaNXnwpY9THi34e7",
    "/ip4/192.168.2.29/tcp/14002/ipfs/QmWxqpnDn1ZFcaJQ8nEpAkcTz9xR8foaNXnwpY9THi34e7"
  ],
  "agentVersion": "js-ipfs",
  "protocolVersion": "9000"
}
#....
# Go side
> LIBP2P_MUX_PREFS="/mplex/6.7.0" ipfs daemon
# ...
> ipfs swarm connect /ip4/127.0.0.1/tcp/14002/ipfs/QmWxqpnDn1ZFcaJQ8nEpAkcTz9xR8foaNXnwpY9THi34e7
Error: connect QmWxqpnDn1ZFcaJQ8nEpAkcTz9xR8foaNXnwpY9THi34e7 failure: dial attempt failed: failed to add conn to ps.Swarm: selected protocol we don't have a transport for

[Kubuxu]

@diasdavid you have to pass: --enable-mplex-experiment to daemon

[daviddias]

It's working \o/

[daviddias]

Continue testing, it works for a lot of cases, but when we start talking about bigfiles (50MB++), go-ipfs kills the connection it has with js-ipfs, doing swarm connect again will result in go-ipfs closing the socket again. This might be related with the tests @whyrusleeping saw failing

Update: got a Panic;

goroutine 2386 [running]:
panic(0x8238c0, 0xc4222325d0)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
gx/ipfs/QmfYR3dFXT5udj6sn6F7dCQkQtjH45v8qyr5vnxTfAzxL3/go-multiplex.(*Stream).receive(0xc42201cbb0, 0xc425b50a80, 0x16, 0x20)
        /Users/ground-control/code/go-projects/src/gx/ipfs/QmfYR3dFXT5udj6sn6F7dCQkQtjH45v8qyr5vnxTfAzxL3/go-multiplex/stream.go:40 +0x122
gx/ipfs/QmfYR3dFXT5udj6sn6F7dCQkQtjH45v8qyr5vnxTfAzxL3/go-multiplex.(*Multiplex).handleIncoming(0xc4203a8000)
        /Users/ground-control/code/go-projects/src/gx/ipfs/QmfYR3dFXT5udj6sn6F7dCQkQtjH45v8qyr5vnxTfAzxL3/go-multiplex/multiplex.go:254 +0x2b8
created by gx/ipfs/QmfYR3dFXT5udj6sn6F7dCQkQtjH45v8qyr5vnxTfAzxL3/go-multiplex.NewMultiplex
        /Users/ground-control/code/go-projects/src/gx/ipfs/QmfYR3dFXT5udj6sn6F7dCQkQtjH45v8qyr5vnxTfAzxL3/go-multiplex/multiplex.go:59 +0x2c8

[whyrusleeping]

@diasdavid Is there more info to that panic? like, the line above that says panic: TEXT DESCRIBING WHAT WENT WRONG

[whyrusleeping]

@diasdavid Ah, I see how a panic could occur there. But the interesting thing is that it could only happen if you sent data after sending a close to me.

[dignifiedquire]

@diasdavid @whyrusleeping what's the status of this? @diasdavid how can I reproduce the panic you are seeing?

[dignifiedquire]

Tried the latest branches mentioned here with the interop tests from js-ipfs. Seems there is still a lot of things not working currently.

Current setup:

• js-ipfs: 78ac8cf13ce76fd53a47a2d76325419a840f38d7 (current master)
• go-ipfs: update go multiplex dep with half closed streams support kubo#3695

Command:

> LIBP2P_MUX_PREFS="/mplex/6.7.0" IPFS_EXEC=`which ipfs` LIBP2P_MUXER=multiplex npm run test:interop:node

• go -> js is very slow
• js -> go starts failing for everything > 1024bit

[dignifiedquire]

I discovered that identify is failing, when using the above mentioned setup, it works fine if I switch to spdy.

[dignifiedquire]

More updates

• @whyrusleeping thinks the issues are related to half closing
• I can confirm this, going back to the version [email protected] (no support for half close) makes things much better.
• In the interop tests I can observer (with half closing)
  • first test js -> go works
  • every other after fails
  • It seems this is related to streams not being properly closed and/or not being properly reused.
  • Testing against spdy these issues are not there, so they are specific to muliplex

[dignifiedquire]

I have a branch with types + logging for multiplex on the js side here: https://github.com/dignifiedquire/multiplex/tree/flow

[dignifiedquire]

@whyrusleeping

<•dignifiedquire> whyrusleeping: good news I figured out why things are broken, due to a bug , the js side is sending multiplex ids starting with 1 again because connection reuse doesn't do what it should be doing
<•dignifiedquire> but because of halfOpen, the original connection no1 is not closed, so go ipfs gets confused, and things start failing
New messages
<•dignifiedquire> whyrusleeping: so I don't think you need to do anything, will need to go and fix the js side first

[dignifiedquire]

Not fully done yet

Still needs

• feat: add multiplex support by default ipfs-inactive/js-libp2p-ipfs-browser#160
• feat: add multiplex to default muxers ipfs-inactive/js-libp2p-ipfs-nodejs#78

[dignifiedquire]

One more PR: libp2p/js-libp2p-switch#179 and we are good to go for browser multiplexing!

[ghost]

I think this is done cc @dignifiedquire

[daviddias]

Shipped and released :)