0.20.0-rc2: rc1 & rc2 fail to boot up with WebTransport in Addresses.Announce

[ylempereur]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.tech

Version

Kubo version: 0.20.0-rc2
Repo version: 13
System version: amd64/darwin
Golang version: go1.19.8

Config

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://webui.ipfs.io.ipns.localhost:8080",
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [
      "/ip4/127.0.0.1/tcp/4001",
      "/ip4/127.0.0.1/udp/4001/quic",
      "/ip4/127.0.0.1/udp/4001/quic-v1",
      "/ip4/127.0.0.1/udp/4001/quic-v1/webtransport",
      "/ip4/75...170/tcp/4001",
      "/ip4/75...170/udp/4001/quic",
      "/ip4/75...170/udp/4001/quic-v1",
      "/ip4/75...170/udp/4001/quic-v1/webtransport",
      "/ip6/::1/tcp/4001",
      "/ip6/::1/udp/4001/quic",
      "/ip6/::1/udp/4001/quic-v1",
      "/ip6/::1/udp/4001/quic-v1/webtransport",
      "/ip6/2001:...:5c59/tcp/4001",
      "/ip6/2001:...:5c59/udp/4001/quic",
      "/ip6/2001:...:5c59/udp/4001/quic-v1",
      "/ip6/2001:...:5c59/udp/4001/quic-v1/webtransport"
    ],
    "AppendAnnounce": null,
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/127.0.0.1/tcp/4001",
      "/ip4/127.0.0.1/udp/4001/quic",
      "/ip4/127.0.0.1/udp/4001/quic-v1",
      "/ip4/127.0.0.1/udp/4001/quic-v1/webtransport",
      "/ip4/192.168.2.9/tcp/4001",
      "/ip4/192.168.2.9/udp/4001/quic",
      "/ip4/192.168.2.9/udp/4001/quic-v1",
      "/ip4/192.168.2.9/udp/4001/quic-v1/webtransport",
      "/ip6/::1/tcp/4001",
      "/ip6/::1/udp/4001/quic",
      "/ip6/::1/udp/4001/quic-v1",
      "/ip6/::1/udp/4001/quic-v1/webtransport",
      "/ip6/2001:...:5c59/tcp/4001",
      "/ip6/2001:...:5c59/udp/4001/quic",
      "/ip6/2001:...:5c59/udp/4001/quic-v1",
      "/ip6/2001:...:5c59/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt"
  ],
  "DNS": {
    "Resolvers": null
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooW...V"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": null,
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": [
      {
        "Addrs": [],
        "ID": "12D3KooW...T"
      }
    ]
  },
  "Pinning": {
    "RemoteServices": {
      "Web3_Storage": {
        "API": {
          "Endpoint": "https://api.web3.storage"
        },
        "Policies": {
          "MFS": {
            "Enable": false,
            "PinName": "",
            "RepinInterval": ""
          }
        }
      },
      "pinata": {
        "API": {
          "Endpoint": "https://api.pinata.cloud/psa"
        },
        "Policies": {
          "MFS": {
            "Enable": false,
            "PinName": "",
            "RepinInterval": ""
          }
        }
      }
    }
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "6h0m0s",
    "Strategy": "all"
  },
  "Routing": {
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.128.0/ipcidr/17",
      "/ip4/192.168.64.0/ipcidr/18",
      "/ip4/192.168.32.0/ipcidr/19",
      "/ip4/192.168.16.0/ipcidr/20",
      "/ip4/192.168.8.0/ipcidr/21",
      "/ip4/192.168.4.0/ipcidr/22",
      "/ip4/192.168.0.0/ipcidr/23",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {}
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

The daemon fails to finish booting if I try to use 20 rc1 or rc2. If I switch back to the 0.19.1 binary, it boots fine.

Here is the log from an rc2 boot:

> ipfs daemon --migrate --enable-gc --routing=auto
Initializing daemon...
Kubo version: 0.20.0-rc2
Repo version: 13
System version: amd64/darwin
Golang version: go1.19.8

Computed default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "8.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 30720

Theses can be inspected with 'ipfs swarm resources'.

It simply hangs there and does nothing until I kill it.

Here is the log if I switch back to 0.19.1 (I removed some lines in the middle):

> ipfs daemon --migrate --enable-gc --routing=auto     
Initializing daemon...
Kubo version: 0.19.1
Repo version: 13
System version: amd64/darwin
Golang version: go1.19.8

Computed default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "8.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 30720

Theses can be inspected with 'ipfs swarm resources'.

Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1
...
Swarm announcing /ip6/::1/tcp/4001
Swarm announcing /ip6/::1/udp/4001/quic
Swarm announcing /ip6/::1/udp/4001/quic-v1
Swarm announcing /ip6/::1/udp/4001/quic-v1/webtransport
API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready

Please advise.

[hacdias]

@ylempereur hey! I'm sorry you're experiencing this. I cannot reproduce your issue on macOS. I tried your configuration (adapting where needed due to secrets being removed). What version of macOS are you running? For how long did you try running the daemon before you killed it?

[ylempereur]

@hacdias First, I just upgraded to 0.19.2 and there's no problem there.

Next, I reran rc2 and waited 15 mins; still stuck (0.19.2 boots in under 2 seconds). I tried the following from another terminal before I killed it:

> ipfs id
Error: lock /Users/yves/.ipfs/repo.lock: someone else has the lock
> 

Also, one ctl-C doesn't kill it, it takes two.

I'm running "macOS Ventura 13.3.1 (a)" on an Intel Mac (16GB RAM).

If you create a debugging/logging branch, I can build it and post the log here.

For reference, while I'm not a go dev, I'm a back end dev with 40+ years of experience. In fact, I was a Mac dev from '84 to '98, and the first half of that was entirely in 68K assembler. So, I can follow directions 😬

[hacdias]

@ylempereur just updated to 13.3.1 (a). I'm using a M1 laptop and I can run both arm64 and amd64 binaries. There's something else going on here. Since you're at it, would you mind trying building from source: https://github.com/ipfs/kubo#build-from-source?

I want to understand if the problem is:

1. Current code
2. The built binaries
3. Your configuration
4. Your laptop

@guseggert has managed to pin-point the issue: removing the WebTransport announce addresses from the configuration fixes the issue. Thanks for reporting. We'll work on finding the root cause to fix it.

Reproducible on macOS, Linux and Windows.

Libp2p upgrade caused the issue: 3c363eb

[p-shahi]

Thread in #libp2p-implementers public slack channel: https://filecoinproject.slack.com/archives/C03K82MU486/p1683208653452199 and potential fix: libp2p/go-libp2p#2268