xud crash (after one successful call)

[reliveyy]

This is related to #729. After starting the SIMNET, gRPC immediately become unresponsive after one successful call (getinfo).

Not sure if this matters: I noticed my local DNS didn't work in the moment of the gRPC crash. Situation might be reproduced by disabling your local DNS lookup for a while.

vagrant@xud-simnet-1:~$ xud-simnet-start
starting all processes
btcd started
ltcd started
lndbtc started
lndltc started
xud started
sync lndbtc to chain
.
lndbtc synced to chain
sync lndltc to chain
.
lndltc synced to chain
Ready!
vagrant@xud-simnet-1:~$ xucli getinfo
{
  "version": "1.0.0-alpha.5",
  "nodePubKey": "037c514ee25ebbfc10726442325bdf3210b6ee80a9b3618aa96788a06b01696535",
  "urisList": [],
  "numPeers": 0,
  "numPairs": 1,
  "orders": {
    "peer": 0,
    "own": 0
  },
  "lndbtc": {
    "error": "",
    "channels": {
      "active": 2,
      "inactive": 0,
      "pending": 0
    },
    "chainsList": [
      "bitcoin"
    ],
    "blockheight": 74128,
    "urisList": [],
    "version": "0.5.0-beta commit=v0.4.2-beta-1248-g7901d38ae57e82c0e1ddaf5ad0230f9f0af18724",
    "alias": "BTC@xud-simnet-1"
  },
  "lndltc": {
    "error": "",
    "channels": {
      "active": 3,
      "inactive": 0,
      "pending": 0
    },
    "chainsList": [
      "litecoin"
    ],
    "blockheight": 133409,
    "urisList": [],
    "version": "0.5.0-beta commit=v0.4.2-beta-1248-g7901d38ae57e82c0e1ddaf5ad0230f9f0af18724",
    "alias": "LTC@xud-simnet-1"
  }
}
vagrant@xud-simnet-1:~$ xucli listpeers
Error: 14 UNAVAILABLE: Connect Failed

[reliveyy]

xud3.log

[sangaman]

I can't reproduce this. DNS should not impact the rpc layer assuming you're making calls on the same machine as the server or using IP addresses. Do you have a different setup?

Does this issue persist for you? Is the xud process crashing?

[kilrau]

It shouldn't impact the RPC, but it's just a guess. Might be something totally different. We are directly using the terminal on the machine where xud is running.

Once it's there, it's persisting and only restart of xud fixes it. We run into it every couple of minutes but feels random.

[kilrau]

We mislead you: xud process is always down when Error: 14 UNAVAILABLE: Connect Failed is thrown. Just reproduced that several times.

[kilrau]

Are you running xud in nomatching mode invoking the ExecuteSwap Call? That's how we reliably reproduce it.

[sangaman]

I haven't been running in nomatching mode so I will give that a try tomorrow and see if I can figure anything out. If I'm still stuck we should maybe try to set up a time to debug this in real time together.

[kilrau]

Updated status of this issue:
I believe we just found it. If no payment channel exists or payment channel is exhausted between two nodes and one calls executeSwap for a swap between the two nodes, xud crashes. xud process is gone. Can reproduce this in 100% of the time. Give it a try @sangaman

FYI: It appeared random yesterday because of ExchangeUnion/xud-simnet#17. We only had it once where it crashed on xucli getinfo with Error: 14 UNAVAILABLE: Connect Failed and couldn't reproduce thereafter.

[kilrau]

Did you manage to reproduce? @sangaman @moshababo

[sangaman]

Not yet but I'm looking at it now, do you happen to have any more logs or any of the output from the crash? I did find a bug with the xucli executeswap command and opened a PR to fix it but I don't think it's the cause here unless you're using the command line.

[kilrau]

Only https://github.com/ExchangeUnion/xud/files/2651717/xud3.log

And no we used the gRPC, so that won't fix it: https://github.com/kilrau/xud-exchange-integration-example/blob/master/engine.py#L392

You can just spin up our test exchange and connect it to xud to reproduce the behavior: https://github.com/kilrau/xud-exchange-integration-example/

[sangaman]

There's really no output when the xud process stops? It just silently goes away? Very weird, maybe I'll have to try this again later with the exchange because I can't reproduce and am running out of ideas.

[kilrau]

Yup, it just crashes and that's it. Let me spin up a test environment with the two exchanges on GCloud for ya

[kilrau]

Done, check Slack @sangaman

[kilrau]

Added @reliveyy , he can help to reproduce

[reliveyy]

Cannot reproduce the crash on the new xud version. So close it.