[bug?]: batchopenchannel hangs indefinitely

[cj-chua]

Background

batchopenchannel hangs indefinitely. I'm not sure whether this is a bug or if I'm doing something wrong.

This is a new node that I'm trying to bootstrap

Your environment

• 0.17.3-beta commit=lightning-terminal-v0.12.2-alpha
• Linux ip-172-31-44-79 6.2.0-1017-aws Refactor the lnwallet package to directly use the WalletController interface #17~22.04.1-Ubuntu SMP Fri Nov 17 21:19:35 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
• bitcoind
• running lit lnd-integrated mode.

Steps to reproduce

lncli batchopenchannel --sat_per_vbyte=31 '[{"node_pubkey":"026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2","local_funding_amount":10000000}, {"node_pubkey":"03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f","local_funding_amount":10000000}]'

Expected behaviour

Batch open should work

Actual behaviour

The relevant part in the logs...

2024-01-09 16:51:51.758 [INF] CRTR: Processed channels=0 updates=143 nodes=15 in last 1m0.000103299s
2024-01-09 16:52:05.437 [INF] FNDG: Initiating fundingRequest(local_amt=0.10000000 BTC (subtract_fees=false), push_amt=0 mSAT, chain_hash=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f, peer=026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2, min_confs=1)
2024-01-09 16:52:05.491 [INF] FNDG: Target commit tx sat/kw for pendingID(8f8cf6b965c68c901d764d8d2131bfd2a3e3ed06f2fff0f4a8e48f3708a1a133): 20625
2024-01-09 16:52:05.491 [INF] FNDG: Dust limit for pendingID(8f8cf6b965c68c901d764d8d2131bfd2a3e3ed06f2fff0f4a8e48f3708a1a133): 0.00000354 BTC
2024-01-09 16:52:05.491 [INF] FNDG: Starting funding workflow with 45.86.229.190:9735 for pending_id(8f8cf6b965c68c901d764d8d2131bfd2a3e3ed06f2fff0f4a8e48f3708a1a133), committype=anchors-zero-fee-second-level
2024-01-09 16:52:05.491 [INF] FNDG: Initiating fundingRequest(local_amt=0.10000000 BTC (subtract_fees=false), push_amt=0 mSAT, chain_hash=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f, peer=03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f, min_confs=1)
2024-01-09 16:52:05.541 [INF] FNDG: Target commit tx sat/kw for pendingID(4975245916fdb6d0a45382cb19a404ea960cff30dd14bda3383ea76ad436a5fd): 20625
2024-01-09 16:52:05.541 [INF] FNDG: Dust limit for pendingID(4975245916fdb6d0a45382cb19a404ea960cff30dd14bda3383ea76ad436a5fd): 0.00000354 BTC
2024-01-09 16:52:05.541 [INF] FNDG: Starting funding workflow with 3.33.236.230:9735 for pending_id(4975245916fdb6d0a45382cb19a404ea960cff30dd14bda3383ea76ad436a5fd), committype=anchors-zero-fee-second-level
2024-01-09 16:52:05.585 [INF] FNDG: Recv'd fundingResponse for pending_id(4975245916fdb6d0a45382cb19a404ea960cff30dd14bda3383ea76ad436a5fd)
2024-01-09 16:52:05.586 [INF] FNDG: pendingChan(4975245916fdb6d0a45382cb19a404ea960cff30dd14bda3383ea76ad436a5fd): remote party proposes num_confs=3, csv_delay=720

[Roasbeef]

Looks like one of the peers never replied to the initial funding attempt? Is it still just hanging there at this point?

[cj-chua]

Just tried again and yes it's still hanging. Seems like 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2 (Boltz) never replied to the initial funcing attempt since this line was missing

2024-01-10 00:11:10.288 [INF] FNDG: Recv'd fundingResponse for pending_id(6a79a5db9c0273e6de69c91ee2417b398c08dc9651daf803c03acfd404ab7161)

I tried opening a single channel to Boltz and it hangs too... So I guess this is not a batchopenchannel problem

That said, does batchopenchannel just hangs if any one of the peers never respond?

[Roasbeef]

I tried opening a single channel to Boltz and it hangs too... So I guess this is not a batchopenchannel problem

Yeah seems like that node is hanging for w/e reason.

That said, does batchopenchannel just hangs if any one of the peers never respond?

Digging through the code, looks to be the case. I think we want to allow CTRL-C to bail things out, and also implement a long-ish timeout here. In theory, the code (if the option is provided) could also bisect and kick out the slow node, if it's imperative that some of the channels are created (can also set a min amt here as well if needed).

[kilrau]

We'll do a restart of our lnd node later today. No idea why we did let you hang and a restart is the only thing we can imagine to fix things. If there is any info you'd want from us let us know @Roasbeef

[kilrau]

Restart done

[ContinentalNode]

I'm experiencing a similar bug with batchopenchannel hanging indefinitely. Boltz was not one of the peers I was attempting an open to so I don't think it's a Boltz specific issue.

Trying to open a single channel afterwards also does not work and causes part of the on-chain balance to go missing so I need to rescan the chain with --reset-wallet-transactions to recover the balance.

PS it took 4+ hours to rescan less than 200000 blocks, I suppose this number will grow as the node ages, which does not seem sustainable.

[kilrau]

Definitely not Boltz, but rather LND specific. Restart fixed it for now.

[ContinentalNode]

I tried twice after restarts, still broken.

Timeline:

• batchopenchannel not working
• normal restart
• batchopenchannel not working
• single channel open not working and onchain balance goes missing
• restart with --reset-wallet-transactions
• batchopenchannel not working
• single channel open not working and onchain balance goes missing

[hieblmi]

@ContinentalNode I don't think you need to restart with --reset-wallet-transactions to make funds reappear, normal restart should do.

@kilrau, is it possible for you to check your logs to see why @ContinentalNode's @cj-chua's single-channel connection attempt is failing? This seems to be a pervasive issue atm.

[ContinentalNode]

Thanks for the response @hieblmi, trying normal restart now.

I attempted the single-channel open with Bitrefill (03d607f3e69fd032524a867b288216bfab263b6eaee4e07783799a6fe69bb84fac), not Boltz.

Here are what I can find in the log for it.

2024-01-11 10:52:16.844 [INF] FNDG: Initiating fundingRequest(local_amt=0.30000000 BTC (subtract_fees=false), push_amt=0 mSAT, chain_hash=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f, peer=03d607f3e69fd032524a867b288216bfab263b6eaee4e07783799a6fe69bb84fac, min_confs=1)
2024-01-11 10:52:16.844 [INF] CHFD: Performing funding tx coin selection using 7766 sat/kw as fee rate
2024-01-11 10:52:16.917 [WRN] CHFD: Unable to find funding output for shim intent: unable to create witness script, no funding keys
2024-01-11 10:52:17.162 [INF] FNDG: Target commit tx sat/kw for pendingID(40e57fa9a56e3938a79d72b048d2672a7b3abe1ca07f65dc7795a184f4340298): 8117
2024-01-11 10:52:17.162 [INF] FNDG: Dust limit for pendingID(40e57fa9a56e3938a79d72b048d2672a7b3abe1ca07f65dc7795a184f4340298): 0.00000354 BTC
2024-01-11 10:52:17.162 [INF] FNDG: Starting funding workflow with 54.155.41.207:9735 for pending_id(40e57fa9a56e3938a79d72b048d2672a7b3abe1ca07f65dc7795a184f4340298), committype=anchors-zero-fee-second-level
2024-01-11 11:02:46.749 [WRN] FNDG: reservation timed out waiting for peer (peer_id:03d607f3e69fd032524a867b288216bfab263b6eaee4e07783799a6fe69bb84fac, chan_id:40e57fa9a56e3938a79d72b048d2672a7b3abe1ca07f65dc7795a184f4340298)

[Roasbeef]

This should be addressed by: #8385