Cannot make payments - signing error "no such channel" when channel does exist

[danielgranhao]

A user reported he was unable to send or receive payments. The logs seem to indicate that the signer cannot find a channel, but a channel does exist.

2024-06-13T12:33:30.902865Z [TRACE] (2) gl_client::signer: [/Users/runner/.cargo/git/checkouts/greenlight-e41cf58dbd255ee0/97e2f41/libs/gl-client/src/signer/mod.rs:443] State updated
2024-06-13T12:33:30.903266Z [DEBUG] (2) vls_protocol_signer::handler: GetPerCommitmentPoint(GetPerCommitmentPoint { commitment_number: 3 })
2024-06-13T12:33:30.903656Z [ERROR] INVALID ARGUMENT: no such channel: 0264a62a4307d701c04a46994ce5f5323b1ca28c80c66b73c631dbcb0990d6e8350100000000000000
2024-06-13T12:33:30.93171Z [ERROR] BACKTRACE:
   0: backtrace::backtrace::trace_unsynchronized
   1: backtrace::backtrace::trace
   2: backtrace::capture::Backtrace::new_unresolved
   3: backtrace::capture::Backtrace::new
   4: lightning_signer::util::status::invalid_argument
   5: lightning_signer::node::Node::get_channel
   6: lightning_signer::node::Node::with_channel_base
   7: <vls_protocol_signer::handler::ChannelHandler as vls_protocol_signer::handler::Handler>::do_handle
   8: vls_protocol_signer::handler::Handler::handle
   9: gl_client::pb::greenlight::amount::Unit::merge
  10: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  11: <breez_sdk_core::greenlight::node_api::Greenlight as breez_sdk_core::node_api::NodeAPI>::start_signer
  12: breez_sdk_core::breez_services::BreezServices::default_config
  13: tokio::runtime::task::core::Core<T,S>::poll
  14: std::panicking::try
  15: tokio::runtime::task::harness::poll_future
  16: tokio::runtime::task::harness::Harness<T,S>::poll_inner
  17: tokio::runtime::task::harness::Harness<T,S>::poll
  18: tokio::runtime::scheduler::multi_thread::worker::run
  19: tokio::runtime::context::scoped::Scoped<T>::set
  20: std::thread::local::LocalKey<T>::with
  21: tokio::runtime::context::set_scheduler
  22: tokio::runtime::context::runtime::enter_runtime
  23: tokio::runtime::scheduler::multi_thread::worker::run
  24: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  25: tokio::runtime::task::core::Core<T,S>::poll
  26: std::panicking::try
  27: tokio::runtime::task::harness::poll_future
  28: tokio::runtime::task::harness::Harness<T,S>::poll_inner
  29: tokio::runtime::task::harness::Harness<T,S>::poll
  30: tokio::runtime::task::LocalNotified<S>::run
  31: std::sys_common::backtrace::__rust_begin_short_backtrace
  32: std::panicking::try
  33: <&T as core::fmt::Display>::fmt
  34: std::sys::pal::unix::thread::thread::new
  35: <unknown>

I can provide more logs if it's helpful.

[cdecker]

Thanks for reporting this @danielgranhao. Do you happen to know the node_id of the node in question? After speaking with the VLS team we have the following potential causes:

• A missed write of the state to the signer state store on the server
• An early removal signal from CLN, i.e., the VLS was told it could forget about the channel, when really we still needed it.
• A removal of a key-value-version tuple from the state store that was not desired

We will look into the state and see if there is anything about that channel in there. If not we likely need to close the channel and have CLN sweep the funds (static return key to the rescue here). We don't yet trace signer state changes, i.e., keep all prior versions of the state, so the root cause analysis is a bit complex, and we may need more occurrences to get to the bottom of this one 🤔

[danielgranhao]

Sure, the node id is 028843175367fcfb42a228bd0b2ddab08d3a2a15c9a4e77009eb20b3e7e430220f

[danielgranhao]

I got this issue again (INVALID ARGUMENT: no such channel) in a different node: 0386170c312d6d8d1e1ec83105379d5c29e7788f287c6400ebd9d727fd73383578

This time it was more severe than before. I couldn't even create invoices. When trying to recover the node using a new client, it couldn't start. After a couple of hours, it resolved itself.

[JssDWt]

The reason invoices could not be created is due to our custom route hint addition in the SDK. That basically means that listpeerchannels is returning an empty response. I would imagine that's the same reason the signer thinks it has no channels @cdecker.

[danielgranhao]

Also, to make it clear, the original node from this issue (referenced here), continues to be unable to make or receive payments, and continues to log no such channel.

[roeierez]

@danielgranhao for the node that is not able to create invoice can you:

1. Put the output here of listpeers or listpeerchannels ?
2. Write here the error you get from the sdk when you attempt to create an invoice?

[danielgranhao]

@roeierez

Put the output here of listpeers or listpeerchannels ?

I could get that for that specific node, but I don't think it will help, given that I can now create invoices. The issue was gone after a couple of hours of reporting it.

Write here the error you get from the sdk when you attempt to create an invoice?

It just hung creating the invoice. I never got an error.

[danielgranhao]

I've checked the node that originally led to this GH issue and it now can send and receive and no longer reports "no such channel". Most likely some GL upgrade had resolved this issue. In the meantime, we haven't seen it in any other node so let's close this for now.