Blockheight disagreement, not aborting - WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS

[JssDWt]

node id: 03226c467e77b102a718acd72e92a65830906c6de186c7ac07cdee021d30469089
session id: 7276342403923968

time="2024-12-21T22:07:59+01:00" level=info msg="BreezSDK: [INFO] node-logs: DEBUG   plugin-pay: cmd 34 partid 8: Blockheight disagreement, not aborting." group=lightning
time="2024-12-21T22:07:59+01:00" level=info msg="BreezSDK: [INFO] node-logs: INFO    plugin-pay: cmd 34 partid 8: failed: WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS (reply from remote)" group=lightning

It appears there's a blockheight mismatch in a regular payment, where the greenlight node is behind. I believe the greenlight node uses the 'current' blockheight rather than the sync block height when sending payments? Why is this node behind?

[cdecker]

It does use the highest known chain length as offset. So either bitcoind is returning an old height (that'd be a new one), or we are using the correct height. Notice that the disagreement could also be on the recipient side (do you have access to verify on the recipient?) and/or the recipient could believe the invoice is expired due to clock drift or it has already been paid (which is the nice thing to do here, and not just claim multiple times).

I'll of course check the session to verify as soon as possible.

[cdecker]

Looking into this a bit more I can see the following:

• 22:07:33+01:00: New trampoline payment via 0264a62a4307d701c04a46994ce5f5323b1ca28c80c66b73c631dbcb0990d6e835: lnbc1909u1
• 22:07:34+01:00: We should be ready to send a payment.
• 22:07:34+01:00: Calling pay on the node. This is strange since I thought this is supposed to be a trampoline payment, and we're calling pay? It also seems to start splitting and routing HTLCs.

@nepet do you know what might be going on here? It seems like the trampoline code is delegating to pay instead.

[JssDWt]

@cdecker I think this is the fallback payment. If a trampoline payment fails, the Breez SDK will fall back to regular payment.

[cdecker]

Ok, that makes sense, but why are we getting the call, not logging anything, and falling back on pay immediately?

As for the original error, we were in sync with the blockchain, according to block 0000000000000000000138dc7311b22155f6cd9763e9ee937c480ae37ab93b55, having been mined at 2024-12-21 21:58:27 GMT +1, and being added at time 2024-12-21T22:07:18+01:00, so the error is on the receiving side. If I had to guess I'd say lnd2.blink.sv was lagging behind the blockchain, and telling us that they couldn't verify that the timelock is safe to use.

[JssDWt]

If I had to guess I'd say lnd2.blink.sv was lagging behind the blockchain, and telling us that they couldn't verify that the timelock is safe to use.

Looking at this code it seems that the block height reported by the recipient is ahead:
https://github.com/ElementsProject/lightning/blob/087a29b0b3216fb39548842712f06b1599a9037d/plugins/libplugin-pay.c#L1249-L1252

*blockheight <= p->start_block == false, meaning p->start_block < *blockheight

It could still be an issue at the recipient where they return a too high blockheight, but it seems likelier that start_block is low somehow.

[cdecker]

I think the return you are looking for is the one a couple of lines above: the recipient returns the opaque error, so we can't infer it is due to blockheight disagreement. Fwiw we've also seen a couple of nodes that'll just forget invoices they issued (WoS is an example of that, as they seem to route them around in their system, resulting in payments being rejected if the passing around isn't complete).

Without further information, all I can say is that we were up-to-date, so blockheight disagreement must've been on the recipient, or it was any other cause.

[JssDWt]

I think the return you are looking for is the one a couple of lines above: the recipient returns the opaque error, so we can't infer it is due to blockheight disagreement.

The function has to return true there, in order to fall into the logline here: https://github.com/ElementsProject/lightning/blob/087a29b0b3216fb39548842712f06b1599a9037d/plugins/libplugin-pay.c#L1289-L1293

I agree it needs recipient information. I reached out to galoy, asking their insights on whether they're customizing the block height there.