[bug]: Fail to chain sync on `testnet3` & `mainnet` errors relating to: script witness item is larger than the max allowed size #7002

rsafier · 2022-10-09T19:34:27Z

Background

Fail to chain sync on testnet3.
The node does show it is up to the tip of the chain , but remains unsynced due to errors

2022-10-09 18:42:39.886 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000032fcf519b61aad2b966348e3f2d27687b26277933cc9881965: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]
....
2022-10-09 18:46:18.715 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]
...

2022-10-09 21:36:23.272 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000014820a254fbcdccce582df2194014c1e3b3f5ecd9259fce663: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

Your environment

0.15.0-beta commit=v0.15.0-beta
Linux ip-10-21-2-70 5.13.0-1031-aws #35~20.04.1-Ubuntu SMP Mon Jun 13 22:30:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
backend bitcoind v23.0.0

Steps to reproduce

Unknown

Expected behaviour

"synced_to_chain": true

Actual behaviour

"synced_to_chain": false with following errors in logs:

2022-10-09 18:42:39.886 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000032fcf519b61aad2b966348e3f2d27687b26277933cc9881965: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]
....
2022-10-09 18:46:18.715 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]
...

2022-10-09 21:36:23.272 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000014820a254fbcdccce582df2194014c1e3b3f5ecd9259fce663: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

The text was updated successfully, but these errors were encountered:

RandyMcMillan · 2022-10-09T19:59:31Z

😬

benthecarman · 2022-10-09T20:05:34Z

Looks like lnd/btcd has a bug in their taproot implementation. In BIP342:

Script size limit The maximum script size of 10000 bytes does not apply. Their size is only implicitly bounded by the block weight limit.[9]

I am surprised this wasn't caught by tests. Are they not using the static test vectors?

HamishMacEwan · 2022-10-09T21:44:45Z

Having this with LND Version v0.15.1 on mainnet:

2022-10-10 10:27:48.835 [INF] LTND: Waiting for chain backend to finish sync, start_height=757924
2022-10-10 10:27:49.535 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

benthecarman · 2022-10-09T22:03:02Z

Got the same error on mainnet:

2022-10-09 16:56:50.707 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

rsafier · 2022-10-09T22:07:00Z

I also also seeing on mainnet v0.15.0:

2022-10-09 21:56:48.549 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

ExImpius · 2022-10-09T22:09:12Z

Same

fotongit · 2022-10-09T22:09:25Z

Same here. Why it suddenly happened to all of us?

Answer: a transaction came in the Blockchain that LND was not able to manage, thus all nodes went down at the same block

hieblmi · 2022-10-09T22:10:37Z

See this on v0.15.1/mainnet


2022-10-09 21:56:55.488 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

benthecarman · 2022-10-09T22:17:04Z

This seems to be the tx that can't be parsed

https://mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

hieblmi · 2022-10-09T22:20:11Z

This seems to be the tx that can't be parsed

https://mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

Posted about here: https://twitter.com/brqgoo/status/1579216353780957185

bnonni · 2022-10-09T22:33:02Z

Same here. Running on testnet. bitcoind v22.0, lnd v0.15.1.

2022-10-09 22:20:06.866 [INF] LTND: Waiting for chain backend to finish sync, start_height=2350195
2022-10-09 22:20:07.828 [INF] LNWL: Started rescan from block 0000000000000027e44f015d141df1aeff3794ac4b0b7303ca1039c6baa35a4c (height 2350075) for 0 addresses
2022-10-09 22:20:08.093 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]

seeing this printed over and over in bitcoind debug.log

2022-10-09T22:30:27Z ThreadRPCServer method=getblockheader user=bitcoinrpc
2022-10-09T22:30:28Z ThreadRPCServer method=getblockchaininfo user=bitcoinrpc

bnonni · 2022-10-09T22:33:18Z

This seems to be the tx that can't be parsed
https://mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

Posted about here: https://twitter.com/brqgoo/status/1579216353780957185

if this is a mainnet tx, why is it impacting nodes running on testnet?

benthecarman · 2022-10-09T22:37:54Z

This seems to be the tx that can't be parsed
mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

Posted about here: twitter.com/brqgoo/status/1579216353780957185

if this is a mainnet tx, why is it impacting nodes running on testnet?

A similar transcation happened on testnet

bnonni · 2022-10-09T22:46:37Z

This seems to be the tx that can't be parsed
mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

Posted about here: twitter.com/brqgoo/status/1579216353780957185

if this is a mainnet tx, why is it impacting nodes running on testnet?

A similar transcation happened on testnet

makes sense. figured he must have tried it on testnet first. but the testnet block referenced doesn't appear to have a tx at the same scale as the one on mainnet
https://mempool.space/testnet/block/0000000000000027e44f015d141df1aeff3794ac4b0b7303ca1039c6baa35a4c

spyhuntergenral · 2022-10-09T22:48:01Z

ahh, few, thought i broke my node for a second .

benthecarman · 2022-10-09T22:48:21Z

The bad tx on testnet is

https://mempool.space/testnet/tx/44692bc2da73192cd0b89bc7a43c0ce43578f6b3567bc945e46e6952e8ec5ca5

bnonni · 2022-10-09T22:50:38Z

Ahh, thanks. I see, its rescanning starting at block height 2350075 meaning the tx is in the next block. Got it. Well damn, I finally got my testnet node up-and-running and was about to make some taro assets and now this 😞

jblachly · 2022-10-09T23:00:25Z

Bug is actually in btcd code:

https://github.com/btcsuite/btcd/blob/fc36cb25a4bdc6d989f9161552e7b0fe08b02939/wire/msgtx.go#L105-L109

brqgoo · 2022-10-09T23:14:20Z

With BIP-342, the maximum script size limit of 10000 bytes no longer applies. The witness script size is only implicitly bounded by the block weight limit.
https://github.com/bitcoin/bips/blob/master/bip-0342.mediawiki#cite_ref-9-0

Roasbeef · 2022-10-09T23:40:57Z

Hey y'all, thanks for bringing this to our attention. I've identified the issue in the btcd wire parsing library, which led to this incident.

AFAICT, the consensus code wasn't the issue here, it was instead that the wire parsing library was erroneously still enforcing a prior check to limit witness sizing left over from segwit v0.

I am surprised this wasn't caught by tests. Are they not using the static test vectors?

btcd is/was using this test vectors. The issue here is that the code the parsed the witnesses for these test vectors isn't the same code that's used to read blocks off the wire. When a new block comes in, we fetch the raw block then attempt to decode it, which triggered this issue.

indomitorum · 2022-10-09T23:48:25Z

Same problem. Have stopped the node now until further notice. 0.15.1

Logs when starting the node

Oct 10 00:34:36 indomitusbtc lnd[1922]: 2022-10-10 00:34:36.014 [INF] LTND: Waiting for chain backend to finish sync, start_height=757941

Oct 10 00:34:36 indomitusbtc lnd[1922]: 2022-10-10 00:34:36.399 [INF] LNWL: Started rescan from block 00000000000000000007477b90ff0ea53cdd1db88e03799af18ff58df0cebaa7 (height 757921) for 2056 addresses

Oct 10 00:34:36 indomitusbtc lnd[1922]: 2022-10-10 00:34:36.685 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

Roasbeef · 2022-10-10T00:24:21Z

The fix to the btcd wire parsing logic can be found here: btcsuite/btcd#1896

This should be safe to apply to those running btcd nodes, which'll allow them to resume validating the main chain (the block was accepted as this wasn't a consensus issue persay).

Once this passes CI and a few more sniff checks, we'll issue a hotfix release for lnd: 0.15.2. This release will only contain the dependency update to the wire parsing library.

Fixes lightningnetwork#7002

chappjc · 2022-10-10T02:27:40Z

Is a bump for neutrino needed too? I think a consumer can hoist the required version themselves, but they might have to require btc directly to do so.

okeygo · 2022-10-10T08:25:46Z

I confirm that my LND nodes are stuck at block 757921 despite I can send and receive Stas normally through established channels, I am proceeding to implement fix.

2022-10-10 08:19:21.786 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]
2022-10-10 08:19:22.258 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]
2022-10-10 08:20:02.063 [INF] CRTR: Processed channels=0 updates=198 nodes=11 in last 1m0.000466268s
2022-10-10 08:20:22.643 [INF] DISC: Broadcasting 321 new announcements in 18 sub batches
2022-10-10 08:21:02.063 [INF] CRTR: Processed channels=0 updates=234 nodes=8 in last 59.999372778s

We need a fast update to the newest lnd version as there was a security issue detected. lightningnetwork/lnd#7002 As the bug was part of the btcd dependencies the fix applied in btcsuite/btcd#1896 brings a long tail of dependency updates. This commit updates the dependencies and adapts for all changes that are introduced by this new dependencies.

rsafier added bug Unintended code behaviour needs triage labels Oct 9, 2022

benthecarman mentioned this issue Oct 9, 2022

October topics austinbitdevs/austinbitdevs.com#11

Closed

rsafier changed the title ~~[bug]: Fail to chain sync on testnet3 errors relating to: script witness item is larger than the max allowed size~~ [bug]: Fail to chain sync on testnet3 & mainnet errors relating to: script witness item is larger than the max allowed size Oct 9, 2022

saubyk added the P0 very high priority issue/PR, blocker on all others label Oct 9, 2022

saubyk removed the needs triage label Oct 10, 2022

Roasbeef mentioned this issue Oct 10, 2022

build: update to version of btcd with wire parsing bug fix #7004

Merged

LightningLuke mentioned this issue Oct 10, 2022

REF: lnd major bug reported #7002 (affecting all nodes) - Unable to deserialize block: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]` mynodebtc/mynode#702

Closed

Roasbeef added a commit to Roasbeef/lnd that referenced this issue Oct 10, 2022

build: update to version of btcd with wire parsing bug fix

b3dc56f

Fixes lightningnetwork#7002

Roasbeef added a commit to Roasbeef/lnd that referenced this issue Oct 10, 2022

build: update to version of btcd with wire parsing bug fix

a502c38

Fixes lightningnetwork#7002

Roasbeef closed this as completed in #7004 Oct 10, 2022

ZeroPointEnergy mentioned this issue Oct 10, 2022

LND suddenly inactive raspiblitz/raspiblitz#3350

Closed

VajraOfIndra mentioned this issue Oct 10, 2022

Update LND to v0.15.2 (URGENT - hot fix) raspibolt/raspibolt#1083

Merged

4 tasks

seberm mentioned this issue Oct 10, 2022

LND not syncing due to a bug (urgent hotfix) mynodebtc/mynode#704

Closed

guggero mentioned this issue Oct 10, 2022

[bug]: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed #7009

Closed

seberm mentioned this issue Oct 10, 2022

lnd: 0.15.1-beta -> 0.15.2-beta NixOS/nixpkgs#195337

Merged

vertiond mentioned this issue Oct 10, 2022

Update LND to v0.15.2-beta to fix stopped mainnet nodes getumbrel/umbrel#1519

Closed

a5an0 mentioned this issue Oct 10, 2022

new fun transaction janoside/btc-rpc-explorer#477

Closed

brh28 mentioned this issue Oct 10, 2022

Socratic Seminar #0 Topics NashBitDevs/nashbitdevs.org#1

Closed

nepet mentioned this issue Oct 10, 2022

gomod: Update to lnd v0.15.2-beta ElementsProject/peerswap#142

Merged

VajraOfIndra mentioned this issue Oct 11, 2022

[FEATURE REQUEST] Add a LND chain/graph sync status raspibolt/raspibolt-pulse#6

Closed

sputn1ck mentioned this issue Oct 11, 2022

not enough witness outputs to create funding transaction, need xxx BTC only have 0 BTC available #7023

Closed

fiatjaf mentioned this issue Oct 12, 2022

plugin-trustedcoin getblock error nbd-wtf/trustedcoin#14

Closed

This was referenced Oct 13, 2022

[feature]: Remove ErrSynchronizingChain from returned error whitelist #7034

Closed

Remove ErrSynchronizingChain #7039

Merged

reez mentioned this issue Oct 15, 2022

Socratic Seminar #1 Topics NashBitDevs/nashbitdevs.org#2

Closed

otech47 mentioned this issue Nov 3, 2022

Explore possible refactors or additional test cases in btcd/wire/msgtx.go btcsuite/btcd#1915

Open

GoVulnBot mentioned this issue Nov 7, 2022

x/vulndb: potential Go vuln in github.com/btcsuite/btcd: CVE-2022-44797, GHSA-2chg-86hq-7w38 golang/vulndb#1098

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bug]: Fail to chain sync on `testnet3` & `mainnet` errors relating to: script witness item is larger than the max allowed size #7002

[bug]: Fail to chain sync on `testnet3` & `mainnet` errors relating to: script witness item is larger than the max allowed size #7002

rsafier commented Oct 9, 2022 •

edited

Loading

RandyMcMillan commented Oct 9, 2022

benthecarman commented Oct 9, 2022

HamishMacEwan commented Oct 9, 2022

benthecarman commented Oct 9, 2022

rsafier commented Oct 9, 2022

ExImpius commented Oct 9, 2022

fotongit commented Oct 9, 2022 •

edited

Loading

hieblmi commented Oct 9, 2022 •

edited

Loading

benthecarman commented Oct 9, 2022

hieblmi commented Oct 9, 2022

bnonni commented Oct 9, 2022

bnonni commented Oct 9, 2022

benthecarman commented Oct 9, 2022

bnonni commented Oct 9, 2022

spyhuntergenral commented Oct 9, 2022

benthecarman commented Oct 9, 2022

bnonni commented Oct 9, 2022

jblachly commented Oct 9, 2022

brqgoo commented Oct 9, 2022 •

edited

Loading

Roasbeef commented Oct 9, 2022

indomitorum commented Oct 9, 2022 •

edited

Loading

Roasbeef commented Oct 10, 2022 •

edited

Loading

chappjc commented Oct 10, 2022 •

edited

Loading

okeygo commented Oct 10, 2022 •

edited

Loading

[bug]: Fail to chain sync on testnet3 & mainnet errors relating to: script witness item is larger than the max allowed size #7002

[bug]: Fail to chain sync on testnet3 & mainnet errors relating to: script witness item is larger than the max allowed size #7002

Comments

rsafier commented Oct 9, 2022 • edited Loading

Background

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

RandyMcMillan commented Oct 9, 2022

benthecarman commented Oct 9, 2022

HamishMacEwan commented Oct 9, 2022

benthecarman commented Oct 9, 2022

rsafier commented Oct 9, 2022

ExImpius commented Oct 9, 2022

fotongit commented Oct 9, 2022 • edited Loading

hieblmi commented Oct 9, 2022 • edited Loading

benthecarman commented Oct 9, 2022

hieblmi commented Oct 9, 2022

bnonni commented Oct 9, 2022

bnonni commented Oct 9, 2022

benthecarman commented Oct 9, 2022

bnonni commented Oct 9, 2022

spyhuntergenral commented Oct 9, 2022

benthecarman commented Oct 9, 2022

bnonni commented Oct 9, 2022

jblachly commented Oct 9, 2022

brqgoo commented Oct 9, 2022 • edited Loading

Roasbeef commented Oct 9, 2022

indomitorum commented Oct 9, 2022 • edited Loading

Roasbeef commented Oct 10, 2022 • edited Loading

chappjc commented Oct 10, 2022 • edited Loading

okeygo commented Oct 10, 2022 • edited Loading

[bug]: Fail to chain sync on `testnet3` & `mainnet` errors relating to: script witness item is larger than the max allowed size #7002

[bug]: Fail to chain sync on `testnet3` & `mainnet` errors relating to: script witness item is larger than the max allowed size #7002

rsafier commented Oct 9, 2022 •

edited

Loading

fotongit commented Oct 9, 2022 •

edited

Loading

hieblmi commented Oct 9, 2022 •

edited

Loading

brqgoo commented Oct 9, 2022 •

edited

Loading

indomitorum commented Oct 9, 2022 •

edited

Loading

Roasbeef commented Oct 10, 2022 •

edited

Loading

chappjc commented Oct 10, 2022 •

edited

Loading

okeygo commented Oct 10, 2022 •

edited

Loading