Goerli returns "UNHANDLED_ERROR: DBError(VectorChunkError(Missing..." for slot 6613004

[quick-pawiromitchel]

Description

Network: ETH Goerli

This slot number https://goerli.beaconcha.in/slot/6613004 returns with {"code":500,"message":"UNHANDLED_ERROR: DBError(VectorChunkError(Missing { chunk_index: 51649 }))","stacktraces":[]}

Version

Lighthouse v4.5.0-441fc16

Present Behaviour

curl -X 'GET' 'http://localhost:5052/eth/v1/beacon/states/6613004/validators?id=0xac6d6c03f91356800b5a7c096dd9fd408ba393191c7b1ad171e4f55b3101c62c6204cf92d5c7fc74de9a9186183478ad' -H 'accept: application/json' -H "Content-Type: application/json"

Returns with this error

{"code":500,"message":"UNHANDLED_ERROR: DBError(VectorChunkError(Missing { chunk_index: 51649 }))","stacktraces":[]}

Expected Behaviour

Should return data correctly

[michaelsproul]

Have you recently updated to v4.5.0 from v4.3.0 or earlier? There's a bug that can cause this sort of issue which will heal within slots-per-restore-point slots (~28 hrs max)

This is the bug: #4697

[quick-pawiromitchel]

Yes, we've updated from 4.3 this morning. Thank you for the context

[quickchase]

So the bug suggests that it's just a scary log message, not that API functionality to return historical states would be broken 😞

I assume that when you say it will heal within the slots-per-restore-point you mean that it will fully heal and we don't need to do anything to backfill data, correct? (archive nodes, we have --reconstruct-historic-states)

[michaelsproul]

@quickchase Sorry, yeah, should clarify in the description that the DB is temporarily corrupt

No need for manual intervention. The lower your slots-per-restore-point the quicker it will heal.

[michaelsproul]

Closing as a dupe of #4697. We'll try to target a fix for #4697 for 4.6.0