feat: add debug_getHistoricalSummaries endpoint

[acolytec3]

Motivation

The Portal Network needs a way to access the HistoricalSummaries property on the BeaconState for post Capella beacon state since the HistoricalSummaries are used to anchor EL block header proofs to the beacon state roots in light client updates that the serve as the source of truth for verifying the authenticity of post capella EL block headers. This currently requires downloading the entire Beacon State from a beacon node willing to serve the debug endpoints and then throwing away everything except for the desired data (or else downloading an ERA file from a willing endpoint). It is desirable to be able to access this data more efficiently directly from beacon nodes.

Description

This adds a new getHistoricalSummaries endpoint to the debug namespace on the REST API that serves the HistoricalSummaries field from the current head BeaconState as SSZ bytes. We only need the most current one for use in constructing proofs of recent EL block headers embedded in a beacon block.

This mirrors similar experimental work being done in the nimbus client

Steps to test or reproduce

git checkout historicalSummaries-endpoint
./lodestar beacon \
--rest.namespace='*' \
--checkpointSyncUrl=https://beaconstate.info \

In a separate terminal

curl -X GET --header "Accept: application/octet-stream" "http://127.0.0.1:9596/eth/v1/debug/historical_summaries" > ./hs.ssz

Then, use a script like below to verify that historical summaries are correctly returned

    import { ssz } from '@lodestar/types'

    const summariesBytes = fs.readFileSync('./test/util/hs.ssz')
    const summaries = ssz.capella.BeaconState.fields.historicalSummaries.deserialize(summariesBytes)
    console.log(summaries)

[CLAassistant]

All committers have signed the CLA.

[g11tech]

lgtm, we can always move the endpoint once we have it in beacon apis, CI running

[nflaig]

Mostly looks good to me, just few remarks regarding loading the state from bytes and need to make CI happy

[acolytec3]

@nflaig general question, should I go ahead and move over to the lodestar namespace for now, following what nimbus is doing? I don't have an opinion on which namespace (nor do I know if this proposal would ultimately end up under debug or under maybe beacon as a light client API (since presumably this will be used mainly by Portal Network clients).

One further question, does it make sense to add the meta to the response mirroring the getState API? We don't use this directly in Portal Network so I don't have the context to know if this is a useful addition to the overal API.

[nflaig]

general question, should I go ahead and move over to the lodestar namespace for now, following what nimbus is doing?

Yeah, might be better to go with lodestar namespace if it's anyhow not standardized between clients.

One further question, does it make sense to add the meta to the response mirroring the getState API? We don't use this directly in Portal Network so I don't have the context to know if this is a useful addition to the overal API.

If it would become standardized as part of beacon-api then it will for sure have the meta as well, if you don't need it we don't have to add it now, we can also always add it later on if there is demand.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.92%. Comparing base (37c4287) to head (a2e9cd7).

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #7245      +/-   ##
============================================
+ Coverage     48.51%   48.92%   +0.41%     
============================================
  Files           600      601       +1     
  Lines         40142    40174      +32     
  Branches       2057     2061       +4     
============================================
+ Hits          19474    19655     +181     
+ Misses        20630    20481     -149     
  Partials         38       38              

[acolytec3]

general question, should I go ahead and move over to the lodestar namespace for now, following what nimbus is doing?

Yeah, might be better to go with lodestar namespace if it's anyhow not standardized between clients.

One further question, does it make sense to add the meta to the response mirroring the getState API? We don't use this directly in Portal Network so I don't have the context to know if this is a useful addition to the overal API.

If it would become standardized as part of beacon-api then it will for sure have the meta as well, if you don't need it we don't have to add it now, we can also always add it later on if there is demand.

Sounds good. I have moved everything to lodestar and added a new unit test file for lodestar since it doesn't appear there was a unit test at all for that namespace. I opted to just create a specific test for that endpoint since running the GenericServerTest looks like it would require me to add unit testData for all of the lodestar endpoints which feels like it's beyond the scope of this PR.

[wemeetagain]

Suggested change

	state instanceof Uint8Array ? loadState(config, chain.getHeadState(), state).state : state.clone();
	(state instanceof Uint8Array ? loadState(config, chain.getHeadState(), state).state : state.clone()) as BeaconStateCapella;

You can avoid the any below by casting this to a BeaconStateCapella here

[acolytec3]

It seems like the proof size for HistoricalSummaries changed in deneb (and will maybe again in Electra). Is it still safe to cast it to BeaconStateCapella here since the shape of the state is changing somewhat?

[nflaig]

LGTM

[nflaig]

@acolytec3 I converted this to lower case to align it with existing json responses

[nflaig]

Changed to use v1 since it's now part of lodestar namespace

[acolytec3]

Thanks for the approval! @nflaig once this gets merged, any idea how long before the public lodestar bootnodes will make it available?

[nflaig]

any idea how long before the public lodestar bootnodes will make it available?

We plan to cut a rc at the end of the week, we discussed in the standup today to get your PR in for that as well, so the next release if testing goes well should be some time next week which means the public nodes would be updated next week as well.

[wemeetagain]

🎉 This PR is included in v1.24.0 🎉

[acolytec3]

Thank you sir 🖖