Tune past epochs in storage #1944

brentstone · 2023-09-26T23:44:13Z

Describe your changes

Closes #1798, closes #1857, closes #1903, closes #1783. All of these relate to changing how many past epochs are kept in storage for various PoS data. A refactor is needed to communicate the max_proposal_period governance parameter to the data.

Closes #1961. The validator's protocol keys are moved into PoS and now epoched as well.

Indicate on which release or other PRs this topic is based on

Based on #1943, which is based on v0.23.0

Checklist before merging to `draft`

I have added a changelog
Git history is in acceptable state

brentstone · 2023-09-28T00:34:42Z

@tzemanovic @Fraccaman @sug0

The PR is ready for a look for anyone who wants to check it out. Here's a summary of the main storage changes:

consensus_validator_set kept for max_proposal_period + 2 epochs in the past (previously 2)
validator_deltas kept for max(max_proposal_period, slash_processing_delay) + 2 epochs in the past (previously 23)
validator_addresses kept for 2 epochs in the past (previously infinite)
validator_positions kept for 2 epochs in the past (previously infinite)

I've used a +2 epochs in the past for these things just to err on the side of safety. We can adjust if desired. The bonds are already kept for all epochs, so this and the validator_deltas should hopefully suffice for governance. At minimum, validator_deltas need to be kept for slash_processing_delay epochs, hence the max.

sug0

left some nits & questions, mostly

core/src/ledger/storage_api/governance.rs

ethereum_bridge/src/protocol/transactions/votes/update.rs

proof_of_stake/src/epoched.rs

apps/src/lib/node/ledger/shell/finalize_block.rs

brentstone · 2023-09-28T16:07:13Z

pls update wasms

Co-authored-by: Tiago Carvalho <[email protected]>

brentstone · 2023-10-02T20:11:27Z

@sug0 All the epoch-length changes to the PoS data are being made in this PR, as this is the relevant topic. The eth keys have been changed in 7c7ee02.

Also want to note that you are able to query the total consensus stake of any epoch in the past directly from storage without computation using the total_consensus_stake_key_handle (may change the name later to remove key though). Currently this is already kept for all past epochs.

@tzemanovic the enqueued slashes are now manually cleared in 85770be.

fix tests

sug0 · 2023-10-04T07:43:00Z

proof_of_stake/src/types.rs

+pub type ValidatorProtocolKeys = crate::epoched::Epoched<
+    common::PublicKey,
+    crate::epoched::OffsetPipelineLen,
+    crate::epoched::OffsetMaxProposalPeriodPlus,


I think the offset can be reduced in the future. I'm not sure what the appropriate value for this should be, though.

* brent/tune-storage-past-epochs: move protocol keys into epoched PoS data keep `total_deltas` for same period of time as `validator_deltas` clear old enqueued slashes when processing slashes keep validator eth keys for max proposal period fixup! pos/types: configure number of past epochs kept for PoS data fixed `epoched` tests and `test_validator_sets` [ci] wasm checksums update changelog: add #1944 PoS: refactor usages of `Epoch::checked_sub` core/types/storage:m ore flexible fn checked_sub param fixup! replace direct storage read of PosParams with an RPC fn replace direct storage read of PosParams with an RPC fn fix PoS crate standalone build fixup! Apply suggestions from code review Apply suggestions from code review docstring cleanup new impl for purging old validator sets pos/types: configure number of past epochs kept for PoS data add PosParams type with added gov param WIP want to pass `max_proposal_period` from gov params into PoS

* origin/brent/tune-storage-past-epochs: move protocol keys into epoched PoS data keep `total_deltas` for same period of time as `validator_deltas` clear old enqueued slashes when processing slashes keep validator eth keys for max proposal period fixup! pos/types: configure number of past epochs kept for PoS data fixed `epoched` tests and `test_validator_sets` [ci] wasm checksums update changelog: add #1944 PoS: refactor usages of `Epoch::checked_sub` core/types/storage:m ore flexible fn checked_sub param fixup! replace direct storage read of PosParams with an RPC fn replace direct storage read of PosParams with an RPC fn fix PoS crate standalone build fixup! Apply suggestions from code review Apply suggestions from code review docstring cleanup new impl for purging old validator sets pos/types: configure number of past epochs kept for PoS data add PosParams type with added gov param WIP want to pass `max_proposal_period` from gov params into PoS

brentstone added 2 commits September 26, 2023 16:34

refactor Epoched for past epochs

4f89775

cleaning: remove unused code + improve docstrings

d8a29ef

brentstone force-pushed the brent/tune-storage-past-epochs branch from 1abd194 to bab044f Compare September 27, 2023 00:29

brentstone marked this pull request as ready for review September 28, 2023 00:23

brentstone force-pushed the brent/tune-storage-past-epochs branch from 256a0bb to 514c7b6 Compare September 28, 2023 00:26

brentstone force-pushed the brent/tune-storage-past-epochs branch from 514c7b6 to 432d143 Compare September 28, 2023 00:39

sug0 requested changes Sep 28, 2023

View reviewed changes

tzemanovic added 2 commits September 28, 2023 14:20

test/eth: init storage with PoS params where needed

8416f0b

test/PoS/epoched: init PoS params and fix PastEpochs type param

f5d4544

brentstone force-pushed the brent/tune-storage-past-epochs branch from d3048b3 to 311c192 Compare September 28, 2023 15:18

changelog: add #1943

d3f1eb3

brentstone added a commit that referenced this pull request Sep 28, 2023

changelog: add #1944

d1535ac

brentstone force-pushed the brent/tune-storage-past-epochs branch from 311c192 to d1535ac Compare September 28, 2023 15:27

github-actions bot and others added 15 commits September 28, 2023 16:22

[ci] wasm checksums update

812b2c2

WIP want to pass max_proposal_period from gov params into PoS

bfd91f9

add PosParams type with added gov param

053fffb

pos/types: configure number of past epochs kept for PoS data

bfe4c97

new impl for purging old validator sets

eaed189

docstring cleanup

85bf4aa

Apply suggestions from code review

6364cc0

Co-authored-by: Tiago Carvalho <[email protected]>

fixup! Apply suggestions from code review

1ddd82b

fix PoS crate standalone build

f1bc264

replace direct storage read of PosParams with an RPC fn

f40bd26

fixup! replace direct storage read of PosParams with an RPC fn

41d3838

core/types/storage:m ore flexible fn checked_sub param

620e4e2

PoS: refactor usages of Epoch::checked_sub

856130e

changelog: add #1944

20180c4

[ci] wasm checksums update

3fa8d38

brentstone mentioned this pull request Oct 2, 2023

Epoched validator protocol key #1961

Closed

brentstone force-pushed the brent/tune-storage-past-epochs branch 2 times, most recently from 7337c5c to 01b6815 Compare October 4, 2023 03:42

brentstone requested review from sug0 and tzemanovic October 4, 2023 03:42

move protocol keys into epoched PoS data

64e817b

fix tests

brentstone force-pushed the brent/tune-storage-past-epochs branch from 01b6815 to 64e817b Compare October 4, 2023 04:13

brentstone added PoS ethereum-bridge labels Oct 4, 2023

sug0 approved these changes Oct 4, 2023

View reviewed changes

tzemanovic mentioned this pull request Oct 4, 2023

Need total staked tokens for older governance proposals #1783

Closed

brentstone added the ready for draft label Oct 4, 2023

brentstone mentioned this pull request Oct 13, 2023

Draft #1980

Closed

tzemanovic approved these changes Oct 17, 2023

View reviewed changes

cwgoes mentioned this pull request Oct 20, 2023

[tracking] Ethereum bridge readiness #2023

Closed

29 tasks

Fraccaman added a commit that referenced this pull request Oct 23, 2023

fixup! Merge branch 'brent/tune-storage-past-epochs' (#1944)

9e97e99

tzemanovic mentioned this pull request Oct 24, 2023

0.24.0 rc2 #2039

Merged

tzemanovic merged commit 64e817b into main Oct 24, 2023
11 of 12 checks passed

tzemanovic deleted the brent/tune-storage-past-epochs branch October 24, 2023 11:26

tzemanovic mentioned this pull request Oct 26, 2023

Store only essential Merkle tree snapshots #2043

Closed

brentstone added a commit that referenced this pull request Nov 11, 2023

changelog: add #1944

631df42

sug0 mentioned this pull request Feb 22, 2024

namada_sdk::rpc::get_all_validators always returns empty set #2693

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tune past epochs in storage #1944

Tune past epochs in storage #1944

brentstone commented Sep 26, 2023 •

edited

Loading

brentstone commented Sep 28, 2023

sug0 left a comment

brentstone commented Sep 28, 2023

brentstone commented Oct 2, 2023 •

edited

Loading

sug0 Oct 4, 2023

Tune past epochs in storage #1944

Tune past epochs in storage #1944

Conversation

brentstone commented Sep 26, 2023 • edited Loading

Describe your changes

Indicate on which release or other PRs this topic is based on

Checklist before merging to draft

brentstone commented Sep 28, 2023

sug0 left a comment

Choose a reason for hiding this comment

brentstone commented Sep 28, 2023

brentstone commented Oct 2, 2023 • edited Loading

sug0 Oct 4, 2023

Choose a reason for hiding this comment

brentstone commented Sep 26, 2023 •

edited

Loading

Checklist before merging to `draft`

brentstone commented Oct 2, 2023 •

edited

Loading