Replay protection storage optimization #1977

grarco · 2023-10-09T14:06:36Z

Describe your changes

Addresses #1566.

Moves the replay protection storage outside of subspace to prevent writing the diff keys (still keeps the keys written at the previous height to allow for a rollback). Removes the previous solution to prevent merkelizing the replay protection keys which isn't needed anymore. Finally, implements a small optimization for which only one hash for tx is ever needed in storage (either the wrapper or the inner one, instead of both).

Improves unit tests.

Indicate on which release or other PRs this topic is based on

v0.23.0

Checklist before merging to `draft`

I have added a changelog
Git history is in acceptable state

yito88

Can we call write_tx_hash or other functions for replay protection without wl_storage.write_log? I mean wl_storage.write_tx_hash().
There are many callers with wl_storage.write_log or wl_storage.storage. They are a bit confusing because they are not clear when these data are persistent in the storage. (Some bugs happened before by writing data with wl_storage.storage directly before the block commit. I think that the direct write to the storage only for testing.

yito88 · 2023-10-12T15:25:19Z

apps/src/lib/node/ledger/storage/rocksdb.rs

 //!   - `pred`: predecessor values of the top-level keys of the same name
 //!     - `tx_queue`
 //!     - `next_epoch_min_start_height`
 //!     - `next_epoch_min_start_time`
+//!     - `replay_protection`


Is this stored in the state column family?

Ah no, this is a typo

… `Storage`

grarco · 2023-10-13T20:25:22Z

Can we call write_tx_hash or other functions for replay protection without wl_storage.write_log? I mean wl_storage.write_tx_hash().
There are many callers with wl_storage.write_log or wl_storage.storage. They are a bit confusing because they are not clear when these data are persistent in the storage. (Some bugs happened before by writing data with wl_storage.storage directly before the block commit. I think that the direct write to the storage only for testing.

So the keys for replay protection are meant to be written even in case the transaction failed, so the replay_protection field of WriteLog is treated as the block_write_log but I actually never directly write to storage. This is also because I need to write these keys during validation in process_proposal and prepare_proposal in which the storage is behind an immutable reference. I can reexpose the functions in WlStorage if you prefer, to avoid calling them directly from the write log

yito88 · 2023-10-16T08:56:40Z

Reexposing the functions in WlStorage is better, I think.
In my opinion, WlStorage callers should not choose the write log or the storage (all writes should be stored in the write log) except for the block commit and testing.

* origin/grarco/replay-protection-storage: Changelog #1977 Expose replay protection methods from `WlStorage` Reworks replay protection check Fixes check for replay protection keys Writes only one hash at a time for replay protection Removes wrapper hash when committed inner tx Renames `finalize_tx_hashes` Fixes unit tests Improves replay protection `WriteLog` API New field in `WriteLog` for replay protection changes Refactors replay protection helper functions Updates `DB` and `DBIter` traits for replay protection Removes replay protecion internal address and vp Renames replay protection storage key getter Removes hacky solution for replay protection merkle tree Writes replay protection keys under separate storage root Replay protection column family and related methods in `DB` trait and `Storage`

grarco force-pushed the grarco/replay-protection-storage branch 4 times, most recently from 895eaf2 to 9178c79 Compare October 11, 2023 09:41

grarco added a commit that referenced this pull request Oct 11, 2023

Changelog #1977

8cf0901

grarco force-pushed the grarco/replay-protection-storage branch from 9178c79 to 8cf0901 Compare October 11, 2023 10:01

grarco added a commit that referenced this pull request Oct 11, 2023

Changelog #1977

7d476fb

grarco force-pushed the grarco/replay-protection-storage branch from 8cf0901 to 7d476fb Compare October 11, 2023 10:02

grarco marked this pull request as ready for review October 11, 2023 10:03

grarco requested a review from yito88 October 11, 2023 10:03

yito88 reviewed Oct 12, 2023

View reviewed changes

grarco added 14 commits October 13, 2023 20:01

Replay protection column family and related methods in DB trait and…

2e407e4

… `Storage`

Writes replay protection keys under separate storage root

58b424e

Removes hacky solution for replay protection merkle tree

935e55f

Renames replay protection storage key getter

61724b4

Removes replay protecion internal address and vp

7caac7f

Updates DB and DBIter traits for replay protection

60f0014

Refactors replay protection helper functions

77dfccb

New field in WriteLog for replay protection changes

2eddc70

Improves replay protection WriteLog API

daeae3e

Fixes unit tests

2661d86

Renames finalize_tx_hashes

b2ec6a3

Removes wrapper hash when committed inner tx

4a7ec74

Writes only one hash at a time for replay protection

1e06990

Fixes check for replay protection keys

37a6602

grarco added a commit that referenced this pull request Oct 13, 2023

Changelog #1977

f77aee3

grarco force-pushed the grarco/replay-protection-storage branch from 7d476fb to f77aee3 Compare October 13, 2023 18:02

Reworks replay protection check

914b6f5

grarco added a commit that referenced this pull request Oct 13, 2023

Changelog #1977

1a00430

grarco force-pushed the grarco/replay-protection-storage branch from f77aee3 to 1a00430 Compare October 13, 2023 20:14

grarco added 2 commits October 16, 2023 15:45

Expose replay protection methods from WlStorage

a77cb5c

Changelog #1977

4fca277

grarco force-pushed the grarco/replay-protection-storage branch from 1a00430 to 4fca277 Compare October 16, 2023 13:45

yito88 approved these changes Oct 19, 2023

View reviewed changes

grarco added the ready for draft label Oct 19, 2023

grarco mentioned this pull request Oct 19, 2023

Draft #1980

Closed

grarco added a commit that referenced this pull request Oct 20, 2023

Merge branch 'grarco/replay-protection-storage' (#1977)

4dfa8bc

grarco added the merged to draft label Oct 20, 2023

cwgoes mentioned this pull request Oct 20, 2023

[tracking] Gas, fee, and replay protection readiness #2019

Closed

10 tasks

Fraccaman added a commit that referenced this pull request Oct 23, 2023

fixup! Merge branch 'grarco/replay-protection-storage' (#1977)

3750365

tzemanovic mentioned this pull request Oct 24, 2023

0.24.0 rc2 #2039

Merged

grarco merged commit 4fca277 into main Oct 24, 2023
11 of 14 checks passed

grarco deleted the grarco/replay-protection-storage branch October 24, 2023 11:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replay protection storage optimization #1977

Replay protection storage optimization #1977

grarco commented Oct 9, 2023 •

edited

Loading

yito88 left a comment

yito88 Oct 12, 2023

grarco Oct 13, 2023

grarco commented Oct 13, 2023

yito88 commented Oct 16, 2023

Replay protection storage optimization #1977

Replay protection storage optimization #1977

Conversation

grarco commented Oct 9, 2023 • edited Loading

Describe your changes

Indicate on which release or other PRs this topic is based on

Checklist before merging to draft

yito88 left a comment

Choose a reason for hiding this comment

yito88 Oct 12, 2023

Choose a reason for hiding this comment

grarco Oct 13, 2023

Choose a reason for hiding this comment

grarco commented Oct 13, 2023

yito88 commented Oct 16, 2023

grarco commented Oct 9, 2023 •

edited

Loading

Checklist before merging to `draft`