Spike: Event sourced persistence #963

ffakenz · 2023-06-30T19:49:16Z

Fixes #913

🏖️ The head logic now produces a list of events as part of its outcome during update.

🏖️ The node now applies a transition function to update the current state (in memory) while processing events.

🏖️ After every processed event, the node will persist it incrementally on disk.

🏖️ The node, upon start, will recover the head state from the persisted events.

🏖️ The below image summarize the transition function updateHeadState

---

🐻 Proposal: Removed the previous reference from ChainStateAt.

This makes head state events very lightweight (and can be improved even more by only persisting the current slot if there are no other changes on-chain after an observation).
By persisting head state events along with chain state events, we ensure atomicity.
This makes the rollback have to load all persisted events to compute the current chain state.
No changes to persisted events will take place after rollback takes place. ⚠️
We are expecting to address this as part of this story.

🐯 Proposal: Re-model emitSnapshot as a new internal effect/event process.

This splits the process in two parts: effect & event
The effect is being produced by the HeadLogic, during ReqTx and AckSn handling, and it carries the action of ReqEmitSn.
The event (EmitSn) is being produced by the Node during the new effect processing, and is expected to be handled by the HeadLogic only when in OpenState.
Because EmitSn is prepended to the queue, it is expected to be processed by the node immediately.
This model is consistent and cohesive with the rest of the code.

CHANGELOG updated or not needed
Documentation updated or not needed
Haddocks updated or not needed
No new TODOs introduced or explained herafter

github-actions · 2023-06-30T21:28:19Z

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2023-07-11 14:34:38.70670517 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	2212a4ee618434b9b2f366d7c330dbdfb5c7072e793a850fd0de6ddd	4294
νCommit	69e1ccf9ad73dc6d37a5bc8de5aec86f3c4c1710fe5fd334e0e16b18	2124
νHead	8ae095dca4d14a1b8edffb37faa6c84ec60340fbf389a62f027e0b76	9355
μHead	33642a45c7fbb955ce1704ef09229bb211bf9af9980530db28c6aafe*	4148

The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per Head.

Cost of Init Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4741	14.94	5.89	0.52
2	4947	15.37	5.98	0.54
3	5151	19.59	7.64	0.59
5	5562	24.37	9.45	0.66
10	6586	33.33	12.72	0.80
38	12328	99.69	37.75	1.77

Cost of Commit Transaction

This is using ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	596	14.89	5.70	0.34
2	787	19.66	7.73	0.40
3	971	24.75	9.88	0.46
5	1347	36.15	14.59	0.61
10	2283	71.73	28.85	1.04
13	2844	98.11	39.18	1.35

Cost of CollectCom Transaction

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	815	27.76	10.78	0.49
2	113	1134	43.50	17.03	0.67
3	170	1462	61.17	24.09	0.88
4	227	1774	82.17	32.52	1.13

Cost of Close Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	639	18.58	8.31	0.39
2	671	17.76	7.45	0.38
3	969	21.37	10.82	0.44
5	1307	24.16	13.33	0.50
10	2125	31.12	19.62	0.64
50	8725	87.17	69.99	1.74

Cost of Contest Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	675	24.34	10.47	0.45
2	842	26.49	12.01	0.49
3	1005	27.77	13.21	0.51
5	1335	31.63	16.12	0.58
10	2161	39.77	22.80	0.73
45	7944	99.74	70.74	1.82

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4854	22.40	9.38	0.61
2	5177	36.59	15.53	0.79
3	5497	53.87	23.04	0.99
4	5816	73.64	31.66	1.23
5	6138	95.99	41.44	1.49

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
5	0	0	4768	8.66	3.57	0.46
5	1	57	4800	10.06	4.39	0.47
5	5	284	4943	15.64	7.69	0.55
5	10	569	5123	22.61	11.82	0.64
5	20	1140	5485	36.56	20.07	0.83
5	30	1708	5844	50.52	28.33	1.02
5	40	2275	6202	64.49	36.60	1.21
5	50	2844	6556	78.46	44.87	1.40
5	65	3703	7104	99.42	57.28	1.68

github-actions · 2023-07-01T11:54:51Z

Test Results

342 tests ±0 336 ✔️ ±0 23m 25s ⏱️ - 1m 10s
114 suites ±0     6 💤 ±0
    6 files ±0     0 ❌ ±0

Results for commit 84dd48f. ± Comparison against base commit 003d05e.

♻️ This comment has been updated with latest results.

pgrange · 2023-07-11T08:04:53Z

hydra-node/exe/hydra-node/Main.hs

+          loadAll persistence >>= \case
+            [] -> do
              traceWith tracer CreatedState
              pure $ Idle IdleState{chainState = initialChainState}
-            Just headState -> do
+            events -> do
              traceWith tracer LoadedState
-              let paramsMismatch = checkParamsAgainstExistingState headState env
+              let
+                initialState = Idle IdleState{chainState = initialChainState}
+                headState = foldl' updateHeadState initialState events
+                paramsMismatch = checkParamsAgainstExistingState headState env


Should move to Node.hs

pgrange · 2023-07-11T08:12:10Z

hydra-cluster/test/Test/EndToEndSpec.hs

-                    )
-
-            -- save altered node state
+            let alice = deriveParty hydraSKey


Could re-write this test:

Start the node with one configuration and persistency

Initialize a head

Start the node with a different configuration and same persistency

…rdingly

…ead state This is in order to delay the computation to the next state after persisting the produced events. Also removed headId from most events types as it is not necessary. Derive instances for HeadStateEvent. Added RolledBack HeadStateEvent. Now the update function produces a list of events for the node to process.

…d persist all HeadStateEvents incrementally.

It needs the next head state in order to produce the SnapshotEmited event.

As it was depending on altering the persisted state to trigger the error. Now we need to generate a list of events that leads to some invalid state, that we can run with miss-matching configuration.

> Add track tx and untrack tx head state events.

- Remove previous from ChainStateAt - Rollback now requires to load all persisted events, in order to know from which chain state it should continue processing - Persisted events remain untouched

…rocess - add new Event 'OffChainEvent' + this will carry the event 'EmitSn' - add new Effect 'OffChainEffect' to 'HeadLogic' + this will carry the action 'ReqEmitSn' - update 'onOpenNetworkReqTx' and 'onOpenNetworkAckSn' to publish 'OffChainEffect RqEmitSn' on success. - update the 'Node.processEffect' function to handle the new effect and put the 'EmitSn' into the queue for it to be processed. - add matching to 'HeadLogic.update' function to publish 'SnapshotEmited', when in 'OpenState', we receive an 'EmitSn' action.

- Added new prependEvent handler to EventQueue. - This is used by the Node, when handling OffChainEffect, to EmitSn. - This allows EmitSn to be process immediately. - For that we needed to update the event ids of the rest of the enqueued events for succ.

- Ignore invalid events instead of throwing an error - Simplify AckSnConfirmed head state event

- This removes the dependency on HeadLogic in the Chain module.

pgrange

Awesome work on this spike @ffakenz , thank you.
We close as it's a spike and move forward with #913
We don't want to pick, at least at first, the changes about chainState persistency.

pgrange · 2023-07-11T15:00:04Z

hydra-node/src/Hydra/Chain/Direct/Handlers.hs

+  rollback tv point events = do
+    let maybeChainState =
+          events
+            & find
+              ( \ChainStateAt{recordedAt} ->
+                  case recordedAt of
+                    Just recordPoint | recordPoint <= point -> True
+                    _ -> False
+              )
+    -- TODO: using the same as defined in Direct module due to cyclic dependnecy.
+    let initialChainState = ChainStateAt{chainState = Idle, recordedAt = Nothing}
+    let rolledBack = fromMaybe initialChainState maybeChainState
+    -- REVIEW: what should we do with persisted events? prune? replace?


Would rather keep the previous state in memory and not explore the file content to do the rollback.

pgrange · 2023-07-11T15:01:59Z

hydra-node/src/Hydra/HeadLogic.hs

+          }
+  (_, RolledBack{rolledBackChainState}) ->
+    setChainState rolledBackChainState st
+  _ -> st


This function should be complete: an event should always be applicable. The decision to do something or not has already been taken in the other update function and now it's just about applying an event which already happened and can not be wrong.

pgrange · 2023-07-11T15:03:08Z

hydra-node/src/Hydra/HeadLogic.hs

@@ -368,7 +387,7 @@ instance Arbitrary (TxIdType tx) => Arbitrary (RequirementFailure tx) where
 data Outcome tx
  = NoOutcome
  | Effects {effects :: [Effect tx]}
-  | NewState {headState :: HeadState tx}
+  | NewState {events :: [HeadStateEvent tx]}


This type should only carry one event. If we really need several events, we can still Combined stuff.

We should name it differently also.

pgrange · 2023-07-11T15:04:41Z

hydra-node/src/Hydra/HeadLogic.hs

+  SeenSnapshot tx ->
+  Outcome tx
+emitSnapshot env params chs seenSnapshot =
+  case newSn env params chs of


We must challenge this function newSn because it's using an already update state and this has some nasty implications.

Also note that it means we would need to update the spec maybe.

pgrange · 2023-07-11T15:08:25Z

hydra-node/src/Hydra/HeadLogic.hs

@@ -382,6 +401,214 @@ deriving instance (IsTx tx, IsChainState tx) => FromJSON (Outcome tx)
 instance (IsTx tx, Arbitrary (ChainStateType tx)) => Arbitrary (Outcome tx) where
  arbitrary = genericArbitrary

+data HeadStateEvent tx


Could we use the events returned by the update function to create the effects and stop returning them?

ffakenz force-pushed the spike-event-sourcing branch 2 times, most recently from 93635b6 to 7ee7ab0 Compare June 30, 2023 21:47

ffakenz force-pushed the spike-event-sourcing branch 11 times, most recently from b940daf to 1fc3704 Compare July 9, 2023 11:49

ffakenz marked this pull request as ready for review July 9, 2023 12:16

ffakenz force-pushed the spike-event-sourcing branch 6 times, most recently from abbbc82 to a0d5f74 Compare July 10, 2023 17:54

pgrange reviewed Jul 11, 2023

View reviewed changes

ffakenz added 7 commits July 11, 2023 15:57

Draft head state events

2870754

Draft head state transition function and alter head state events acco…

1ea9775

…rdingly

Draft node changes to update the HeadState during event processing an…

e3a58a6

…d persist all HeadStateEvents incrementally.

Fix specs

a7b8788

Replace HydraNode Persistance for PersistenceIncremental

3477041

Add tmp fix for emitSnapshot

89283e3

It needs the next head state in order to produce the SnapshotEmited event.

ffakenz added 8 commits July 11, 2023 15:57

Minor fix to golden file

ea0823c

Fix e2e to detects misconfiguration

d64c6c8

As it was depending on altering the persisted state to trigger the error. Now we need to generate a list of events that leads to some invalid state, that we can run with miss-matching configuration.

Fix branch after rebase

5bddb3d

> Add track tx and untrack tx head state events.

Draft proposal to handle chain-state evolution

a3b5cce

- Remove previous from ChainStateAt - Rollback now requires to load all persisted events, in order to know from which chain state it should continue processing - Persisted events remain untouched

Fix emitSnapshot proposal

6023faa

- Added new prependEvent handler to EventQueue. - This is used by the Node, when handling OffChainEffect, to EmitSn. - This allows EmitSn to be process immediately. - For that we needed to update the event ids of the rest of the enqueued events for succ.

Add first round of improvements

530ed86

- Ignore invalid events instead of throwing an error - Simplify AckSnConfirmed head state event

Introduce a projection view for ChainStateAt

84dd48f

- This removes the dependency on HeadLogic in the Chain module.

ch1bo force-pushed the spike-event-sourcing branch from a0d5f74 to 84dd48f Compare July 11, 2023 13:58

pgrange suggested changes Jul 11, 2023

View reviewed changes

pgrange closed this Jul 11, 2023

ffakenz deleted the spike-event-sourcing branch August 6, 2023 08:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spike: Event sourced persistence #963

Spike: Event sourced persistence #963

ffakenz commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jul 1, 2023 •

edited

Loading

pgrange Jul 11, 2023

pgrange Jul 11, 2023

pgrange left a comment

pgrange Jul 11, 2023

pgrange Jul 11, 2023

pgrange Jul 11, 2023

pgrange Jul 11, 2023

pgrange Jul 11, 2023

Spike: Event sourced persistence #963

Spike: Event sourced persistence #963

Conversation

ffakenz commented Jun 30, 2023 • edited Loading

github-actions bot commented Jun 30, 2023 • edited Loading

Transactions Costs

Script summary

Cost of Init Transaction

Cost of Commit Transaction

Cost of CollectCom Transaction

Cost of Close Transaction

Cost of Contest Transaction

Cost of Abort Transaction

Cost of FanOut Transaction

github-actions bot commented Jul 1, 2023 • edited Loading

Test Results

pgrange Jul 11, 2023

Choose a reason for hiding this comment

pgrange Jul 11, 2023

Choose a reason for hiding this comment

pgrange left a comment

Choose a reason for hiding this comment

pgrange Jul 11, 2023

Choose a reason for hiding this comment

pgrange Jul 11, 2023

Choose a reason for hiding this comment

pgrange Jul 11, 2023

Choose a reason for hiding this comment

pgrange Jul 11, 2023

Choose a reason for hiding this comment

pgrange Jul 11, 2023

Choose a reason for hiding this comment

ffakenz commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jul 1, 2023 •

edited

Loading