[db]log mptrie node #3634

millken · 2022-09-09T07:12:06Z

Description

record mptrie node infomation to postgres DB.

Node Type	Node Key	Node Path	Node Children
`b`ranchNode	☐	☐	☑
`e`xtensionNode	☐	☑	☐
`l`eafNode	☑	☐	☐

test=# select * from log_mptrie order by id asc limit 10;
 id | node_type |                             node_key                             |                          node_path                           | node_children |        last_visited        
----+-----------+------------------------------------------------------------------+--------------------------------------------------------------+---------------+----------------------------
  1 | b         |                                                                  |                                                              |               | 2022-10-11 15:13:52.755372 
  7 | l         | 0000000000000000000000000000000000000000000000000000000000000000 |                                                              |               | 2022-10-11 15:13:00.320674 
  8 | b         |                                                                  |                                                              | 00            | 2022-10-11 15:13:00.349906 
 23 | l         | 0000000000000000000000000000000000000000000000000000000000000002 |                                                              |               | 2022-10-11 15:13:00.322241 
 24 | b         |                                                                  |                                                              | 0002          | 2022-10-11 15:13:00.339253 
 25 | e         |                                                                  | 000000000000000000000000000000000000000000000000000000000000 |               | 2022-10-11 15:13:00.483378 
 36 | l         | 0000000000000000000000000000000000000000000000000000000000000003 |                                                              |               | 2022-10-11 15:13:00.340525 
 37 | b         |                                                                  |                                                              | 000203        | 2022-10-11 15:13:00.480425 
 44 | l         | 17ef568e3e12ab5b9c7254a8d58478811de00f9e6eb34345acd53bf8fd09d3ec |                                                              |               | 2022-10-11 15:13:52.794232 
 45 | b         |                                                                  |                                                              | 0017          | 2022-10-11 15:13:00.4656   
(10 rows)

Fixes #3204

Type of change

Please delete options that are not relevant.

[] Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
[] Code refactor or improvement
[] Breaking change (fix or feature that would cause a new or changed behavior of existing functionality)
[] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

make test
[] fullsync
[] Other test (please specify)

Test Configuration:

Firmware version:
Hardware:
Toolchain:
SDK:

Checklist:

[] My code follows the style guidelines of this project
[] I have performed a self-review of my code
[] I have commented my code, particularly in hard-to-understand areas
[] I have made corresponding changes to the documentation
[] My changes generate no new warnings
[] I have added tests that prove my fix is effective or that my feature works
[] New and existing unit tests pass locally with my changes
[] Any dependent changes have been merged and published in downstream modules

codecov · 2022-09-09T07:29:26Z

Codecov Report

Merging #3634 (1b1944d) into master (263ff82) will increase coverage by 0.60%.
The diff coverage is 81.32%.

@@            Coverage Diff             @@
##           master    #3634      +/-   ##
==========================================
+ Coverage   74.47%   75.07%   +0.60%     
==========================================
  Files         269      299      +30     
  Lines       23925    25202    +1277     
==========================================
+ Hits        17818    18921    +1103     
- Misses       5174     5302     +128     
- Partials      933      979      +46

Impacted Files	Coverage Δ
action/actctx.go	`90.62% <ø> (+6.25%)`	⬆️
action/action.go	`82.35% <ø> (ø)`
action/action_deserializer.go	`57.14% <ø> (ø)`
action/candidate_update.go	`88.88% <0.00%> (-4.14%)`	⬇️
action/consignment_transfer.go	`90.41% <ø> (ø)`
action/const.go	`100.00% <ø> (ø)`
action/execution.go	`83.78% <ø> (ø)`
action/grantreward.go	`68.57% <ø> (ø)`
action/protocol/account/protocol.go	`86.90% <ø> (ø)`
action/protocol/account/transfer.go	`84.61% <ø> (ø)`
... and 276 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Liuhaai · 2022-09-09T17:27:45Z

how does this pr solve #3204 ?

This reverts commit dc8b725.

This reverts commit fd984d8.

* check nil for response.EpochData (#3642) * remove unused db/sql (#3645) * [ioctl] Build Read unittest (#3635) * build Read unittest * fix code smells * remove redundant argument * fix error * remove redundant code * remove redundant space * delete useless code Co-authored-by: huof6890 <[email protected]> * [pkg] rotate log file (#3638) * add lumberjack * add lograte conf in Dockerfile * del space line * [ioctl] Build did geturi command line into new ioctl (#3650) * [ioctl] build did geturi command line into new ioctl * add test data * move abi reader to init * build unittest to cover the modification * move global var _didABI to did.go * move abi reader to did * alter did string with did prefix * rename did string Co-authored-by: huof6890 <[email protected]> * [ioctl] build did gethash command line into new ioctl (#3639) * [ioctl] build did gethash command line into new ioctl * remove redundant code * build unittest to cover the modification * remove redundant args * add command description * modify test value * remove func encodeGet * remove func encodeGet * build unittest to cover the modification * reset default gas limit * use fault address from identityset * remove redundant space * update command name * move abi reader to init * move abi reader to did * alter did string with did prefix * rename did string Co-authored-by: huofei <[email protected]> * [factory] add error details (#3657) Co-authored-by: huofei <[email protected]> Co-authored-by: Jeremy Chou <[email protected]> Co-authored-by: CoderZhi <[email protected]>

sonarqubecloud · 2022-10-31T08:35:51Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
5.0% Duplication

CoderZhi · 2022-11-29T23:07:12Z

db/trie/mptrie/lognode.go

+}
+
+// ParseNodeEvent parse the node event
+func ParseNodeEvent(b []byte) (NodeEvent, error) {


move to unit test and rename to private function

CoderZhi · 2022-11-29T23:12:28Z

db/trie/mptrie/lognode.go

+	return logFile.Close()
+}
+
+func logNode(n node) error {


parameters:

node type, leaf, extension, and branch

action type, e.g., read and write, or search, delete, and new

hashvalue, creating via cacheNode.Hash()

create hashvalue inside of logNode function. Other than that, it looks good.

CoderZhi · 2022-11-30T18:28:42Z

server/main.go

+		defer func() {
+			if err = mptrie.CloseLogDB(); err != nil {
+				log.L().Error("Failed to close mptrie log DB.", zap.Error(err))
+			}
+		}()


this function will be called right away. It seems that the PR has not been fully tested.

has been tested OK

good to know. I have wrong understanding of defer

CoderZhi · 2022-12-31T08:35:11Z

db/trie/mptrie/extensionnode.go

@@ -55,15 +54,14 @@ func newExtensionNodeFromProtoPb(pb *triepb.ExtendPb, hashVal []byte) *extension
 		child: newHashNode(pb.Value),
 	}
 	e.cacheNode.serializable = e
-	if err := logNode(_nodeTypeExtension, _actionTypeNew, hashVal, e); err != nil {
+	if err := logNode(_nodeTypeExtension, _actionTypeNew, e, nil); err != nil {


not working here

CoderZhi · 2022-12-31T08:35:23Z

db/trie/mptrie/branchnode.go

@@ -93,7 +91,7 @@ func newBranchNodeFromProtoPb(pb *triepb.BranchPb, hashVal []byte) *branchNode {
 	}
 	bnode.indices = NewSortedList(bnode.children)
 	bnode.cacheNode.serializable = bnode
-	if err := logNode(_nodeTypeBranch, _actionTypeNew, hashVal, bnode); err != nil {
+	if err := logNode(_nodeTypeBranch, _actionTypeNew, bnode, nil); err != nil {


not working here

CoderZhi · 2022-12-31T08:35:56Z

db/trie/mptrie/leafnode.go

@@ -55,7 +54,7 @@ func newLeafNodeFromProtoPb(pb *triepb.LeafPb, hashVal []byte) *leafNode {
 		value: pb.Value,
 	}
 	l.cacheNode.serializable = l
-	if err := logNode(_nodeTypeLeaf, _actionTypeNew, hashVal, l); err != nil {
+	if err := logNode(_nodeTypeLeaf, _actionTypeNew, l, nil); err != nil {


not working here

I thought so too at first.
see the following code, first, it will check hashVal, only use cli if hashVal is not set.

iotex-core/db/trie/mptrie/cachenode.go

Lines 25 to 32 in 715ccb7

func (cn *cacheNode) hash(cli client, flush bool) ([]byte, error) {

if len(cn.hashVal) != 0 {

return cn.hashVal, nil

}

if cli == nil {

return []byte{}, errors.New("client cannot be nil")

}

pb, err := cn.proto(cli, flush)

dustinxie · 2023-01-13T04:58:24Z

db/trie/mptrie/lognode_test.go

+		}
+		if event.KeyLen != test.event.KeyLen {
+			t.Errorf("unexpected key length %v", event.KeyLen)
+		}


can we use same test code style? like

require.NoError(err) require.True(event.NodeType == test.event.NodeType)

sonarqubecloud · 2023-01-13T05:53:00Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
3.5% Duplication

Liuhaai · 2023-01-13T23:39:32Z

db/trie/mptrie/branchnode.go

@@ -85,6 +91,9 @@ func newBranchNodeFromProtoPb(pb *triepb.BranchPb, hashVal []byte) *branchNode {
 	}
 	bnode.indices = NewSortedList(bnode.children)
 	bnode.cacheNode.serializable = bnode
+	if err := logNode(_nodeTypeBranch, _actionTypeNew, bnode, nil); err != nil {
+		panic(err)


why is panic used when logging fails?

newBranchNodeFromProtoPb does not handle errors, directly returns *branchNode, logNode should not fail here either

Liuhaai · 2023-01-13T23:42:22Z

db/trie/mptrie/branchnode.go

@@ -69,6 +72,9 @@ func newRootBranchNode(cli client, children map[byte]node, indices *SortedList,
 			}
 		}
 	}
+	if err := logNode(_nodeTypeBranch, _actionTypeNew, bnode, cli); err != nil {


why is logging used at the end of function instead of at the head?

Wait for the bnode Initialized

Liuhaai · 2023-01-13T23:43:10Z

db/trie/mptrie/lognode.go

@@ -0,0 +1,127 @@
+package mptrie


IMO, the name lognode.go is misleading. log.go is a better choice

Liuhaai · 2023-01-13T23:45:27Z

db/trie/mptrie/lognode.go

+type actionType byte
+
+const (
+	_nodeTypeLeaf      nodeType = 'l'


Could a byte array [10]byte be used here? The meaning of the data as a byte is hard to read

Liuhaai · 2023-01-13T23:49:14Z

db/trie/mptrie/lognode.go

+		return err
+	}
+	// write event body
+	_, err = logWriter.Write(event.Bytes())


How to read the binary data in the db? Do we have the tool or component to print the data?

A byte is used to save the size of the binrary. Binary itself is not readable and needs to be processed by tool. In the lognode_test.go, there is a parseNodeEvent that can parse the binrary.

How the tools are provided needs to be discussed (print, database)

add collector package to monitor node cpu/loadavg/memory usage

fd984d8

millken requested a review from a team as a code owner September 9, 2022 07:12

fix test

dc8b725

millken marked this pull request as draft September 13, 2022 02:18

millken added 5 commits September 13, 2022 11:53

Revert "fix test"

ee087b0

This reverts commit dc8b725.

Revert "add collector package to monitor node cpu/loadavg/memory usage"

fc7ddd6

This reverts commit fd984d8.

Merge branch 'master' into issue3204

3a91c4c

add event for node

7cd23c4

add event for node

0acee52

millken changed the title ~~add collector package to monitor node cpu/loadavg/memory usage~~ events for node Sep 13, 2022

millken and others added 5 commits September 13, 2022 20:41

update event

549a8ec

saving

76e00cf

log mptrie to db

af14413

Merge branch 'iotexproject:master' into issue3204

6f736e4

millken changed the title ~~events for node~~ [db]log mptrie node Oct 11, 2022

millken marked this pull request as ready for review October 11, 2022 07:38

millken added 5 commits October 11, 2022 22:16

fix ci test

12fd4cf

Merge branch 'master' into issue3204

c7bbe1e

rewrite log node event, change to append-only Log

c9b5792

update test

23622ed

update test

8166a5d

CoderZhi reviewed Nov 30, 2022

View reviewed changes

millken added 4 commits December 1, 2022 13:57

Merge branch 'master' into issue3204

6521e42

fix comments

a575201

fix comments

33ed183

fix tests

99efe1a

Merge branch 'master' into issue3204

d485731

millken requested a review from dustinxie as a code owner December 29, 2022 02:44

create hashval inside logNode func

c501416

CoderZhi reviewed Dec 31, 2022

View reviewed changes

Merge branch 'master' into issue3204

715ccb7

CoderZhi approved these changes Jan 5, 2023

View reviewed changes

dustinxie reviewed Jan 13, 2023

View reviewed changes

dustinxie approved these changes Jan 13, 2023

View reviewed changes

millken added 2 commits January 13, 2023 13:48

Merge branch 'master' into issue3204

070a33c

fix comments

1b1944d

millken merged commit dab026e into iotexproject:master Jan 13, 2023

Liuhaai reviewed Jan 13, 2023

View reviewed changes

millken mentioned this pull request Jan 17, 2023

[config]Fix missing config MptrieLogPath #3769

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[db]log mptrie node #3634

[db]log mptrie node #3634

millken commented Sep 9, 2022 •

edited

Loading

codecov bot commented Sep 9, 2022 •

edited

Loading

Liuhaai commented Sep 9, 2022

sonarqubecloud bot commented Oct 31, 2022

CoderZhi Nov 29, 2022

CoderZhi Nov 29, 2022

millken Dec 1, 2022

CoderZhi Dec 29, 2022

millken Dec 29, 2022

CoderZhi Nov 30, 2022

millken Dec 1, 2022

CoderZhi Dec 29, 2022

CoderZhi Dec 31, 2022

CoderZhi Dec 31, 2022

CoderZhi Dec 31, 2022

millken Jan 3, 2023 •

edited

Loading

dustinxie Jan 13, 2023

sonarqubecloud bot commented Jan 13, 2023

Liuhaai Jan 13, 2023

millken Jan 14, 2023

Liuhaai Jan 13, 2023

millken Jan 14, 2023

Liuhaai Jan 13, 2023

Liuhaai Jan 13, 2023 •

edited

Loading

Liuhaai Jan 13, 2023 •

edited

Loading

millken Jan 14, 2023

	func (cn *cacheNode) hash(cli client, flush bool) ([]byte, error) {
	if len(cn.hashVal) != 0 {
	return cn.hashVal, nil
	}
	if cli == nil {
	return []byte{}, errors.New("client cannot be nil")
	}
	pb, err := cn.proto(cli, flush)

[db]log mptrie node #3634

[db]log mptrie node #3634

Conversation

millken commented Sep 9, 2022 • edited Loading

Description

Type of change

How Has This Been Tested?

Checklist:

codecov bot commented Sep 9, 2022 • edited Loading

Codecov Report

Liuhaai commented Sep 9, 2022

sonarqubecloud bot commented Oct 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

millken Jan 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Liuhaai Jan 13, 2023 • edited Loading

Choose a reason for hiding this comment

Liuhaai Jan 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

millken commented Sep 9, 2022 •

edited

Loading

codecov bot commented Sep 9, 2022 •

edited

Loading

millken Jan 3, 2023 •

edited

Loading

Liuhaai Jan 13, 2023 •

edited

Loading

Liuhaai Jan 13, 2023 •

edited

Loading