Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[db]log mptrie node #3634

Merged
merged 26 commits into from
Jan 13, 2023
Merged

[db]log mptrie node #3634

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
fd984d8
add collector package to monitor node cpu/loadavg/memory usage
millken Sep 9, 2022
dc8b725
fix test
millken Sep 9, 2022
ee087b0
Revert "fix test"
millken Sep 13, 2022
fc7ddd6
Revert "add collector package to monitor node cpu/loadavg/memory usage"
millken Sep 13, 2022
3a91c4c
Merge branch 'master' into issue3204
millken Sep 13, 2022
7cd23c4
add event for node
millken Sep 13, 2022
0acee52
add event for node
millken Sep 13, 2022
549a8ec
update event
millken Sep 13, 2022
76e00cf
saving
millken Sep 27, 2022
b54aaca
merge master (#3)
millken Oct 10, 2022
af14413
log mptrie to db
millken Oct 11, 2022
6f736e4
Merge branch 'iotexproject:master' into issue3204
millken Oct 11, 2022
12fd4cf
fix ci test
millken Oct 11, 2022
c7bbe1e
Merge branch 'master' into issue3204
millken Oct 31, 2022
c9b5792
rewrite log node event, change to append-only Log
millken Oct 31, 2022
23622ed
update test
millken Oct 31, 2022
8166a5d
update test
millken Oct 31, 2022
6521e42
Merge branch 'master' into issue3204
millken Dec 1, 2022
a575201
fix comments
millken Dec 1, 2022
33ed183
fix comments
millken Dec 1, 2022
99efe1a
fix tests
millken Dec 1, 2022
d485731
Merge branch 'master' into issue3204
millken Dec 29, 2022
c501416
create hashval inside logNode func
millken Dec 29, 2022
715ccb7
Merge branch 'master' into issue3204
millken Jan 3, 2023
070a33c
Merge branch 'master' into issue3204
millken Jan 13, 2023
1b1944d
fix comments
millken Jan 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 15 additions & 17 deletions config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,23 +114,21 @@ type (

// Config is the root config struct, each package's config should be put as its sub struct
Config struct {
Plugins map[int]interface{} `ymal:"plugins"`
Network p2p.Config `yaml:"network"`
Chain blockchain.Config `yaml:"chain"`
ActPool actpool.Config `yaml:"actPool"`
Consensus Consensus `yaml:"consensus"`
DardanellesUpgrade DardanellesUpgrade `yaml:"dardanellesUpgrade"`
BlockSync BlockSync `yaml:"blockSync"`
Dispatcher dispatcher.Config `yaml:"dispatcher"`
API API `yaml:"api"`
System System `yaml:"system"`
DB db.Config `yaml:"db"`
Indexer blockindex.Config `yaml:"indexer"`
Log log.GlobalConfig `yaml:"log"`
SubLogs map[string]log.GlobalConfig `yaml:"subLogs"`
Genesis genesis.Genesis `yaml:"genesis"`
// MptrieLogPath is the path to store mptrie logs
MptrieLogPath string `yaml:"mptrieLogPath"`
Plugins map[int]interface{} `ymal:"plugins"`
Network p2p.Config `yaml:"network"`
Chain blockchain.Config `yaml:"chain"`
ActPool actpool.Config `yaml:"actPool"`
Consensus consensus.Config `yaml:"consensus"`
DardanellesUpgrade consensusfsm.DardanellesUpgrade `yaml:"dardanellesUpgrade"`
BlockSync blocksync.Config `yaml:"blockSync"`
Dispatcher dispatcher.Config `yaml:"dispatcher"`
API api.Config `yaml:"api"`
System System `yaml:"system"`
DB db.Config `yaml:"db"`
Indexer blockindex.Config `yaml:"indexer"`
Log log.GlobalConfig `yaml:"log"`
SubLogs map[string]log.GlobalConfig `yaml:"subLogs"`
Genesis genesis.Genesis `yaml:"genesis"`
}

// Validate is the interface of validating the config
Expand Down
17 changes: 6 additions & 11 deletions db/trie/mptrie/branchnode.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,7 @@ func newBranchNode(
}
}
}
hashVal, _ := bnode.cacheNode.Hash(cli)
if err := logNode(_nodeTypeBranch, _actionTypeNew, hashVal, bnode); err != nil {
if err := logNode(_nodeTypeBranch, _actionTypeNew, bnode, cli); err != nil {
return nil, err
}
return bnode, nil
Expand All @@ -73,8 +72,7 @@ func newRootBranchNode(cli client, children map[byte]node, indices *SortedList,
}
}
}
hashVal, _ := bnode.cacheNode.Hash(cli)
if err := logNode(_nodeTypeBranch, _actionTypeNew, hashVal, bnode); err != nil {
if err := logNode(_nodeTypeBranch, _actionTypeNew, bnode, cli); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is logging used at the end of function instead of at the head?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait for the bnode Initialized

return nil, err
}
return bnode, nil
Expand All @@ -93,7 +91,7 @@ func newBranchNodeFromProtoPb(pb *triepb.BranchPb, hashVal []byte) *branchNode {
}
bnode.indices = NewSortedList(bnode.children)
bnode.cacheNode.serializable = bnode
if err := logNode(_nodeTypeBranch, _actionTypeNew, hashVal, bnode); err != nil {
if err := logNode(_nodeTypeBranch, _actionTypeNew, bnode, nil); err != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not working here

panic(err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is panic used when logging fails?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newBranchNodeFromProtoPb does not handle errors, directly returns *branchNode, logNode should not fail here either

}
return bnode
Expand All @@ -112,8 +110,7 @@ func (b *branchNode) Children() []node {
}

func (b *branchNode) Delete(cli client, key keyType, offset uint8) (node, error) {
hashVal, _ := b.cacheNode.Hash(cli)
if err := logNode(_nodeTypeBranch, _actionTypeDelete, hashVal, b); err != nil {
if err := logNode(_nodeTypeBranch, _actionTypeDelete, b, cli); err != nil {
return nil, err
}
offsetKey := key[offset]
Expand Down Expand Up @@ -169,8 +166,7 @@ func (b *branchNode) Delete(cli client, key keyType, offset uint8) (node, error)
}

func (b *branchNode) Upsert(cli client, key keyType, offset uint8, value []byte) (node, error) {
hashVal, _ := b.cacheNode.Hash(cli)
if err := logNode(_nodeTypeBranch, _actionTypeUpsert, hashVal, b); err != nil {
if err := logNode(_nodeTypeBranch, _actionTypeUpsert, b, cli); err != nil {
return nil, err
}
var newChild node
Expand All @@ -190,8 +186,7 @@ func (b *branchNode) Upsert(cli client, key keyType, offset uint8, value []byte)
}

func (b *branchNode) Search(cli client, key keyType, offset uint8) (node, error) {
hashVal, _ := b.cacheNode.Hash(cli)
if err := logNode(_nodeTypeBranch, _actionTypeSearch, hashVal, b); err != nil {
if err := logNode(_nodeTypeBranch, _actionTypeSearch, b, cli); err != nil {
return nil, err
}
child, err := b.child(key[offset])
Expand Down
14 changes: 5 additions & 9 deletions db/trie/mptrie/extensionnode.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,7 @@ func newExtensionNode(
return nil, err
}
}
hashVal, _ := e.cacheNode.Hash(cli)
if err := logNode(_nodeTypeExtension, _actionTypeNew, hashVal, e); err != nil {
if err := logNode(_nodeTypeExtension, _actionTypeNew, e, cli); err != nil {
return nil, err
}
return e, nil
Expand All @@ -55,15 +54,14 @@ func newExtensionNodeFromProtoPb(pb *triepb.ExtendPb, hashVal []byte) *extension
child: newHashNode(pb.Value),
}
e.cacheNode.serializable = e
if err := logNode(_nodeTypeExtension, _actionTypeNew, hashVal, e); err != nil {
if err := logNode(_nodeTypeExtension, _actionTypeNew, e, nil); err != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not working here

panic(err)
}
return e
}

func (e *extensionNode) Delete(cli client, key keyType, offset uint8) (node, error) {
hashVal, _ := e.cacheNode.Hash(cli)
if err := logNode(_nodeTypeExtension, _actionTypeDelete, hashVal, e); err != nil {
if err := logNode(_nodeTypeExtension, _actionTypeDelete, e, cli); err != nil {
return nil, err
}
matched := e.commonPrefixLength(key[offset:])
Expand Down Expand Up @@ -96,8 +94,7 @@ func (e *extensionNode) Delete(cli client, key keyType, offset uint8) (node, err
}

func (e *extensionNode) Upsert(cli client, key keyType, offset uint8, value []byte) (node, error) {
hashVal, _ := e.cacheNode.Hash(cli)
if err := logNode(_nodeTypeExtension, _actionTypeUpsert, hashVal, e); err != nil {
if err := logNode(_nodeTypeExtension, _actionTypeUpsert, e, cli); err != nil {
return nil, err
}
matched := e.commonPrefixLength(key[offset:])
Expand Down Expand Up @@ -135,8 +132,7 @@ func (e *extensionNode) Upsert(cli client, key keyType, offset uint8, value []by
}

func (e *extensionNode) Search(cli client, key keyType, offset uint8) (node, error) {
hashVal, _ := e.cacheNode.Hash(cli)
if err := logNode(_nodeTypeExtension, _actionTypeSearch, hashVal, e); err != nil {
if err := logNode(_nodeTypeExtension, _actionTypeSearch, e, cli); err != nil {
return nil, err
}
matched := e.commonPrefixLength(key[offset:])
Expand Down
14 changes: 5 additions & 9 deletions db/trie/mptrie/leafnode.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,7 @@ func newLeafNode(
return nil, err
}
}
hashVal, _ := l.cacheNode.Hash(cli)
if err := logNode(_nodeTypeLeaf, _actionTypeNew, hashVal, l); err != nil {
if err := logNode(_nodeTypeLeaf, _actionTypeNew, l, cli); err != nil {
return nil, err
}
return l, nil
Expand All @@ -55,7 +54,7 @@ func newLeafNodeFromProtoPb(pb *triepb.LeafPb, hashVal []byte) *leafNode {
value: pb.Value,
}
l.cacheNode.serializable = l
if err := logNode(_nodeTypeLeaf, _actionTypeNew, hashVal, l); err != nil {
if err := logNode(_nodeTypeLeaf, _actionTypeNew, l, nil); err != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not working here

Copy link
Contributor Author

@millken millken Jan 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought so too at first.
see the following code, first, it will check hashVal, only use cli if hashVal is not set.

iotex-core/db/trie/mptrie/cachenode.go

Lines 25 to 32 in 715ccb7

func (cn *cacheNode) hash(cli client, flush bool) ([]byte, error) {
if len(cn.hashVal) != 0 {
return cn.hashVal, nil
}
if cli == nil {
return []byte{}, errors.New("client cannot be nil")
}
pb, err := cn.proto(cli, flush)

panic(err)
}
return l
Expand All @@ -70,8 +69,7 @@ func (l *leafNode) Value() []byte {
}

func (l *leafNode) Delete(cli client, key keyType, offset uint8) (node, error) {
hashVal, _ := l.cacheNode.Hash(cli)
if err := logNode(_nodeTypeLeaf, _actionTypeDelete, hashVal, l); err != nil {
if err := logNode(_nodeTypeLeaf, _actionTypeDelete, l, cli); err != nil {
return nil, err
}
if !bytes.Equal(l.key[offset:], key[offset:]) {
Expand All @@ -81,8 +79,7 @@ func (l *leafNode) Delete(cli client, key keyType, offset uint8) (node, error) {
}

func (l *leafNode) Upsert(cli client, key keyType, offset uint8, value []byte) (node, error) {
hashVal, _ := l.cacheNode.Hash(cli)
if err := logNode(_nodeTypeLeaf, _actionTypeUpsert, hashVal, l); err != nil {
if err := logNode(_nodeTypeLeaf, _actionTypeUpsert, l, cli); err != nil {
return nil, err
}
matched := commonPrefixLength(l.key[offset:], key[offset:])
Expand Down Expand Up @@ -122,8 +119,7 @@ func (l *leafNode) Upsert(cli client, key keyType, offset uint8, value []byte) (
}

func (l *leafNode) Search(cli client, key keyType, offset uint8) (node, error) {
hashVal, _ := l.cacheNode.Hash(cli)
if err := logNode(_nodeTypeLeaf, _actionTypeSearch, hashVal, l); err != nil {
if err := logNode(_nodeTypeLeaf, _actionTypeSearch, l, cli); err != nil {
return nil, err
}
if !bytes.Equal(l.key[offset:], key[offset:]) {
Expand Down
13 changes: 8 additions & 5 deletions db/trie/mptrie/lognode.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,11 @@ func CloseLogDB() error {
return logFile.Close()
}

func logNode(nt nodeType, at actionType, hashvalue []byte, n node) error {
func logNode(nt nodeType, at actionType, n node, cli client) error {
if !enabledLogMptrie {
return nil
}
nodeKey, nodePath, nodeChildren, err := parseNode(n)
nodeKey, nodePath, nodeChildren, hashvalue, err := parseNode(n, cli)
if err != nil {
return err
}
Expand All @@ -100,23 +100,26 @@ func logNode(nt nodeType, at actionType, hashvalue []byte, n node) error {
HashLen: uint8(len(hashvalue)),
HashVal: hashvalue,
}
// write events length
// write event length
if err = logWriter.WriteByte(byte(len(event.Bytes()))); err != nil {
return err
}
// write events body
// write event body
_, err = logWriter.Write(event.Bytes())
Copy link
Member

@Liuhaai Liuhaai Jan 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How to read the binary data in the db? Do we have the tool or component to print the data?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A byte is used to save the size of the binrary. Binary itself is not readable and needs to be processed by tool. In the lognode_test.go, there is a parseNodeEvent that can parse the binrary.

How the tools are provided needs to be discussed (print, database)

return err
}

func parseNode(n node) (nodeKey []byte, nodePath []byte, nodeChildren []byte, err error) {
func parseNode(n node, cli client) (nodeKey, nodePath, nodeChildren, hashvalue []byte, err error) {
switch n := n.(type) {
case *leafNode:
nodeKey = n.key
hashvalue, err = n.cacheNode.Hash(cli)
case *extensionNode:
nodePath = n.path
hashvalue, err = n.cacheNode.Hash(cli)
case *branchNode:
nodeChildren = n.indices.List()
hashvalue, err = n.cacheNode.Hash(cli)
default:
err = errors.Errorf("unknown node type %T", n)
}
Expand Down