Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add etcd compact #18

Merged
merged 3 commits into from
May 6, 2022
Merged

add etcd compact #18

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
63daabd
feat: add etcd compact
lukepark327 May 6, 2022
78e7ad9
add etcdCompactWindow
lukepark327 May 6, 2022
73062f1
feat: apply compact window && update check logic
lukepark327 May 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion metadium/admin.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ var (
ErrNotRunning = errors.New("not running")
ErrAlreadyRunning = errors.New("already running")
ErrInvalidEnode = errors.New("invalid enode")

etcdCompactFrequency = int64(100)
)

func (n *metaNode) eq(m *metaNode) bool {
Expand Down Expand Up @@ -1084,12 +1086,22 @@ func LogBlock(height int64, hash common.Hash) {
}

tstart := time.Now()
if err := admin.etcdPut("metadium-work", string(work)); err != nil {
rev, err := admin.etcdPut("metadium-work", string(work))
if err != nil {
log.Error("Metadium - failed to log the latest block",
"height", height, "hash", hash, "took", time.Since(tstart))
} else {
log.Info("Metadium - logged the latest block",
"height", height, "hash", hash, "took", time.Since(tstart))

if (rev%etcdCompactFrequency == 0) && (rev > 100) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't "100" be replaced by etcdCompactFrequency? I.e.

if rev%etcdCompactFreqency == 0 && rev > etcdCompactFreqency {

Copy link
Author

@lukepark327 lukepark327 May 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I added distinct var, named etcdCompactWindow, for more generalization and flexibility.
etcdCompactWindow is set to 100 as default.

78e7ad9

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And, I updated codes to apply concept of compact window. Please check the following commits: 73062f1

go func() {
if err := admin.etcdCompact(rev); err != nil {
log.Error("Metadium - failed to compact",
"rev", rev, "took", time.Since(tstart))
}
}()
}
}

admin.blocksMined++
Expand Down
23 changes: 19 additions & 4 deletions metadium/etcdutil.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"strings"
"time"

"github.com/coreos/etcd/clientv3"
"github.com/coreos/etcd/embed"
"github.com/coreos/etcd/etcdserver/api/membership"
"github.com/coreos/etcd/etcdserver/api/v3client"
Expand Down Expand Up @@ -442,16 +443,16 @@ func (ma *metaAdmin) etcdLeader(locked bool) (uint64, *metaNode) {
return 0, nil
}

func (ma *metaAdmin) etcdPut(key, value string) error {
func (ma *metaAdmin) etcdPut(key, value string) (int64, error) {
if !ma.etcdIsRunning() {
return ErrNotRunning
return 0, ErrNotRunning
}

ctx, cancel := context.WithTimeout(context.Background(),
ma.etcd.Server.Cfg.ReqTimeout())
defer cancel()
_, err := ma.etcdCli.Put(ctx, key, value)
return err
resp, err := ma.etcdCli.Put(ctx, key, value)
return resp.Header.Revision, err
}

func (ma *metaAdmin) etcdGet(key string) (string, error) {
Expand Down Expand Up @@ -487,6 +488,20 @@ func (ma *metaAdmin) etcdDelete(key string) error {
return err
}

func (ma *metaAdmin) etcdCompact(rev int64) error {
if !ma.etcdIsRunning() {
return ErrNotRunning
}

ctx, cancel := context.WithTimeout(context.Background(),
ma.etcd.Server.Cfg.ReqTimeout())
defer cancel()
_, err := ma.etcdCli.Compact(ctx, rev, clientv3.WithCompactPhysical())
// WithCompactPhysical makes Compact wait until all compacted entries are
// removed from the etcd server's storage.
return err
}

func (ma *metaAdmin) etcdInfo() interface{} {
if ma.etcd == nil {
return ErrNotRunning
Expand Down