Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot restore etcd V2 data after backup when data-dir exists snapshot files. #9890

Closed
zhousoft opened this issue Jun 28, 2018 · 3 comments
Closed

Comments

@zhousoft
Copy link

etcd version:

etcd Version: 3.2.4
Git SHA: c31bec0
Go Version: go1.8.3
Go OS/Arch: linux/amd64

step 1.
I start a new etcd node and set several data(only V2 data),make sure there is no snapshot file,then make a backup.

etcdctl backup --data-dir /home/etcd_data --backup-dir /home/etcd_data_backup

kill the old etcd process, start a new etcd:

/etcd -name etcd1  --data-dir=/home/etcd_data_backup/ -force-new-cluster

it wokers well, the data set before has be recovered.

step 2.
Removing all data dir created in setp 1 and start a new etcd with "--snapshot-count '10'" and set enough data to generate snapshot file,then:

etcdctl backup --data-dir /home/etcd_data --backup-dir /home/etcd_data_backup

kill the old etcd process, try to restore from backup,

/etcd -name etcd1  --data-dir=/home/etcd_data_backup/ -force-new-cluster

restoring failed, get the message as:

panic: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xb5129c]

goroutine 1 [running]:
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer.func1(0xc4201e6678, 0xc4201e6470)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:279 +0x3c
panic(0xd628e0, 0xc42016b410)
        /usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc42018bfc0, 0xf31eff, 0x2a, 0xc4201e64e0, 0x1, 0x1)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x15c
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer(0xc42025e000, 0x0, 0x1402540, 0xc42016b350)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:374 +0x2e39
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.StartEtcd(0xc420093180, 0x0, 0x0, 0x0)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/etcd.go:147 +0x7c0
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcd(0xc420093180, 0x6, 0xf0e97d, 0x6, 0x1)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:186 +0x58
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2()
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:103 +0x15ba
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main()
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:39 +0x61
main.main()
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/etcd/main.go:28 +0x2

a workaround works for me is to copy the db file form /home/etcd_data/member/snap/ to /home/etcd_data_backup/member/snap/, but I am not sure is it correct way?

@hexfusion
Copy link
Contributor

a workaround works for me is to copy the db file form /home/etcd_data/member/snap/ to /home/etcd_data_backup/member/snap/, but I am not sure is it correct way?

@zhousoft I believe that you are running into a known issue that to restore etcd v3 you require v3 data. Please review the note in the upgrade doc below. I understand you are not upgrading from v2 but as you only have v2 data the problem is the same.

https://github.com/coreos/etcd/blob/master/Documentation/upgrades/upgrade_3_0.md#upgrade-checklists

It is recommended to write a dummy v3 key so that some v3 data exists.

ref: #9480

@hexfusion
Copy link
Contributor

So to answer your original question I believe you could also do a v3 snapshot when the node is still running as an alternative to copying the directory. This would give you latest v3 snapshot then restore v2 as you are doing.

$ ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save /home/etcd_data_backup/member/snap/db

@zhousoft
Copy link
Author

@hexfusion Thanks for the reply, it is helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants