Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: can't recover after a upgrade failed #120

Merged
merged 28 commits into from
Oct 19, 2018
Merged
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
a7de64c
tidb graceful upgrade
xiaojingchen Sep 30, 2018
2cfbde5
Merge branch 'master' into tidb-graceful-upgrade
weekface Oct 11, 2018
2b78c7b
Merge branch 'master' into tidb-graceful-upgrade
xiaojingchen Oct 11, 2018
fc67d9a
Merge branch 'tidb-graceful-upgrade' of https://github.com/xiaojingch…
xiaojingchen Oct 11, 2018
19ac0fa
address comment
xiaojingchen Oct 11, 2018
16b03a3
address comment
xiaojingchen Oct 11, 2018
99f3a37
rewrite upgrade to fix some bugs
xiaojingchen Oct 12, 2018
bf8b5a4
Merge branch 'master' into fix-ugrade
xiaojingchen Oct 13, 2018
5781db6
Merge branch 'master' into fix-ugrade
gregwebs Oct 13, 2018
756691a
Merge branch 'master' into fix-ugrade
xiaojingchen Oct 14, 2018
645f067
Merge branch 'fix-ugrade' of https://github.com/xiaojingchen/tidb-ope…
xiaojingchen Oct 14, 2018
4cf1d11
fix bugs
xiaojingchen Oct 15, 2018
24524f2
address comment
xiaojingchen Oct 15, 2018
8d149b7
Merge branch 'master' into fix-ugrade
tennix Oct 15, 2018
19eddb5
fix upgrade state sync bug
xiaojingchen Oct 16, 2018
3afa97a
Merge branch 'fix-ugrade' of https://github.com/xiaojingchen/tidb-ope…
xiaojingchen Oct 16, 2018
3caece1
Merge branch 'fix-ugrade' of https://github.com/xiaojingchen/tidb-ope…
xiaojingchen Oct 16, 2018
bd16aaf
Merge branch 'fix-ugrade' of https://github.com/xiaojingchen/tidb-ope…
xiaojingchen Oct 16, 2018
2de52da
fix tikv upgrade bug
xiaojingchen Oct 17, 2018
f666f20
Merge branch 'master' into fix-ugrade
weekface Oct 18, 2018
5938988
fix empty point bug
xiaojingchen Oct 18, 2018
7cb3f53
Merge branch 'fix-ugrade' of https://github.com/xiaojingchen/tidb-ope…
xiaojingchen Oct 18, 2018
5429058
address comment
xiaojingchen Oct 18, 2018
8ea29e7
address comment
xiaojingchen Oct 18, 2018
7239470
address comment
xiaojingchen Oct 18, 2018
e0b1163
Merge branch 'master' into fix-ugrade
xiaojingchen Oct 18, 2018
63d333f
Merge branch 'master' into fix-ugrade
tennix Oct 19, 2018
8b29919
Merge branch 'master' into fix-ugrade
weekface Oct 19, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions pkg/manager/member/tikv_upgrader.go
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,16 @@ func (tku *tikvUpgrader) Upgrade(tc *v1alpha1.TidbCluster, oldSet *apps.Stateful
return controller.RequeueErrorf("tidbcluster: [%s/%s]'s tikv pod: [%s] have not label: %s", ns, tcName, podName, apps.ControllerRevisionHashLabelKey)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/have not/has no/

}

if store == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this to after line 76 to avoid retrieving pod when store is nil.

setUpgradePartition(newSet, i)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why set partition to i?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should return an error if store == nil

continue
}
if revision == tc.Status.TiKV.StatefulSet.UpdateRevision {

if pod.Status.Phase != corev1.PodRunning {
return controller.RequeueErrorf("tidbcluster: [%s/%s]'s upgraded tikv pods are not all running", ns, tcName)
}
if store == nil || store.State != v1alpha1.TiKVStateUp {
if store.State != v1alpha1.TiKVStateUp {
return controller.RequeueErrorf("tidbcluster: [%s/%s]'s upgraded tikv are not all ready", ns, tcName)
}
err := tku.endEvictLeader(tc, i)
Expand Down Expand Up @@ -199,11 +204,6 @@ func (tku *tikvUpgrader) getStoreByOrdinal(tc *v1alpha1.TidbCluster, ordinal int
return &store
}
}
for _, store := range tc.Status.TiKV.TombstoneStores {
if store.PodName == podName {
return &store
}
}
return nil
}

Expand Down