-
Notifications
You must be signed in to change notification settings - Fork 0
/
leaderelection-in-k8s.html
160 lines (99 loc) · 3.16 KB
/
leaderelection-in-k8s.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
<!DOCTYPE html>
<html>
<head>
<title>Leader Election in K8s</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<link rel="stylesheet" type="text/css" href="styles.css">
</head>
<body>
<textarea id="source">
class: middle, center
# Leader Election in K8s
<img width="100%" src="https://d33wubrfki0l68.cloudfront.net/eb4e41f2cba0cbc8d119f8d0eb2bd6935cb78fc8/ba7d6/images/community/kubernetes-community-final-02.jpg">
<img
style="border-radius: 50px"
src="https://www.gravatar.com/avatar/67644641ead7ae60a795a14b7e102973?s=100"
alt="Weihang Lo">
<a href="https://github.com/weihanglo">@weihanglo</a>
---
### Goals
- Understand common properties of learder election algorithm
- Understand leader election algorithm in K8s
- Demo time?
---
---
### Leader election algorithm
Pros
- Redundancy
- Reducing coordination
- Improving efficiency
Cons
- Introduces failure mode
- Scaling bottleneck
---
### Properties 1, 2, 3, 4
One leader
- Should only get one leader in a election
Two states
- Elected
- Non-elected
Three conditions
- Termination
- Uniqueness
- Agreement
Four aspects
- Communication mechanism
- Process names
- Network topology
- Size of the network
---
### How `client-go` elects leaders?
- Atomicity depends on K8s API atomicity (resource as some sort of Lock primitive)
- ResourceVersions - Every API object has a unique ResourceVersion, and you can use these versions to perform compare-and-swap on Kubernetes objects
- Annotations - Every API object can be annotated with arbitrary key/value pairs to be used by clients.
- Lease-based
- Single database store who's current leader
- Heartbeat telling everyone I am alive
- Depends on time elapse duration (not wall-clock time)
---
### `client-go` properties
- Does not guarantee that only one client is acting as a leader (a.k.a. no fencing).
- A client only acts on timestamps captured locally to infer the state of the leader election (instead of a global timestamp).
- Depnds on changes not accuracy
- Tolerant to arbitrary clock skew
- Not tolerant to arbitrary clock skew rate.
---
### `client-go` architecture
Two parts
- Resource Lock (API Object)
- Create/Get/Update lock
- Eelector
- Run the election
- Acquire/Release lock -> depends on resource lock
- Update lock -> depends on resource lock
---
### What happens when a leader fails?
- Two separate steps
- "make work durable"
- "tell others it is complete"
- One important property: idempotency (or via optimistic locking)
---
### Use Cases
- kube-scheduler
- kube-controller-manager
- snapshot-controller
- cluster-autoscaler
---
### References
- https://en.wikipedia.org/wiki/Leader_election
- https://pkg.go.dev/k8s.io/client-go/tools/leaderelection
- https://aws.amazon.com/builders-library/leader-election-in-distributed-systems/
</textarea>
<script src="https://remarkjs.com/downloads/remark-latest.min.js" type="text/javascript">
</script>
<script type="text/javascript">
var slideshow = remark.create();
</script>
</body>
</html>