Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HA proxy is not restarted after host restart #1562

Closed
khenidak opened this issue May 5, 2020 · 2 comments · Fixed by #1564
Closed

HA proxy is not restarted after host restart #1562

khenidak opened this issue May 5, 2020 · 2 comments · Fixed by #1564
Assignees
@BenTheElder
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@khenidak
Copy link

khenidak commented May 5, 2020

What happened:
Restarting the host breaks an HA kind cluster. HA Proxy container is not restarted as the rest of the nodes.
What you expected to happen:
HA Proxy restart just like the rest of the node
Here is docker ps output (after restarting the host)

$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                       NAMES
2bfea21bda67        kindest/node:v1.18.2   "/usr/local/bin/entr…"   37 minutes ago      Up 30 minutes       127.0.0.1:45923->6443/tcp   kind-control-plane
c5b11e978988        kindest/node:v1.18.2   "/usr/local/bin/entr…"   37 minutes ago      Up 30 minutes                                   kind-worker
e91543d62d01        kindest/node:v1.18.2   "/usr/local/bin/entr…"   37 minutes ago      Up 30 minutes                                   kind-worker2
c32452cad1b7        kindest/node:v1.18.2   "/usr/local/bin/entr…"   37 minutes ago      Up 30 minutes                                   kind-worker3
992a88407c8c        kindest/node:v1.18.2   "/usr/local/bin/entr…"   37 minutes ago      Up 30 minutes       127.0.0.1:46539->6443/tcp   kind-control-plane3
b6fbdafa1dac        kindest/node:v1.18.2   "/usr/local/bin/entr…"   37 minutes ago      Up 30 minutes       127.0.0.1:41709->6443/tcp   kind-control-plane2

here is docker ps -a (after restart)

$ docker ps -a
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                      PORTS                       NAMES
95bc4954e99a        kindest/haproxy:2.1.1-alpine   "/docker-entrypoint.…"   37 minutes ago      Exited (0) 33 minutes ago                               kind-external-load-balancer
2bfea21bda67        kindest/node:v1.18.2           "/usr/local/bin/entr…"   37 minutes ago      Up 31 minutes               127.0.0.1:45923->6443/tcp   kind-control-plane
c5b11e978988        kindest/node:v1.18.2           "/usr/local/bin/entr…"   37 minutes ago      Up 31 minutes                                           kind-worker
e91543d62d01        kindest/node:v1.18.2           "/usr/local/bin/entr…"   37 minutes ago      Up 31 minutes                                           kind-worker2
c32452cad1b7        kindest/node:v1.18.2           "/usr/local/bin/entr…"   37 minutes ago      Up 31 minutes                                           kind-worker3
992a88407c8c        kindest/node:v1.18.2           "/usr/local/bin/entr…"   37 minutes ago      Up 31 minutes               127.0.0.1:46539->6443/tcp   kind-control-plane3
b6fbdafa1dac        kindest/node:v1.18.2           "/usr/local/bin/entr…"   37 minutes ago      Up 31 minutes               127.0.0.1:41709->6443/tcp   kind-control-plane2

How to reproduce it (as minimally and precisely as possible):
create ha cluster via

# a cluster with 3 control-plane nodes and 3 workers
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker

then restart host, kubectl should fail because ha proxy has not been restarted

Anything else we need to know?:

Environment:

  • kind version: (use kind version):
    kind v0.8.0 go1.14.2 linux/amd64

  • Kubernetes version: (use kubectl version):

  • Docker version: (use docker info):

  • OS (e.g. from /etc/os-release):
@khenidak khenidak added the kind/bug Categorizes issue or PR as related to a bug. label May 5, 2020
@BenTheElder
Copy link
Member

/assign
/priority important-soon

@k8s-ci-robot k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label May 5, 2020
@aojea aojea mentioned this issue May 5, 2020
Merged
@aojea
Copy link
Contributor

aojea commented May 5, 2020 •
edited
Loading

nice catch
we have several issues here, the load balancer wasn't using hostnames,
and we need some additional directives in haproxy to not die until the backends are alive

Fixed by
#1564

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BenTheElder BenTheElder

Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

allow loadbalancer to restart gracefully aojea/kind
4 participants
@BenTheElder @aojea @khenidak @k8s-ci-robot