Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arm64/Unsupported Architectures on etcd will fail when etcd_kubeadm_enabled is true #6921

Closed
anthr76 opened this issue Nov 17, 2020 · 6 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@anthr76
Copy link
Contributor

anthr76 commented Nov 17, 2020

Environment:

  • Cloud provider or hardware configuration: Bare Metal/Mixed Architecture

  • OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): 20.04.1 LTS (Focal Fossa) 5.4.0-1015-raspi aarch64 & amd64

  • Version of Ansible (ansible --version): 2.9.15

  • Version of Python (python --version): 3.9.0

Kubespray version (commit) (git rev-parse --short HEAD): 0f7341b

Network plugin used: Calico

Full inventory with variables: https://gist.github.com/anthr76/d4ece8f0ddff9f26ad2557e82255354e

Command used to invoke ansible: ansible-playbook -i inventory/nwk1/inventory.ini --become --become-user=root cluster.yml

Output of ansible run: https://gist.github.com/anthr76/25bdb7af962cebed01605bc0756edaad

Anything else do we need to know:

Journalctl output of kubelet: https://gist.github.com/anthr76/60a7490125834a25debd35f3279ed634

In order to get here I had to debug through a few tasks outlined here: #6919

Any idea what might be causing this?

@anthr76 anthr76 added the kind/bug Categorizes issue or PR as related to a bug. label Nov 17, 2020
@anthr76
Copy link
Contributor Author

anthr76 commented Nov 17, 2020

It looks like etcd and kube-apiserver are failing

@anthr76
Copy link
Contributor Author

anthr76 commented Nov 19, 2020

This issue lied with etcd_kubeadm_enabled being true. Like kubespray does with the host managed etcd, if you're on arm platform you will need to set the environment variable ETCD_UNSUPPORTED_ARCH=arm64. It looks like to have etcd managed by kubeadm we would have to make a modification to the static Pod manifest for local etcd. I'm not entirely sure if there's a pretty way on doing that https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#generate-static-pod-manifest-for-local-etcd . Since I'm still new to K8s I'll let some other more experienced folks chime in here.

@anthr76 anthr76 changed the title TASK [kubernetes/master] failing on Initialize first master Arm64/Unsupported Architectures on etcd will fail when etcd_kubeadm_enabled is true Nov 19, 2020
anthr76 added a commit to anthr76/infra that referenced this issue Nov 19, 2020
anthr76 added a commit to anthr76/infra that referenced this issue Nov 20, 2020
@LuckySB
Copy link
Contributor

LuckySB commented Dec 5, 2020

you can try set tag for etcd container v3.4.14-arm64
https://quay.io/repository/coreos/etcd?tab=tags

in kubeadm config

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
etcd:
  # one of local or external
  local:
    imageRepository: "quay.io/coreos"
    imageTag: "v3.4.14-arm64"

@anthr76
Copy link
Contributor Author

anthr76 commented Dec 6, 2020

Hmm I can give this a try on my next cluster installation. I have a multi-arch environment so I would need to create checks based on architecture.

I will consult the docs to see if kubeadm supports providing an image instead of editing the already generated manifest.

To my understanding to have ansible automate this we would need an if statement somewhere around here and likely a few other places right?

@anthr76
Copy link
Contributor Author

anthr76 commented Dec 11, 2020

crictl already shows the arm image present

quay.io/coreos/etcd v3.4.13-arm6

@anthr76
Copy link
Contributor Author

anthr76 commented Dec 12, 2020

Closing in favor of #7025

@anthr76 anthr76 closed this as completed Dec 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

2 participants