K8s HA installation - etcdserver: request timed out

[przemyslavic]

Describe the bug
Kubernetes HA installation fails from time to time on Azure RedHat with canal network plugin.

> 2020-03-28T14:29:36.4847438Z 14:29:36 INFO cli.engine.ansible.AnsibleCommand - TASK [kubernetes_master : Apply flannel definition] ****************************
> 2020-03-28T14:29:48.6915791Z 14:29:48 INFO cli.engine.ansible.AnsibleCommand - fatal: [ci-haazurubuflannel-kubernetes-master-vm-0]: FAILED! => {"changed": true, "cmd": "kubectl apply  -f /home/operations/kube-flannel.yml\n", "delta": "0:00:11.930905", "end": "2020-03-28 14:29:48.652780", "msg": "non-zero return code", "rc": 1, "start": "2020-03-28 14:29:36.721875", "stderr": "Error from server: error when retrieving current configuration of:\nResource: \"policy/v1beta1, Resource=podsecuritypolicies\", GroupVersionKind: \"policy/v1beta1, Kind=PodSecurityPolicy\"\nName: \"psp.flannel.unprivileged\", Namespace: \"\"\nObject: &{map[\"apiVersion\":\"policy/v1beta1\" \"kind\":\"PodSecurityPolicy\" \"metadata\":map[\"annotations\":map[\"apparmor.security.beta.kubernetes.io/allowedProfileNames\":\"runtime/default\" \"apparmor.security.beta.kubernetes.io/defaultProfileName\":\"runtime/default\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\" \"seccomp.security.alpha.kubernetes.io/allowedProfileNames\":\"docker/default\" \"seccomp.security.alpha.kubernetes.io/defaultProfileName\":\"docker/default\"] \"name\":\"psp.flannel.unprivileged\"] \"spec\":map[\"allowPrivilegeEscalation\":%!q(bool=false) \"allowedCapabilities\":[\"NET_ADMIN\"] \"allowedHostPaths\":[map[\"pathPrefix\":\"/etc/cni/net.d\"] map[\"pathPrefix\":\"/etc/kube-flannel\"] map[\"pathPrefix\":\"/run/flannel\"]] \"defaultAddCapabilities\":[] \"defaultAllowPrivilegeEscalation\":%!q(bool=false) \"fsGroup\":map[\"rule\":\"RunAsAny\"] \"hostIPC\":%!q(bool=false) \"hostNetwork\":%!q(bool=true) \"hostPID\":%!q(bool=false) \"hostPorts\":[map[\"max\":'\\uffff' \"min\":'\\x00']] \"privileged\":%!q(bool=false) \"readOnlyRootFilesystem\":%!q(bool=false) \"requiredDropCapabilities\":[] \"runAsUser\":map[\"rule\":\"RunAsAny\"] \"seLinux\":map[\"rule\":\"RunAsAny\"] \"supplementalGroups\":map[\"rule\":\"RunAsAny\"] \"volumes\":[\"configMap\" \"secret\" \"emptyDir\" \"hostPath\"]]]}\nfrom server for: \"/home/operations/kube-flannel.yml\": etcdserver: request timed out", "stderr_lines": ["Error from server: error when retrieving current configuration of:", "Resource: \"policy/v1beta1, Resource=podsecuritypolicies\", GroupVersionKind: \"policy/v1beta1, Kind=PodSecurityPolicy\"", "Name: \"psp.flannel.unprivileged\", Namespace: \"\"", "Object: &{map[\"apiVersion\":\"policy/v1beta1\" \"kind\":\"PodSecurityPolicy\" \"metadata\":map[\"annotations\":map[\"apparmor.security.beta.kubernetes.io/allowedProfileNames\":\"runtime/default\" \"apparmor.security.beta.kubernetes.io/defaultProfileName\":\"runtime/default\" \"kubectl.kubernetes.io/last-applied-configuration\":\"\" \"seccomp.security.alpha.kubernetes.io/allowedProfileNames\":\"docker/default\" \"seccomp.security.alpha.kubernetes.io/defaultProfileName\":\"docker/default\"] \"name\":\"psp.flannel.unprivileged\"] \"spec\":map[\"allowPrivilegeEscalation\":%!q(bool=false) \"allowedCapabilities\":[\"NET_ADMIN\"] \"allowedHostPaths\":[map[\"pathPrefix\":\"/etc/cni/net.d\"] map[\"pathPrefix\":\"/etc/kube-flannel\"] map[\"pathPrefix\":\"/run/flannel\"]] \"defaultAddCapabilities\":[] \"defaultAllowPrivilegeEscalation\":%!q(bool=false) \"fsGroup\":map[\"rule\":\"RunAsAny\"] \"hostIPC\":%!q(bool=false) \"hostNetwork\":%!q(bool=true) \"hostPID\":%!q(bool=false) \"hostPorts\":[map[\"max\":'\\uffff' \"min\":'\\x00']] \"privileged\":%!q(bool=false) \"readOnlyRootFilesystem\":%!q(bool=false) \"requiredDropCapabilities\":[] \"runAsUser\":map[\"rule\":\"RunAsAny\"] \"seLinux\":map[\"rule\":\"RunAsAny\"] \"supplementalGroups\":map[\"rule\":\"RunAsAny\"] \"volumes\":[\"configMap\" \"secret\" \"emptyDir\" \"hostPath\"]]]}", "from server for: \"/home/operations/kube-flannel.yml\": etcdserver: request timed out"], "stdout": "clusterrole.rbac.authorization.k8s.io/flannel unchanged\nclusterrolebinding.rbac.authorization.k8s.io/flannel unchanged\nserviceaccount/flannel unchanged\nconfigmap/kube-flannel-cfg unchanged\ndaemonset.apps/kube-flannel-ds-amd64 unchanged\ndaemonset.apps/kube-flannel-ds-arm64 unchanged\ndaemonset.apps/kube-flannel-ds-arm unchanged\ndaemonset.apps/kube-flannel-ds-ppc64le unchanged\ndaemonset.apps/kube-flannel-ds-s390x unchanged", "stdout_lines": ["clusterrole.rbac.authorization.k8s.io/flannel unchanged", "clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged", "serviceaccount/flannel unchanged", "configmap/kube-flannel-cfg unchanged", "daemonset.apps/kube-flannel-ds-amd64 unchanged", "daemonset.apps/kube-flannel-ds-arm64 unchanged", "daemonset.apps/kube-flannel-ds-arm unchanged", "daemonset.apps/kube-flannel-ds-ppc64le unchanged", "daemonset.apps/kube-flannel-ds-s390x unchanged"]}
> 
> 

To Reproduce
Steps to reproduce the behavior:

1. Create configuration for Azure with Kubernetes HA with Canal network plugin
2. Execute epicli apply ...

Expected behavior
Build should run properly.

Config files
None

OS (please complete the following information):

• OS: RHEL 7

Cloud Environment (please complete the following information):

• Cloud Provider: Azure

Additional context
Canal network plugin

[przemyslavic]

Another task with te same issue:

2020-03-29T19:21:31.7026889Z 19:21:31 INFO cli.engine.ansible.AnsibleCommand - TASK [kubernetes_master : Apply /etc/epiphany/manifests/users.yml file] ********
2020-03-29T19:21:40.6886997Z 19:21:40 INFO cli.engine.ansible.AnsibleCommand - fatal: [ci-haazurrhelcanal-kubernetes-master-vm-0]: FAILED! => {"changed": true, "cmd": "kubectl apply  -f /etc/epiphany/manifests/users.yml\n", "delta": "0:00:07.677764", "end": "2020-03-29 19:21:40.481624", "msg": "non-zero return code", "rc": 1, "start": "2020-03-29 19:21:32.803860", "stderr": "Error from server: error when retrieving current configuration of:\nResource: \"/v1, Resource=serviceaccounts\", GroupVersionKind: \"/v1, Kind=ServiceAccount\"\nName: \"admin\", Namespace: \"kube-system\"\nObject: &{map[\"apiVersion\":\"v1\" \"kind\":\"ServiceAccount\" \"metadata\":map[\"annotations\":map[\"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"name\":\"admin\" \"namespace\":\"kube-system\"]]}\nfrom server for: \"/etc/epiphany/manifests/users.yml\": etcdserver: request timed out", "stderr_lines": ["Error from server: error when retrieving current configuration of:", "Resource: \"/v1, Resource=serviceaccounts\", GroupVersionKind: \"/v1, Kind=ServiceAccount\"", "Name: \"admin\", Namespace: \"kube-system\"", "Object: &{map[\"apiVersion\":\"v1\" \"kind\":\"ServiceAccount\" \"metadata\":map[\"annotations\":map[\"kubectl.kubernetes.io/last-applied-configuration\":\"\"] \"name\":\"admin\" \"namespace\":\"kube-system\"]]}", "from server for: \"/etc/epiphany/manifests/users.yml\": etcdserver: request timed out"], "stdout": "serviceaccount/operator unchanged\nserviceaccount/reader unchanged", "stdout_lines": ["serviceaccount/operator unchanged", "serviceaccount/reader unchanged"]}

[mkyc]

@przemyslavic can you review this bug? If that is still existing can you please provide steps to reproduce?

[mkyc]

@przemyslavic please check.

[przemyslavic]

The problem was reported quite a long time ago and was related to the old version of Epiphany and Kubernetes. Currently I'm not being able to reproduce the problem.
Closing this task. If noticed again, a new issue will be created.