Sporadic error during startup: no_epmd_port

[ansd]

Observed in Kubernetes. To reproduce:

kind create cluster --image kindest/node:v1.21.12@sha256:f316b33dd88f8196379f38feb80545ef3ed44d9197dca1bfd48bcb1583210207
kubectl rabbitmq install-cluster-operator
kubectl rabbitmq create my-rabbit --image rabbitmq:3.10.6-management --replicas 5
kubectl get pods -lapp.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=my-rabbit --watch

# Wait until all pods are READY.
# If you don't see a RESTART, delete that Rabbit cluster.
kubectl rabbitmq delete my-rabbit

# Wait a few seconds until all pods are deleted.
# In case, the pods are stuck in STATUS 'Terminating':
kubectl delete pods -lapp.kubernetes.io/name=my-rabbit --force=true --grace-period=0

# For me a RESTART occurs after the 2nd cluster creation already.
# In below output my-rabbit-server-0 was restarted (but it can be any other pod)
kubectl rabbitmq create my-rabbit --image rabbitmq:3.10.6-management --replicas 5
kubectl get pods -lapp.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=my-rabbit --watch

NAME                 READY   STATUS    RESTARTS   AGE
my-rabbit-server-0   0/1     Running   0          35s
my-rabbit-server-1   1/1     Running   0          35s
my-rabbit-server-2   0/1     Running   0          35s
my-rabbit-server-3   1/1     Running   0          35s
my-rabbit-server-4   1/1     Running   0          35s
my-rabbit-server-2   1/1     Running   0          36s
my-rabbit-server-0   0/1     Completed   0          42s
my-rabbit-server-0   0/1     Running     1          43s
my-rabbit-server-0   1/1     Running     1          54s

# To see the logs of the failed container:
kubectl logs my-rabbit-server-0 -c rabbitmq --previous

...
BOOT FAILED
===========
Error during startup: {error,no_epmd_port}

2022-07-22 16:28:12.538492+00:00 [error] <0.627.0>
2022-07-22 16:28:12.538492+00:00 [error] <0.627.0> BOOT FAILED
2022-07-22 16:28:12.538492+00:00 [error] <0.627.0> ===========
2022-07-22 16:28:12.538492+00:00 [error] <0.627.0> Error during startup: {error,no_epmd_port}
2022-07-22 16:28:12.538492+00:00 [error] <0.627.0>

[HoloRin]

Related: #2722

[ansd]

Also related: #4233

It's worth mentioning that we've observed all these DNS related issues on only some K8s environments, especially on kind.

[ansd]

Issues

• Wait between retries if nxdomain #4233
• Retry epmd name lookups N times during pre-launch #2722
• RabbitMQ Node sometimes fails with nxdomain on startup cluster-operator#958
• RabbitMQ Cluster failed with 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims on Kubernetes cluster-operator#752
• Using RabbitMQ Cluster Kubernetes Operator step by step but failed to get pod ready cluster-operator#269
• Look into setting podManagementPolicy to Parallel cluster-operator#298 (comment)
• Sporadic error during startup: no_epmd_port #5322

are all symptoms of the following CoreDNS issues:

• Headless service name resolution delay coredns/coredns#2434
• Headless Service not publishing not ready addresses for statefulset kubernetes/kubernetes#92559

On the headless service of the RabbitMQ cluster, the rabbitmq/cluster-operator sets publishNotReadyAddresses to true for the purpose of peer discovery since RabbitMQ nodes need to talk to their peer nodes early at Pod startup. However, with CoreDNS default cache value of 30 it takes up to 30 seconds for CoreDNS to resolve correctly.

Therefore we had to add all kind of retries within RabbitMQ code.

In kind (kindest/node:v1.24.0), I can't reproduce #5322 (comment) once I set CoreDNS cache value to 2.

To try it out, you can edit the coredns ConfigMap in namespace kube-system directly. Make sure that reload is present and also add log into the config to see when both CoreDNS Pods reload the config. This takes up to 2 minutes. For example:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 2
        log
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2022-07-26T17:44:22Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "536"
  uid: 00874286-42cb-4310-a9e0-d758d6c6f8e6

Reducing the cache TTL is recommended in https://github.com/kubernetes/website/pull/22446/files. Prior to switching the CoreDNS default to 30, it was set to 5.

Alternative workaround

Since this is a Kubernetes CoreDNS issue, instead of adding retries into RabbitMQ, and because the rabbitmq/cluster-operator already defines an init container, we could alternatively just wait in that init container until CoreDNS resolves correctly:
The following patch to current rabbitmq/cluster-operator main branch works (when the host tool gets installed):

diff --git a/internal/resource/statefulset.go b/internal/resource/statefulset.go
index 21a90c2..b87c108 100644
--- a/internal/resource/statefulset.go
+++ b/internal/resource/statefulset.go
@@ -722,7 +722,8 @@ func setupContainer(instance *rabbitmqv1beta1.RabbitmqCluster) corev1.Container
                                "cp /tmp/rabbitmq-plugins/enabled_plugins /operator/enabled_plugins ; " +
                                "echo '[default]' > /var/lib/rabbitmq/.rabbitmqadmin.conf " +
                                "&& sed -e 's/default_user/username/' -e 's/default_pass/password/' %s >> /var/lib/rabbitmq/.rabbitmqadmin.conf " +
-                               "&& chmod 600 /var/lib/rabbitmq/.rabbitmqadmin.conf",
+                               "&& chmod 600 /var/lib/rabbitmq/.rabbitmqadmin.conf ; " +
+                               "until host \"$HOSTNAME_DOMAIN\"; do echo waiting for 'host \"$HOSTNAME_DOMAIN\"' to succeed; sleep 5; done",
                },
                Resources: corev1.ResourceRequirements{
                        Limits: corev1.ResourceList{
@@ -756,6 +757,11 @@ func setupContainer(instance *rabbitmqv1beta1.RabbitmqCluster) corev1.Container
                                MountPath: "/var/lib/rabbitmq/mnesia/",
                        },
                },
+               Env: append(envVarsK8sObjects(instance),
+                       corev1.EnvVar{
+                               Name:  "HOSTNAME_DOMAIN",
+                               Value: "$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)",
+                       }),
        }

        if instance.VaultDefaultUserSecretEnabled() {

However, this works only when the CoreDNS deployment is scaled down from 2 replicas to 1 replica. (I suppose this is because every CoreDNS pod has its own cache.)