Problems mounting EBS volumes in ETCD nodes in the same availability zone

[jmolinavizcaino]

Hi;

We have a similar issue then #281 but with different effects. We have a Kubernetes cluster deployed with kops in AWS. The master nodes with etcd-manager 3.0.202005.31 are running in EC2 instances in the same availability zone (this is not a production cluster) . The cluster is started every morning and destroyed every night.

When a master node starts the etcd-manager mount the EBS volume (where is the Kubernetes API objects) based in the next tags:

I1218 05:50:26.648925 4224 main.go:279] Mounting available etcd volumes matching tags [k8s.io/etcd/main k8s.io/role/master=1 kubernetes.io/cluster/kubernetes.cluster.test=owned]; nameTag=k8s.io/etcd/main

This tags are common in all volumes of the ETCD cluster nodes and how they are in the same availability zone the etcd-manager cannot identify which volume belongs to which node. This causes two effects:

1. Sometimes one ETCD node has attached the volume of the other ETCD node and vice versa. Although it's not a desired scenario really this don't take bad effects in the clusters because the information is synchronized and the kubernetes cluster can run normally

2. Other times, one ETCD node mount the 3 EBS volumes and this is a bad scenario because only one ETCD node can run and the Kubernetes API is not accessible.

Somebody have the same issue and have a workaround?

We think it's necessary to add another tag to permit to the etcd-manager identify the volume that belongs to the node and modify the boot to check it. Maybe it's possible checking the value of the k8s.io/etcd/main:

node / node1,node2,node3

Thanks to everybody