Fails to mount volume

[johanneswuerbach]

kopeio/etcd-manager:1.0.20180729 fails to mount etcd volumes with Failed to create bus connection: No data available

kops Version 1.10.0 (git-8b52ea6d1)

I0822 11:02:52.053982       1 mounter.go:150] Creating mount directory "/rootfs/mnt/master-vol-04c550c9347a13de8"
I0822 11:02:52.053995       1 mounter.go:155] Mounting device "/dev/xvdu" on "/mnt/master-vol-04c550c9347a13de8"
I0822 11:02:52.054005       1 mount_linux.go:472] Checking for issues with fsck on disk: /dev/xvdu
I0822 11:02:52.054011       1 nsenter_exec.go:50] Running command : nsenter [--mount=/rootfs/proc/1/ns/mnt -- fsck -a /dev/xvdu]
I0822 11:02:52.083063       1 mount_linux.go:491] Attempting to mount disk:  /dev/xvdu /mnt/master-vol-04c550c9347a13de8
I0822 11:02:52.083097       1 nsenter_mount.go:81] nsenter mount /dev/xvdu /mnt/master-vol-04c550c9347a13de8  [defaults]
I0822 11:02:52.083117       1 nsenter.go:106] Running nsenter command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /bin/systemd-run --description=Kubernetes transient mount for /mnt/master-vol-04c550c9347a13de8 --scope -- /bin/mount -o defaults /dev/xvdu /mnt/master-vol-04c550c9347a13de8]
I0822 11:02:52.092447       1 nsenter_mount.go:85] Output of mounting /dev/xvdu to /mnt/master-vol-04c550c9347a13de8: Failed to create bus connection: No data available
I0822 11:02:52.092465       1 mount_linux.go:542] Attempting to determine if disk "/dev/xvdu" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/xvdu])
I0822 11:02:52.092478       1 nsenter_exec.go:50] Running command : nsenter [--mount=/rootfs/proc/1/ns/mnt -- blkid -p -s TYPE -s PTTYPE -o export /dev/xvdu]
I0822 11:02:52.106696       1 mount_linux.go:545] Output: "DEVNAME=/dev/xvdu\nTYPE=ext4\n", err: <nil>
W0822 11:02:52.106733       1 mounter.go:79] unable to mount master volume: "error formatting and mounting disk \"/dev/xvdu\" on \"/mnt/master-vol-04c550c9347a13de8\": exit status 1"

cluster.yml

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2017-06-20T09:30:00Z
  name: xxx
spec:
  api:
    loadBalancer:
      type: Public
  authorization:
    alwaysAllow: {}
  channel: alpha
  cloudProvider: aws
  configBase: s3://xxx
  dnsZone: xxx
  docker:
    bridgeIP: 192.168.5.1/24
    storage: overlay2
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-eu-west-1a
      name: a
      encryptedVolume: true
    - instanceGroup: master-eu-west-1b
      name: b
      encryptedVolume: true
    - instanceGroup: master-eu-west-1c
      name: c
      encryptedVolume: true
    name: main
    manager:
      image: kopeio/etcd-manager:1.0.20180729
  - etcdMembers:
    - instanceGroup: master-eu-west-1a
      name: a
      encryptedVolume: true
    - instanceGroup: master-eu-west-1b
      name: b
      encryptedVolume: true
    - instanceGroup: master-eu-west-1c
      name: c
      encryptedVolume: true
    name: events
    manager:
      image: kopeio/etcd-manager:1.0.20180729
  kubeAPIServer:
    runtimeConfig:
      batch/v2alpha1: "true"
    logLevel: 1
  kubelet:
    logLevel: 1
    podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.1
  kubeProxy:
    logLevel: 1
  kubeControllerManager:
    logLevel: 1
  kubeScheduler:
    logLevel: 1
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.10.7
  masterPublicName: api.xxx
  networkCIDR: xxx
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  hooks:
  - name: disable-locksmithd.service
    before:
    - locksmithd.service
    manifest: |
      Type=oneshot
      ExecStart=/usr/bin/systemctl mask locksmithd.service
      ExecStart=-/usr/bin/systemctl stop locksmithd.service
  sshAccess:
  - xxx
  subnets:
  - cidr: xxx
    name: eu-west-1a
    type: Public
    zone: eu-west-1a
  - cidr: xxx
    name: eu-west-1b
    type: Public
    zone: eu-west-1b
  - cidr: xxx
    name: eu-west-1c
    type: Public
    zone: eu-west-1c
  topology:
    dns:
      type: Public
    masters: public
    nodes: public

one of the three masters:

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-06-22T07:08:46Z
  labels:
    kops.k8s.io/cluster: xxx
  name: master-eu-west-1a
spec:
  detailedInstanceMonitoring: true
  image: coreos.com/CoreOS-stable-*-hvm
  machineType: t2.medium
  maxSize: 1
  minSize: 1
  nodeLabels:
    beta.kubernetes.io/fluentd-ds-ready: "true"
  role: Master
  subnets:
  - eu-west-1a

[johanneswuerbach]

Reverting back to non etcd-manager based etcd deployments worked fine and also running the commands on the host seems to work. Probably this is some issue with the systemd inside the container vs. the systemd on the host?

[gotwarlost]

I had the exact same problem today. The problem goes away if you use the host's pid namespace when running the docker container. Initially I was running the container as:

$ docker run --privileged --net=host -v /:/rootfs kopeio/etcd-manager:3.0.20190930 <args>

where it exhibited the same error Failed to create bus connection: No data available

Some googling led me to systemd/systemd#11300 and changing the command to:

$ docker run --privileged --pid=host --net=host -v /:/rootfs kopeio/etcd-manager:3.0.20190930 <args>

fixed the problem. Hope this helps someone.