MountVolume.SetUp failed for volume "data" : invalid character '/'

[oportingale]

Hi,

I am trying to get this running on Azure AKS but am getting the error-

MountVolume.SetUp failed for volume "data" : invalid character '/' looking for beginning of value

I have tried this on AKS version 1.96 and 1.8.11
The container doesnt complete creating so I cannot connect to get the logs.
I'm attempting to create a pod using the nginx image.
Any help would be appreciated!

apiVersion: v1
kind: Pod
metadata:
name: nginx-flex-blobfuse1
spec:
containers:

• name: nginx-flex-blobfuse1
  image: nginx
  volumeMounts:
  • name: data
    mountPath: /data
    volumes:
• name: data
  flexVolume:
  driver: "azure/blobfuse"
  readOnly: false
  secretRef:
  name: blobfusecreds
  options:
  container: video

[andyzhangx]

I think you need to ssh to every agent node(you may assisn a public IP first), and install blobfuse binary:
https://github.com/Azure/kubernetes-volume-drivers/tree/master/flexvolume/blobfuse#2-install-blobfuse-driver-on-every-agent-node

We are make blobfuse built-in in acs-engine and AKS, while currently we need to install blobfuse manually.

[oportingale]

Hi Andy, thanks for the prompt response.
I apologise for my naivety, I have only just started using Kubernetes.
I was under the impression that running this command would install the blobfuse on the agent-

kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer-1.9.yaml

Do you know if its possible to install blobfuse directly on a container?

If it needs to be installed on each agent how would node scaling work?

Thanks im advance.

Olly.

[andyzhangx]

Following command will only install blobfuse flexvolume driver, while you still need to intall blobfuse on every node

kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer-1.9.yaml

We are working on preinstall blobfuse & flexvolume driver on acs-engine and AKS, so when node scale in acs-engine or AKS, new node will also have blobfuse installed. Currently you need to install blobfuse manually.

[oportingale]

Ok thanks, I will try that.
Do you know how far off the preinstalled blobfuse driver will be please?

[andyzhangx]

How do you create k8s cluster on Azure? For acs-engine, there is already a working in progress PR: Azure/acs-engine#3401, after that merged, AKS will adopt that code change within one month.

[oportingale]

Hi, I am creating the Kubernetes service directly via the portal. Latest version is 1.10.3.
Thanks

[oportingale]

Hi, I can confirm that installing the blobfuse driver directly on the agent resolved the issue.
Many thanks for your assistance in this.
I am holding out for the version of AKS with this rolled in. I am not sure how we will handle scaling right now.

[andyzhangx]

just provided you an easy way to deploy blobfuse driver:
https://github.com/Azure/kubernetes-volume-drivers/blob/master/flexvolume/blobfuse/README.md#option1-use-az-vm-extension-to-install-blobfuse-driver-on-every-agent-vm

Option#1. Use az vm extension to install blobfuse driver on every agent VM

#az login
az vm extension set \
  --resource-group RESOURCE_GROUP_NAME \
  --vm-name VM_NAME \
  --name customScript \
  --publisher Microsoft.Azure.Extensions \
  --protected-settings '{"fileUris": ["https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/deployment/install-blobfuse-flexvol-ubuntu.sh"],"commandToExecute": "./install-blobfuse-flexvol-ubuntu.sh"}'

[oportingale]

Thats fantastic! Thanks so much 👍

[andyzhangx]

since blobfuse is already installed by default on AKS and acs-engine, I will close this issue now.
/close

[IvanovOleg]

@andyzhangx I have the same issue on kubernetes installed on CoreOS (blobfuse installed on each node)

Tue Mar 5 18:30:19 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at 
/var/lib/kubelet/pods/01e692db-3f73-11e9-884c-000d3a4793c1/volumes/azure~blobfuse/pgdatabase-
pv-blobfuse-flexvol, accountname:xxxxxxxxxxxx, error log:Tue Mar 5 18:30:19 UTC 2019 EXEC: 
/usr/bin/blobfuse /var/lib/kubelet/pods/01e692db-3f73-11e9-884c-000d3a4793c1/volumes
/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol --container-name=postgresql-backup --tmp-
path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120" }

on each VM blobefuse path is /run/torcx/bin/blobfuse which is added to the $PATH and works on the node itself.
Looks like it tries to find it in /usr/bin/blobfuse