cdi importer failed due to permission denied

[p-alik]

What happened

A clear and concise description of what the bug is.

An attempt to follow instruction Use CDI to Import a Disk Image run into issue similar to #2433

11:55 $ kubectl logs importer-fedora 
I0822 09:54:48.178731       1 importer.go:103] Starting importer
I0822 09:54:48.178759       1 importer.go:172] begin import process
I0822 09:54:49.466663       1 data-processor.go:356] Calculating available size
I0822 09:54:49.466706       1 data-processor.go:368] Checking out file system volume size.
I0822 09:54:49.466725       1 data-processor.go:376] Request image size not empty.
I0822 09:54:49.466743       1 data-processor.go:381] Target size 5Gi.
I0822 09:54:49.580816       1 data-processor.go:255] New phase: TransferDataFile
E0822 09:54:49.580864       1 data-processor.go:251] open /data/disk.img: permission denied
could not open file "/data/disk.img"
kubevirt.io/containerized-data-importer/pkg/util.OpenFileOrBlockDevice
	pkg/util/util.go:182
kubevirt.io/containerized-data-importer/pkg/util.StreamDataToFile
	pkg/util/util.go:189
kubevirt.io/containerized-data-importer/pkg/importer.(*HTTPDataSource).TransferFile
	pkg/importer/http-datasource.go:183
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func4
	pkg/importer/data-processor.go:198
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:248
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:157
main.handleImport
	cmd/cdi-importer/importer.go:178
main.main
	cmd/cdi-importer/importer.go:144
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1594
Unable to transfer source data to target file
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func4
	pkg/importer/data-processor.go:200
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:248
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:157
main.handleImport
	cmd/cdi-importer/importer.go:178
main.main
	cmd/cdi-importer/importer.go:144
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1594
E0822 09:54:49.580928       1 importer.go:181] open /data/disk.img: permission denied
could not open file "/data/disk.img"
kubevirt.io/containerized-data-importer/pkg/util.OpenFileOrBlockDevice
	pkg/util/util.go:182
kubevirt.io/containerized-data-importer/pkg/util.StreamDataToFile
	pkg/util/util.go:189
kubevirt.io/containerized-data-importer/pkg/importer.(*HTTPDataSource).TransferFile
	pkg/importer/http-datasource.go:183
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func4
	pkg/importer/data-processor.go:198
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:248
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:157
main.handleImport
	cmd/cdi-importer/importer.go:178
main.main
	cmd/cdi-importer/importer.go:144
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1594
Unable to transfer source data to target file
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func4
	pkg/importer/data-processor.go:200
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	pkg/importer/data-processor.go:248
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	pkg/importer/data-processor.go:157
main.handleImport
	cmd/cdi-importer/importer.go:178
main.main
	cmd/cdi-importer/importer.go:144
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1594

How to reproduce it (as minimally and precisely as possible)

• setup minikube profile

11:59 $ minikube profile list 
|----------|-----------|---------|----------------|------|---------|---------|-------|--------|
| Profile  | VM Driver | Runtime |       IP       | Port | Version | Status  | Nodes | Active |
|----------|-----------|---------|----------------|------|---------|---------|-------|--------|
| kubevirt | kvm2      | docker  | 192.168.39.158 | 8443 | v1.27.2 | Running |     3 | *      |
|----------|-----------|---------|----------------|------|---------|---------|-------|--------|

• Follow the Experiment with the Containerized Data Importer (CDI)

Environment

• CDI version: v1.57.0
• Kubernetes version: v1.27.2
• OS: Ubuntu 23.04
• Kernel: 6.2.0-27-generic

[akalenyu]

similar to #2433

So this is probably not the same issue since you are using a Filesystem PVC
(as opposed to Block which may require the extra config)

Could you attach the PVC that was created alongside the storage class that is used to provision it?

Also note we are actually doing some refactoring to this lab page at kubevirt/kubevirt.github.io#914
So may be worth to try creating the same scenario with a DataVolume instead

[p-alik]

thanks for quick response, @akalenyu.

• PVC

$ kubectl get pvc/fedora  --output yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.condition.running: "false"
    cdi.kubevirt.io/storage.condition.running.message: 'Unable to process data: Unable
      to transfer source data to target file: could not open file "/data/disk.img":
      open /data/disk.img: permission denied'
    cdi.kubevirt.io/storage.condition.running.reason: Error
    cdi.kubevirt.io/storage.import.endpoint: https://download.fedoraproject.org/pub/fedora/linux/releases/36/Cloud/x86_64/images/Fedora-Cloud-Base-36-1.5.x86_64.raw.xz
    cdi.kubevirt.io/storage.import.importPodName: importer-fedora
    cdi.kubevirt.io/storage.pod.phase: Running
    cdi.kubevirt.io/storage.pod.restarts: "31"
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
    volume.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
  creationTimestamp: "2023-08-22T09:43:36Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: containerized-data-importer
  name: fedora
  namespace: default
  resourceVersion: "42125"
  uid: 3290183a-a2b8-420b-99d4-e0477446939e
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: standard
  volumeMode: Filesystem
  volumeName: pvc-3290183a-a2b8-420b-99d4-e0477446939e
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  phase: Bound

• storage class "standard"

$ kubectl describe storageclasses.storage.k8s.io/standard 
Name:            standard
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists"},"name":"standard"},"provisioner":"k8s.io/minikube-hostpath"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           k8s.io/minikube-hostpath
Parameters:            <none>
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

• I'll try to create the same scenario with a DataVolume

[p-alik]

scenario with a DataVolume based on the first sample of Data Volumes failed also

$ cat <<'HERE' | kubectl create --filename -
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: "example-import-dv"
spec:
  source:
      http:
         url: "https://github.com/cirros-dev/cirros/releases/download/0.4.0/cirros-0.4.0-x86_64-disk.img"
         secretRef: "" # Optional
         certConfigMap: "" # Optional
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: "64Mi"
HERE
datavolume.cdi.kubevirt.io/example-import-dv created
$ kubectl describe datavolume.cdi.kubevirt.io/example-import-dv
...
Events:
  Type     Reason                Age                From                          Message
  ----     ------                ----               ----                          -------
  Normal   Pending               38s                datavolume-import-controller  PVC example-import-dv Pending
  Warning  ImportFailed          32s                datavolume-import-controller  Failed to import into PVC example-import-dv
  Normal   Bound                 31s (x2 over 38s)  datavolume-import-controller  PVC example-import-dv Bound
  Normal   CreatingScratchSpace  31s                datavolume-import-controller  Creating scratch space
  Normal   ImportInProgress      26s (x2 over 35s)  datavolume-import-controller  Import into example-import-dv in progress
  Warning  Error                 17s (x2 over 24s)  datavolume-import-controller  Unable to process data: Unable to transfer source data to scratch space: could not open file "/scratch/tmpimage": open /scratch/tmpimage: permission denied

Beside of "permission denied" there is "error: no kind "DataVolume" error

$ kubectl logs datavolume.cdi.kubevirt.io/example-import-dv
error: no kind "DataVolume" is registered for version "cdi.kubevirt.io/v1beta1" in scheme "pkg/scheme/scheme.go:28"

[akalenyu]

I think you are hitting this minikube issue - kubernetes/minikube#1990
If you want to verify that, you can try to spin up a regular pod and write to a PVC as non-root, as described in the issue

[p-alik]

@akalenyu, I've successfully created both DataVolume and PVC in a cluster-up based k8s environment.
So the problem I've faced has nothing with CDI project.
Thank you.