Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PVC migration fails when migrating to csi #45

Open
rajeshkodali-sf opened this issue Aug 4, 2023 · 5 comments
Open

PVC migration fails when migrating to csi #45

rajeshkodali-sf opened this issue Aug 4, 2023 · 5 comments

Comments

@rajeshkodali-sf
Copy link

pv-migrator error out when copying to csi storage class setup on same blockpool

Here is the ceph version and rook version:

$ ceph version

ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)

rook version: 1.7.11

Here is the non-csi Storageclass definition:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"rook-ceph-block"},"parameters":{"blockPool":"blockpool001","clusterNamespace":"rook-ceph","fstype":"xfs"},"provisioner":"ceph.rook.io/block"}
  creationTimestamp: "2022-05-06T19:51:18Z"
  name: rook-ceph-block
  resourceVersion: "218730968"
  uid: bbc5ad00-3e59-4e3f-9363-3708e5024d08
parameters:
  blockPool: blockpool001
  clusterNamespace: rook-ceph
  fstype: xfs
provisioner: ceph.rook.io/block
reclaimPolicy: Delete
volumeBindingMode: Immediate

Here is CSI StorageClass:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"rook-ceph-block-csi"},"parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/controller-expand-secret-name":"rook-csi-rbd-provisioner","csi.storage.k8s.io/controller-expand-secret-namespace":"rook-ceph","csi.storage.k8s.io/fstype":"xfs","csi.storage.k8s.io/node-stage-secret-name":"rook-csi-rbd-node","csi.storage.k8s.io/node-stage-secret-namespace":"rook-ceph","csi.storage.k8s.io/provisioner-secret-name":"rook-csi-rbd-provisioner","csi.storage.k8s.io/provisioner-secret-namespace":"rook-ceph","imageFeatures":"layering","imageFormat":"2","pool":"blockpool001"},"provisioner":"rook-ceph-system.rbd.csi.ceph.com","reclaimPolicy":"Delete"}
  creationTimestamp: "2023-08-02T21:40:19Z"
  name: rook-ceph-block-csi
  resourceVersion: "473173630"
  uid: a724514f-951c-43fd-a123-a049538f9ab5
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: xfs
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  imageFeatures: layering
  imageFormat: "2"
  pool: blockpool001
provisioner: rook-ceph-system.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Error:

$ pv-migrator --pvc=prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 --pvc-ns=fabric --destination-sc=rook-ceph-block-csi --rook-ns=rook-ceph-system
I0803 19:22:04.686852   25077 log.go:34] Create Kubernetes Client
I0803 19:22:04.713324   25077 log.go:34] List all the PVC from the source storageclass
I0803 19:22:04.716064   25077 log.go:34] 1 PVCs found with source StorageClass
I0803 19:22:04.716097   25077 log.go:34] Start Migration of PVCs to CSI
I0803 19:22:04.716111   25077 log.go:34] migrating PVC "prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2" from namespace "sf-fabric"
I0803 19:22:04.716122   25077 log.go:34] Fetch PV information from PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2
I0803 19:22:04.720047   25077 log.go:34] PV found "pvc-41e06349-b07c-4662-941c-f38d914eabc4"
I0803 19:22:04.720068   25077 log.go:34] Update Reclaim policy from Delete to Reclaim for PV: pvc-41e06349-b07c-4662-941c-f38d914eabc4
I0803 19:22:04.725395   25077 log.go:34] Retrieving old ceph volume name from PV object: pvc-41e06349-b07c-4662-941c-f38d914eabc4
I0803 19:22:04.725416   25077 log.go:34] rbd image name is "pvc-41e06349-b07c-4662-941c-f38d914eabc4"
I0803 19:22:04.725420   25077 log.go:34] Deleting pvc object: prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2
I0803 19:22:04.729446   25077 log.go:34] waiting for PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{161061273600 0} {<nil>} 150Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed)
I0803 19:22:06.732848   25077 log.go:34] waiting for PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{161061273600 0} {<nil>} 150Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed)
I0803 19:22:06.736645   25077 log.go:34] Generate new PVC with same name in destination storageclass
I0803 19:22:06.736736   25077 log.go:34] Create new csi pvc
I0803 19:22:06.744044   25077 log.go:34] Waiting up to &PersistentVolumeClaim{ObjectMeta:{prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2  sf-fabric  41e06349-b07c-4662-941c-f38d914eabc4  0 2023-08-03 19:21:23 +0000 UTC <nil> <nil> map[app:prometheus app.kubernetes.io/instance:fabric-layer-1 app.kubernetes.io/managed-by:prometheus-operator app.kubernetes.io/name:prometheus app.kubernetes.io/version:2.27.1 operator.prometheus.io/name:fabric-layer-1 operator.prometheus.io/shard:0 prometheus:fabric-layer-1] map[] [] [kubernetes.io/pvc-protection]  [{kube-controller-manager Update v1 2023-08-03 19:21:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}}} {kube-controller-manager Update v1 2023-08-03 19:21:23 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}}} {pv-migrator Update v1 2023-08-03 19:21:23 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"kubernetes.io/pvc-protection\"":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/version":{},"f:operator.prometheus.io/name":{},"f:operator.prometheus.io/shard":{},"f:prometheus":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}}}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{161061273600 0} {<nil>} 150Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*rook-ceph-block-csi,VolumeMode:*Filesystem,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state
I0803 19:22:06.744078   25077 log.go:34] waiting for PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 (0 seconds elapsed)
I0803 19:22:08.748111   25077 log.go:34] waiting for PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 (2 seconds elapsed)
I0803 19:22:08.753034   25077 log.go:34] Waiting for PV "pvc-4566e992-8290-4787-b4ea-5c7b61da6634" to bind to PVC "prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2"
I0803 19:22:08.753070   25077 log.go:34] Waiting up to 30ms for PersistentVolumeClaims [prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2] to have phase Bound
I0803 19:22:08.755251   25077 log.go:34] PersistentVolumeClaim prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 found and phase=Bound (2.162732ms)
I0803 19:22:08.755269   25077 log.go:34] Waiting up to 3s for PersistentVolume pvc-4566e992-8290-4787-b4ea-5c7b61da6634 to have phase Bound
I0803 19:22:08.757184   25077 log.go:34] PersistentVolume pvc-4566e992-8290-4787-b4ea-5c7b61da6634 found and phase=Bound (1.905595ms)
I0803 19:22:08.764451   25077 log.go:34] New PVC with same name "prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2" created via CSI
I0803 19:22:08.764467   25077 log.go:34] Extracting new volume name from CSI PV
I0803 19:22:08.764486   25077 log.go:34] CSI new volume name: csi-vol-08d0af44-3233-11ee-94b3-2e7d69f465aa
I0803 19:22:08.764500   25077 log.go:34] Fetching csi pool name
I0803 19:22:08.764517   25077 log.go:34] csi poolname: blockpool001
I0803 19:22:08.764529   25077 log.go:34] Create new Ceph connection
I0803 19:22:08.764541   25077 log.go:34] csi poolname: blockpool001
I0803 19:22:08.766817   25077 log.go:34] clusterID: rook-ceph, monitors: 10.3.238.27:6789,10.3.159.40:6789,10.3.237.157:6789,10.3.213.17:6789,10.3.215.174:6789, poolname: blockpool001
I0803 19:22:08.769419   25077 log.go:34] New connection arg monitors: 10.3.238.27:6789,10.3.159.40:6789,10.3.237.157:6789,10.3.213.17:6789,10.3.215.174:6789, id: csi-rbd-provisioner, keyfile: /tmp/csi/keys/keyfile-779656205, pool: blockpool001, datapool:
I0803 19:22:08.769554   25077 log.go:34] Cluster connection created
I0803 19:22:08.769570   25077 log.go:34] Delete the placeholder CSI volume in ceph cluster
I0803 19:22:15.784896   25077 log.go:34] Successfully removed volume csi-vol-08d0af44-3233-11ee-94b3-2e7d69f465aa
I0803 19:22:15.784920   25077 log.go:34] Rename old ceph volume to new CSI volume
Error: failed to migrate PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 : failed to rename old ceph volume pvc-41e06349-b07c-4662-941c-f38d914eabc4 to new CSI volume csi-vol-08d0af44-3233-11ee-94b3-2e7d69f465aa: exit status 2. failed to rename rbd image, command output: rbd: rename error: (2) No such file or directory
2023-08-03T19:22:15.828+0000 7ff562ead2c0 -1 librbd: error opening source image: (2) No such file or directory

What could be the issue? How do I troubleshoot?
@rajeshkodali-sf rajeshkodali-sf mentioned this issue Aug 4, 2023
Closed
@subhamkrai
Copy link
Collaborator

@rajeshkodali-sf can you try running this command manual

ceph rename <oldImageName> <newImageName> --pool  <poolName> --id  <CephRbdsecretUser> -m  <monIp's> --keyfile=<cephRBDSecretFile>

@rajeshkodali-sf
Copy link
Author

I don't see the old image pvc-41e06349-b07c-4662-941c-f38d914eabc4 mentioned in the error above

cephtoolbox rbd ls blockpool001
csi-vol-34a41cf3-317d-11ee-94b3-2e7d69f465aa
csi-vol-a3a6d817-317d-11ee-94b3-2e7d69f465aa
csi-vol-dd9646c5-317d-11ee-94b3-2e7d69f465aa
csi-vol-eed3d8b2-3232-11ee-94b3-2e7d69f465aa
csi-vol-f8ecea2c-3220-11ee-94b3-2e7d69f465aa
pvc-117d3087-b6a0-4fe0-b2b3-fd9e99ad4a27
pvc-23423bc7-1860-4144-94e0-e95fd16792b2
pvc-34da41d9-69a4-4a30-80b4-804ccb66a7a1
pvc-39210f75-5ed3-4e78-b170-e466c22f9b34
pvc-5e2dc1fd-0abe-4c0a-ada4-e850c31d8d76
pvc-65ae19e6-8f90-469d-9ad3-5bbcd61f2c47
pvc-a9571f29-7425-4e23-8069-321a5afae4bc
pvc-bff7b4f2-66da-4c5d-9888-ce8d4079928a
pvc-ee24b0f1-3394-4a06-8d5a-cf2530b64453
pvc-f2507454-9cb6-41ae-bdb3-792f018e3ca1
pvc-f681d35a-607f-458c-839a-840679570816

@subhamkrai
Copy link
Collaborator

@rajeshkodali-sf I would like to repro this. Could you help me with yamls files I could use to repo this?

@rajeshkodali-sf
Copy link
Author

@subhamkrai let me know which yamls? All cluster, operator and block?

@subhamkrai
Copy link
Collaborator

@rajeshkodali-sf to create the pvc you have

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@subhamkrai @rajeshkodali-sf