when changing storageclass, it doesn't change volume type in Openstack

[mikkeschiren]

Describe the bug

When using snapshots, and adding config for changing storageclass when restoring, in Kubernetes it looks like it has the right storageclass, but in reality, when checking in openstack, it has the same volume type as before.

Steps to reproduce the behavior

Create a Velero backup from a namespace using snapshot as method for volumeSnapshotLocation. Existing volumeclass is "foo", apply velero config for changing storageclass on restore to "bar".
Delete volume and deployment using that volume.
Restore the namespace.
New volume created using storageclass bar.
Check in openstack, volume has volume type foo.

Expected behavior
When using restore and you have configured a different restore storageclass, the restore should be using that volume type in openstack.

Used versions

• Velero version(velero version): 1.13.2
• Plugin version(kubectl describe pod velero-...): v0.7.0
• Kubernetes version(kubectl version): v1.26.15

If this is expected behaviour, that we can't change storageclass when using snapshots, it should be documented.

[mikkeschiren]

Velero conf for changing storageclass as reference:

apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storageclass
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  foo: bar

[Lirt]

Hello @mikkeschiren,

I will look into this soon.

[Lirt]

Hello again,

When restore is executed it should do following (correct me if I'm wrong please):

1. Velero finds PV/PVC to be restored and executes RestoreItemAction (https://velero.io/docs/v1.14/restore-reference/#detailed-restore-workflow).
2. It will find there is a backup for the PV (in your case a snapshot).
3. It will try to create new volume from this snapshot (CreateVolumeFromSnapshot()).
4. This part of the code should log volumeType: "bar" (could you please check this happens and what is volumeType?)

	logWithFields := b.log.WithFields(logrus.Fields{
		"snapshotID":      snapshotID,
		"volumeType":      volumeType,
		"volumeAZ":        volumeAZ,
		"snapshotTimeout": b.snapshotTimeout,
		"volumeTimeout":   b.volumeTimeout,
		"method":          b.config["method"],
	})
	logWithFields.Info("BlockStore.CreateVolumeFromSnapshot called")

5. Then I see everything correct regarding volumeType in volume.create in the code.

	// Create Cinder Volume from snapshot (backup)
	logWithFields.Info("Starting to create volume from snapshot")
	opts := volumes.CreateOpts{
		Description:      "Velero backup from snapshot",
		Name:             volumeName,
		VolumeType:       volumeType,
		AvailabilityZone: volumeAZ,
		SnapshotID:       snapshotID,
		Metadata:         originVolume.Metadata,
	}

	volume, err := volumes.Create(b.client, opts).Extract()

I remember this older issue #46 also had problem with a storage-class rename. I will need to dig into the code again to explore what's happening.

It would be however nice if you could try to tell me what does Velero log say about volumeType when volume from snapshot is created. I bet Velero sends unchanged storage class name (unchanged volumeType), because in v0.7.0 we are correctly passing volumeType from the function argument towards openstack volume.create.

[mikkeschiren]

I will do a new try in a day or two, and will come back with the output.

[mikkeschiren]

Here is the output:

time="2024-07-24T12:17:13Z" level=info msg="Restoring persistent volume from snapshot." logSource="pkg/restore/restore.go:2460" restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:13Z" level=info msg="BlockStore.Init called" cmd=/plugins/velero-plugin-for-openstack config="map[backupTimeout:5m cascadeDelete:true cloneTimeout:5m cloud: ensureDeleted:true ensureDeletedDelay:10s imageTimeout:5m method:snapshot region: snapshotTimeout:5m volumeTimeout:5m]" logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/cinder/block_store.go:117" pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:13Z" level=info msg="Authentication will be done for cloud " cmd=/plugins/velero-plugin-for-openstack logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/utils/auth.go:33" pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:13Z" level=info msg="Trying to authenticate against OpenStack using environment variables (including application credentials) or using files ~/.config/openstack/clouds.yaml, /etc/openstack/clouds.yaml and ./clouds.yaml" cmd=/plugins/velero-plugin-for-openstack logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/utils/auth.go:68" pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:14Z" level=info msg="Authentication against identity endpoint https://OPENSTACKENDPOINT:5000/v3/ was successful" cmd=/plugins/velero-plugin-for-openstack logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/utils/auth.go:113" pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:14Z" level=info msg="Successfully created block storage service client" cmd=/plugins/velero-plugin-for-openstack endpoint="https://OPENSTACKENDPOINT:8776/v3/ef9331419364434d866ea9c7fd062430/" logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/cinder/block_store.go:218" pluginName=velero-plugin-for-openstack region=se-sto restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:14Z" level=info msg="BlockStore.CreateVolumeFromSnapshot called" cmd=/plugins/velero-plugin-for-openstack logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/cinder/block_store.go:249" method=snapshot pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648 snapshotID=dff44a58-b639-484c-9454-603f07f78b79 snapshotTimeout=300 volumeAZ=sto2 volumeTimeout=300 volumeType=foo
time="2024-07-24T12:17:14Z" level=info msg="Waiting for snapshot to be in 'available' state" cmd=/plugins/velero-plugin-for-openstack logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/cinder/block_store.go:253" method=snapshot pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648 snapshotID=dff44a58-b639-484c-9454-603f07f78b79 snapshotTimeout=300 volumeAZ=sto2 volumeTimeout=300 volumeType=foo
time="2024-07-24T12:17:15Z" level=info msg="Snapshot is in 'available' state" cmd=/plugins/velero-plugin-for-openstack logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/cinder/block_store.go:260" method=snapshot pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648 snapshotID=dff44a58-b639-484c-9454-603f07f78b79 snapshotTimeout=300 volumeAZ=sto2 volumeTimeout=300 volumeType=foo
time="2024-07-24T12:17:15Z" level=info msg="Starting to create volume from snapshot" cmd=/plugins/velero-plugin-for-openstack logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/cinder/block_store.go:270" method=snapshot pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648 snapshotID=dff44a58-b639-484c-9454-603f07f78b79 snapshotTimeout=300 volumeAZ=sto2 volumeTimeout=300 volumeType=foo
time="2024-07-24T12:17:18Z" level=info msg="Backup volume was created" cmd=/plugins/velero-plugin-for-openstack logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/cinder/block_store.go:294" method=snapshot pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648 snapshotID=dff44a58-b639-484c-9454-603f07f78b79 snapshotTimeout=300 volumeAZ=sto2 volumeID=b29b0df6-e841-4717-86e7-df874c4d1a0b volumeTimeout=300 volumeType=foo
time="2024-07-24T12:17:18Z" level=info msg="successfully restored persistent volume from snapshot" logSource="pkg/restore/pv_restorer.go:85" persistentVolume=pvc-8a5fa95c-88b7-4ebd-861a-83f93594ee51 providerSnapshotID=dff44a58-b639-484c-9454-603f07f78b79 restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="BlockStore.SetVolumeID called" cmd=/plugins/velero-plugin-for-openstack logSource="/go/src/github.com/Lirt/velero-plugin-for-openstack/src/cinder/block_store.go:889" pluginName=velero-plugin-for-openstack restore=velero/my-namespace-20240724141648 unstructuredPV="map[apiVersion:v1 kind:PersistentVolume metadata:map[annotations:map[pv.kubernetes.io/provisioned-by:cinder.csi.openstack.org volume.kubernetes.io/provisioner-deletion-secret-name: volume.kubernetes.io/provisioner-deletion-secret-namespace:] creationTimestamp:2024-07-10T08:47:38Z finalizers:[kubernetes.io/pv-protection external-attacher/cinder-csi-openstack-org] managedFields:[map[apiVersion:v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:pv.kubernetes.io/provisioned-by:map[] f:volume.kubernetes.io/provisioner-deletion-secret-name:map[] f:volume.kubernetes.io/provisioner-deletion-secret-namespace:map[]]] f:spec:map[f:accessModes:map[] f:capacity:map[.:map[] f:storage:map[]] f:claimRef:map[.:map[] f:apiVersion:map[] f:kind:map[] f:name:map[] f:namespace:map[] f:resourceVersion:map[] f:uid:map[]] f:csi:map[.:map[] f:driver:map[] f:fsType:map[] f:volumeAttributes:map[.:map[] f:storage.kubernetes.io/csiProvisionerIdentity:map[]] f:volumeHandle:map[]] f:nodeAffinity:map[.:map[] f:required:map[]] f:persistentVolumeReclaimPolicy:map[] f:storageClassName:map[] f:volumeMode:map[]]] manager:csi-provisioner operation:Update time:2024-07-10T08:47:38Z] map[apiVersion:v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:phase:map[]]] manager:kube-controller-manager operation:Update subresource:status time:2024-07-10T08:47:38Z] map[apiVersion:v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[v:\"external-attacher/cinder-csi-openstack-org\":map[]]]] manager:csi-attacher operation:Update time:2024-07-10T08:47:39Z]] name:pvc-8a5fa95c-88b7-4ebd-861a-83f93594ee51 resourceVersion:542359793 uid:d7835922-c773-4051-8792-3215e16f71bb] spec:map[accessModes:[ReadWriteOnce] capacity:map[storage:8Gi] claimRef:map[apiVersion:v1 kind:PersistentVolumeClaim name:redis-data-directus-redis-master-0 namespace:my-namespace] csi:map[driver:cinder.csi.openstack.org fsType:ext4 volumeAttributes:map[storage.kubernetes.io/csiProvisionerIdentity:1719500833361-8081-cinder.csi.openstack.org] volumeHandle:eedea675-50ad-4f6c-93be-de52f4a0b799] nodeAffinity:map[required:map[nodeSelectorTerms:[map[matchExpressions:[map[key:topology.cinder.csi.openstack.org/zone operator:In values:[sto2]]]]]]] persistentVolumeReclaimPolicy:Delete storageClassName:foo volumeMode:Filesystem] status:map[phase:Bound]]" volumeID=b29b0df6-e841-4717-86e7-df874c4d1a0b
time="2024-07-24T12:17:18Z" level=info msg="restore status includes excludes: <nil>" logSource="pkg/restore/restore.go:1304" restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:1318" restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/actions/change_storageclass_action.go:68" pluginName=velero restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Updating item's storage class name to bar" cmd=/velero kind=PersistentVolume logSource="pkg/restore/actions/change_storageclass_action.go:134" name=pvc-8a5fa95c-88b7-4ebd-861a-83f93594ee51 namespace= pluginName=velero restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/actions/change_storageclass_action.go:140" pluginName=velero restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Attempting to restore PersistentVolume: pvc-8a5fa95c-88b7-4ebd-861a-83f93594ee51" logSource="pkg/restore/restore.go:1487" restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="the managed fields for pvc-8a5fa95c-88b7-4ebd-861a-83f93594ee51 is patched" logSource="pkg/restore/restore.go:1688" restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1318" restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Executing ChangePVCNodeSelectorAction" cmd=/velero logSource="pkg/restore/actions/change_pvc_node_selector.go:66" pluginName=velero restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Done executing ChangePVCNodeSelectorAction" cmd=/velero logSource="pkg/restore/actions/change_pvc_node_selector.go:138" pluginName=velero restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1318" restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/actions/change_storageclass_action.go:68" pluginName=velero restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Updating item's storage class name to bar" cmd=/velero kind=PersistentVolumeClaim logSource="pkg/restore/actions/change_storageclass_action.go:134" name=redis-data-directus-redis-master-0 namespace=my-namespace pluginName=velero restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/actions/change_storageclass_action.go:140" pluginName=velero restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Skip action velero.io/csi-pvc-restorer for resource persistentvolumeclaims:my-namespace/redis-data-directus-redis-master-0, because the CSI feature is not enabled. Feature setting is ." logSource="pkg/restore/restore.go:1313" restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="Attempting to restore PersistentVolumeClaim: redis-data-directus-redis-master-0" logSource="pkg/restore/restore.go:1487" restore=velero/my-namespace-20240724141648
time="2024-07-24T12:17:18Z" level=info msg="the managed fields for my-namespace/redis-data-directus-redis-master-0 is patched" logSource="pkg/restore/restore.go:1688" restore=velero/my-namespace-20240724141648

Some extra information that could help:

• The Volume that is created in k8s, says storageclass bar
• The Volume in Openstack is created as foo (as the original)
• The volume in openstack gets a"wrong" name - it get named "dff44a58-b639-484c-9454-603f07f78b79.backup.2907897963052842866" instead of expected - like "pvc-8a5fa95c-88b7-4ebd-861a-83f93594ee51"

[mikkeschiren]

Here is the helm values I use, with secrets removed:

credentials:
  extraSecretRef: "velero-credentials"
  extraEnvVars:
    OS_AUTH_URL:
    [...]
configuration:
  backupStorageLocation:
  - name: swift
    provider: community.openstack.org/openstack
    bucket: velero
  volumeSnapshotLocation:
  - name: cinder
    provider: community.openstack.org/openstack-cinder
    config:
      cloud: ""
      region: ""
      method: snapshot
      volumeTimeout: 5m
      snapshotTimeout: 5m
      cloneTimeout: 5m
      backupTimeout: 5m
      imageTimeout: 5m
      ensureDeleted: "true"
      ensureDeletedDelay: 10s
      cascadeDelete: "true"
initContainers:
- name: velero-plugin-openstack
  image: lirt/velero-plugin-for-openstack:v0.7.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins
snapshotsEnabled: true
backupsEnabled: true

[Lirt]

I will need more time to look into this. Thank you for patience.

[Lirt]

I created issue in Velero - vmware-tanzu/velero#8164.

I'm afraid based on the code I studied that it is impossible in current Velero code to change volume type of restored volume using change storage class action. Let's see what does Velero community think about it. Maybe it could be considered a bug 🤷.