"Invalid input for field/attribute metadata" when creating a snapshot

[AlwaySummit]

Hi @Lirt

Sorry to trouble you again but I meet an error when I try to create a volume backup. Here are the partial logs:

time="2021-03-22T06:01:00Z" level=info msg="Trying to create snapshot%!(EXTRA string=e7ec5394-114b-4181-9fa0-a81f3c1c6d76.snap.11963748953446345529)" backup=default/velero-backup-pvs-20210322060028 cmd=/plugins/velero-plugin-openstack logSource="/go/src/github.com/Lirt/velero-plugin-openstack/src/cinder/block_store.go:144" pluginName=velero-plugin-openstack
time="2021-03-22T06:01:00Z" level=info msg="1 errors encountered backup up item" backup=default/velero-backup-pvs-20210322060028 logSource="pkg/backup/backup.go:451" name=storage-grafana-loki
time="2021-03-22T06:01:00Z" level=error msg="Error backing up item" backup=default/velero-backup-pvs-20210322060028 error="error taking snapshot of volume: rpc error: code = Unknown desc = Bad request with: [POST https://volume-3.test.com:443/v3/x/snapshots], error message: {\"badRequest\": {\"message\": \"Invalid input for field/attribute metadata. Value: {u'velero.io/pv': u'pvc-6ae4002e-f715-4887-a6b4-41377d984f3c', u'velero.io/storage-location': u'default', u'helm.sh/chart': u'velero-test', u'app.kubernetes.io/name': u'velero', u'velero.io/backup': u'velero-backup-pvs-20210322060028', u'test/is-provisioned': u'True', u'velero.io/schedule-name': u'velero-backup-pvs'}. u'app.kubernetes.io/name', u'test/is-provisioned', u'helm.sh/chart', u'velero.io/backup', u'velero.io/pv', u'velero.io/schedule-name', u'velero.io/storage-location' do not match any of the regexes: '^[a-zA-Z0-9-_:. ]{1,255}$'\", \"code\": 400}}" logSource="pkg/backup/backup.go:455" name=storage-grafana-loki

I did some investigation though, still no explicit solution for this. I guess it is due to the '/' disallowed likely or something else. Moreover, I found this https://bugs.launchpad.net/cinder/+bug/1798798 bug reported for cinder and I'm not sure whether or not it is relevant. Could you please check with that if you have time? Thanks ahead.

[Lirt]

Hi!

no problem I'm here to solve all issues :-).

So you are trying to backup PVC with name pvc-6ae4002e-f715-4887-a6b4-41377d984f3c and the generated snapshot name is e7ec5394-114b-4181-9fa0-a81f3c1c6d76.snap.11963748953446345529.

According to the error message the . is allowed (regex a-zA-Z0-9-_:.) so there should be no problem in the snapshot name, but as you pointed out the properties in metadata contain slashes and the issue you linked looks relevant.

It would be nice if you could confirm this with your cloud provider -- by this I mean which cinder version are they using. Can you please ask them?

The properties are generated by Velero and now I'm not sure if I can change their format. That could break a lot of things (like searching by property) but I will look into the code and see what I can do.

[AlwaySummit]

Hi @Lirt

I reached out to the maintainer and was told that the cinder version is still Rocky(13.0.0). It makes sense because the bug was fixed in 14.0.0.0rc1 release. I suppose we can close this ticket and wait for the upgrate.

[Lirt]

They also mention the fix is in openstack/cinder 13.0.1. That should be more easily upgradeable for them.