gzip: invalid header - persistence.decode function

[amayacitta]

What steps did you take and what happened:

Hi guys, we have done a clean install of 1.15.0 with the following, back end storage is Cohesity. There is no publicUrl we're aware of with the Cohesity implementation.

velero install
--provider aws
--plugins velero/velero-plugin-for-aws:v1.11.0
--bucket Tanzu-Dev-Test-TKG-Backups
--backup-location-config region=cohesity,s3ForcePathStyle="true",s3Url=https://s3.maskedexample.com:3000
--secret-file velero/credentials-velero
--use-volume-snapshots=false
--namespace infra-velero
--cacert velero/subcacert.crt

Logs show success for backup location, so does velero backup-location get.

S3 bucket contains the files which we can see with aws client or S3 browser client. Download is not possible via velero client it seems. We are aware of s3ForcePathStyle=true but have this set. We are also aware of publicUrl but this doesnt exist for the Cohesity S3 setup.

time="2024-12-16T21:13:53Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=infra-velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:127"

We then run a simple backup:
velero backup create test-backup --include-namespaces=developer-ext-amayacitta-gill --exclude-cluster-scoped-resources=true --snapshot-volumes=false

What did you expect to happen:
Backup completes, however we get this debug log thrown. A larger sample is pasted below, debug file also attached.

time="2024-12-16T21:13:53Z" level=error msg="Error getting backup item operations" backup=infra-velero/test-backup controller=backup-finalizer error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:416" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="pkg/controller/backup_finalizer_controller.go:155"

---

time="2024-12-16T21:13:53Z" level=error msg="Reconciler error" Backup="{"name":"test-backup","namespace":"infra-velero"}" controller=backup controllerGroup=velero.io controllerKind=Backup error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:416" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="/go/pkg/mod/github.com/bombsimon/logrusr/[email protected]/logrusr.go:123" name=test-backup namespace=infra-velero reconcileID=""183f5531-8938-477e-85e7-3cb58485ddd7""
time="2024-12-16T21:13:53Z" level=debug msg="Getting Backup" backup=infra-velero/test-backup controller=backup-finalizer logSource="pkg/controller/backup_finalizer_controller.go:95"
time="2024-12-16T21:13:53Z" level=debug msg="looking for plugin in registry" backup=infra-velero/test-backup controller=backup-finalizer kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:141" name=velero.io/aws
time="2024-12-16T21:13:53Z" level=debug msg="creating new restartable plugin process" backup=infra-velero/test-backup command=/plugins/velero-plugin-for-aws controller=backup-finalizer kind=ObjectStore logSource="pkg/plugin/clientmgmt/manager.go:156" name=velero.io/aws
time="2024-12-16T21:13:53Z" level=debug msg="starting plugin" args="[/plugins/velero-plugin-for-aws --features= --uploader-type=kopia --log-level debug]" backup=infra-velero/test-backup cmd=/plugins/velero-plugin-for-aws controller=backup-finalizer logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" path=/plugins/velero-plugin-for-aws
time="2024-12-16T21:13:53Z" level=debug msg="plugin started" backup=infra-velero/test-backup cmd=/plugins/velero-plugin-for-aws controller=backup-finalizer logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" path=/plugins/velero-plugin-for-aws pid=52
time="2024-12-16T21:13:53Z" level=debug msg="waiting for RPC address" backup=infra-velero/test-backup cmd=/plugins/velero-plugin-for-aws controller=backup-finalizer logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" plugin=/plugins/velero-plugin-for-aws
time="2024-12-16T21:13:53Z" level=debug msg="Setting log level to DEBUG" backup=infra-velero/test-backup cmd=/plugins/velero-plugin-for-aws controller=backup-finalizer logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/plugin/framework/server.go:269" pluginName=velero-plugin-for-aws
time="2024-12-16T21:13:53Z" level=debug msg="plugin address" address=/tmp/plugin948232077 backup=infra-velero/test-backup cmd=/plugins/velero-plugin-for-aws controller=backup-finalizer logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" network=unix pluginName=velero-plugin-for-aws
time="2024-12-16T21:13:53Z" level=debug msg="using plugin" backup=infra-velero/test-backup cmd=/plugins/velero-plugin-for-aws controller=backup-finalizer logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" version=2
time="2024-12-16T21:13:53Z" level=debug msg="waiting for stdio data" backup=infra-velero/test-backup cmd=/plugins/velero-plugin-for-aws controller=backup-finalizer logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:75" pluginName=stdio
time="2024-12-16T21:13:53Z" level=debug msg="Checking if object exists" backup=infra-velero/test-backup bucket=Tanzu-Dev-Test-TKG-Backups cmd=/plugins/velero-plugin-for-aws controller=backup-finalizer key=backups/test-backup/test-backup-itemoperations.json.gz logSource="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:303" pluginName=velero-plugin-for-aws
time="2024-12-16T21:13:53Z" level=debug msg="Object exists" backup=infra-velero/test-backup bucket=Tanzu-Dev-Test-TKG-Backups cmd=/plugins/velero-plugin-for-aws controller=backup-finalizer key=backups/test-backup/test-backup-itemoperations.json.gz logSource="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:319" pluginName=velero-plugin-for-aws
time="2024-12-16T21:13:53Z" level=error msg="Error getting backup item operations" backup=infra-velero/test-backup controller=backup-finalizer error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:416" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="pkg/controller/backup_finalizer_controller.go:155"

The following information will help us better understand what's going on:

bundle-2024-12-16-21-36-35.tar.gz

Environment:

• Cohesity 7.1.1
• Velero 1.15.0
• No additional features
• Kubernetes: v1.29.4+vmware.3-fips.1
• Tanzu TKGS cluster on latest vSphere 8
• VMware Photon OS/Linux

[reasonerjt]

@amayacitta

It looks like you're a VMware(by Broadcom) customer, I suggest you contact GSS support of VMware(by Broadcom) to have official support.

As for the specific problem you are seeing, I feel it MAY be compatibility issues of Cohesity or some network issue that the .gz file is not served correctly from Cohesity to velero via S3 API. Could you test velero with other storage service like MinIO or AWS S3?

Have you used velero on the same storage before? If the answer is "yes", please provide the versions of velero and the plugin. velero-plugin-for-aws v1.10.x and later releases use S3 SDK v2 and we have observed some storage services are seeing compatibility issue b/c their "S3-Compatible" APIs are acting differently from AWS S3.

More details see: https://github.com/vmware-tanzu/velero-plugin-for-aws

[amayacitta]

Hi @reasonerjt - We are indeed a commercial customer and are in process of logging the ticket. We're also logging one with Cohesity to see if there is anything they can do. I'll keep this issue updated so there is visibility of the fix.

For example we also use Cohesity for Grafana Loki and that works with the following connection details.

  s3: s3://xxxx:xxxx@cohesity/Tanzu-Dev-Test-TKG-Loki-Chunk
  s3forcepathstyle: false
  endpoint: https://s3.maskedexample.com:3000/

S3 on Cohesity also works generally for everything else we've thrown at it, it's only Velero that has an issue - is there anything that can be done the the below go code to make it work?

error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:416" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="pkg/controller/backup_finalizer_controller.go:155"

I can use AWS cli to download the files from cohesity, as well as a vanilla S3 browser client. Its seems to be an issue unique to the Velero client. This is the first time we're trying Velero on Cohesity.

I'll try again with s3ForcePathStyle="false" as we do with Loki.

velero install
--provider aws
--plugins velero/velero-plugin-for-aws:v1.11.0
--bucket Tanzu-Dev-Test-TKG-Backups
--backup-location-config region=cohesity,s3ForcePathStyle="false",s3Url=https://s3.maskedexample.com:3000/
--secret-file velero/credentials-velero
--use-volume-snapshots=false
--namespace infra-velero
--cacert velero/subcacert.crt