velero-plugin-for-gcp [1.13.0]: storage: object doesn't exist:

[maciej-p]

Using velero with gcp bucket.

Upgraded:
velero: 1.16.2 -> 1.17.0
velero-plugin-for-gcp: 1.12.2 -> 1.13.0

example backup:
velero backup create test-ns-$(date +%Y%m%d%H%M%S) --ttl=72h --include-resources pvc,pv --snapshot-move-data=true --include-namespaces test

Backups failing immediately with:

Phase:  Failed (run `velero backup logs test-ns-20251016115526` for more information)

% velero backup logs test-ns-20251016115526
An error occurred: file not found

Status:
  Completion Timestamp:  2025-10-16T09:55:27Z
  Expiration:            2025-10-19T09:55:27Z
  Failure Reason:        error checking if backup already exists in object storage: rpc error: code = Unknown desc = storage: object doesn't exist: googleapi: Error 404: No such object: velero/staging/backups/test-ns-20251016115526/velero-backup.json, notFound
  Format Version:        1.1.0
  Phase:                 Failed
  Start Timestamp:       2025-10-16T09:55:27Z
  Version:               1
Events:                  <none>

After plugin downgrade:
velero-plugin-for-gcp: 1.13.0 -> 1.12.2

Working again

Looks related to: vmware-tanzu/velero-plugin-for-gcp#239

[blackpiglet]

This was not caused by the GCP plugin change.
The failed backup will not be uploaded to the Object Storage.
As a result, you cannot find the related log and metadata for it.

[maciej-p]

Somehow after upgrade GCP plugin all backups not worked, tested multiple times.
After only plugin downgrade to 1.12.2 all working fine.
The issue isn't with missing logs but backups failing after GCP plugin change.

[kaovilai]

can you post velero pod logs during backup and after it fails?

[PrasadJoshi12]

@blackpiglet @kaovilai Looks like the fix is not build in v1.13 tag
vmware-tanzu/velero-plugin-for-gcp#239

[blackpiglet]

@PrasadJoshi12
I see. Please try with velero/velero-plugin-for-gcp:main or velero/velero-plugin-for-gcp:release-1.13-dev.
The fix you mentioned will be included in the future releases v1.13.1 and v1.14.0.

If the main and the release-1.13-dev works, we should close this issue as duplicated to #9268.

[maciej-p]

Executed backup:
velero backup create test-ns-$(date +%Y%m%d%H%M%S) --ttl=72h --include-resources pvc,pv --snapshot-move-data=true --include-namespaces test

Logs on version 1.13.0:

time="2025-10-17T07:21:48Z" level=info msg="Setting up backup log" backup=velero/test-ns-20251017092147 logSource="pkg/controller/backup_controller.go:679"
time="2025-10-17T07:21:48Z" level=info msg="Setting up backup temp file" backup=velero/test-ns-20251017092147 logSource="pkg/controller/backup_controller.go:690"
time="2025-10-17T07:21:48Z" level=info msg="Setting up plugin manager" backup=velero/test-ns-20251017092147 logSource="pkg/controller/backup_controller.go:697"
time="2025-10-17T07:21:48Z" level=info msg="Getting backup item actions" backup=velero/test-ns-20251017092147 logSource="pkg/controller/backup_controller.go:701"
time="2025-10-17T07:21:48Z" level=info msg="Getting ItemBlock actions" backup=velero/test-ns-20251017092147 logSource="pkg/controller/backup_controller.go:706"
time="2025-10-17T07:21:48Z" level=info msg="Setting up backup store to check for backup existence" backup=velero/-test-ns-20251017092147 logSource="pkg/controller/backup_controller.go:711"
time="2025-10-17T07:21:48Z" level=info msg="plugin process exited" backup=velero/test-ns-20251017092147 cmd=/velero id=246 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/velero
time="2025-10-17T07:21:48Z" level=info msg="plugin process exited" backup=velero/test-ns-20251017092147 cmd=/plugins/velero-plugin-for-gcp id=257 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-gcp
time="2025-10-17T07:21:48Z" level=error msg="backup failed" backuprequest=velero/test-ns-20251017092147 controller=backup error="error checking if backup already exists in object storage: rpc error: code = Unknown desc = storage: object doesn't exist: googleapi: Error 404: No such object: velero/staging-eu01/backups/test-ns-20251017092147/velero-backup.json, notFound" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_controller.go:722" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupReconciler).runBackup" logSource="pkg/controller/backup_controller.go:317"
time="2025-10-17T07:21:48Z" level=info msg="Updating backup's status" backuprequest=velero/test-ns-20251017092147 controller=backup logSource="pkg/controller/backup_controller.go:336"

With image: velero/velero-plugin-for-gcp:main all good.

[blackpiglet]

@maciej-p
Thanks for the confirmation, then I will close this issue for now.

[Nexus2k]

What's the rationale of closing an unfixed issue here? please trigger a 1.13.1 release of the gcp-plugin.

[blackpiglet]

Velero-plugin-for-gcp v1.13.1 has been released and is available at https://github.com/vmware-tanzu/velero-plugin-for-gcp/releases/tag/v1.13.1.

We understand that closing an issue before a working version is released may seem unreasonable. Our process involves linking issues to specific versions using a label. All related issues are closed before a tag is created for the version. This helps us avoid the cumbersome task of triaging open issues every time, especially with the constant influx of new reports.

Thank you for your patience and effort.