Backups are failing in velero 1.17 with GCP plugin

[PrasadJoshi12]

What steps did you take and what happened:

Installed velero 1.17 and triggered a backup. Backup resulted in failure attached logs below. This was a fresh cluster with empty bucket.

status:
    completionTimestamp: "2025-09-16T16:21:39Z"
    expiration: "2025-10-16T16:21:38Z"
    failureReason: 'error checking if backup already exists in object storage: rpc
      error: code = Unknown desc = storage: object doesn''t exist: googleapi: Error
      404: No such object: oadp129131mgkpl/backups/test-backup8/velero-backup.json,
      notFound'
    formatVersion: 1.1.0
    phase: Failed
    startTimestamp: "2025-09-16T16:21:38Z"

Tested with latest and v1.13.0 gcp plugin image
velero install --provider gcp --image velero/velero:v1.17.0 --plugins velero/velero-plugin-for-gcp:latest,quay.io/konveyor/openshift-velero-plugin:latest --bucket oadp129131mgkpl --secret-file /tmp/credentials1 --use-node-agent --use-volume-snapshots=false --privileged-node-agent --features=EnableCSI

What did you expect to happen:

Backup should be completed successfully

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version): Velero 1.17
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): OCP 4.19
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[PrasadJoshi12]

Opened a PR
vmware-tanzu/velero-plugin-for-gcp#239

[PrasadJoshi12]

The same thing happens with minikube cluster.

velero install --provider gcp --image docker.io/velero/velero:v1.17.0 --plugins docker.io/velero/velero-plugin-for-gcp:latest --bucket oadp12933164q7g --secret-file /tmp/credentials --use-volume-snapshots=false --features=EnableCSI

$ kubectl get bsl -n velero
NAME      PHASE       LAST VALIDATED   AGE   DEFAULT
default   Available   31s              17m   true
prajoshi@prajoshi-mac ~ % kubectl get ns
NAME              STATUS   AGE
default           Active   101m
kube-node-lease   Active   101m
kube-public       Active   101m
kube-system       Active   101m
test-12           Active   100s
test-backup       Active   29m
velero            Active   19m
 ~ % kubectl get pod -n velero
NAME                     READY   STATUS    RESTARTS   AGE
velero-959499cd8-6mmh2   1/1     Running   1          19m
 ~ % kubectl get bsl -n velero
NAME      PHASE       LAST VALIDATED   AGE   DEFAULT
default   Available   29s              20m   true
 ~ % kubectl get backup -n velero
NAME            AGE
test-backup34   2m5s

                                                                                        
 ~ % kubectl get backup -n velero test-backup34 -ojsonpath={.status} | jq
{
  "completionTimestamp": "2025-09-17T09:26:34Z",
  "expiration": "2025-10-17T09:26:32Z",
  "failureReason": "error checking if backup already exists in object storage: rpc error: code = Unknown desc = storage: object doesn't exist: googleapi: Error 404: No such object: oadp12933164q7g/backups/test-backup34/velero-backup.json, notFound",
  "formatVersion": "1.1.0",
  "phase": "Failed",
  "startTimestamp": "2025-09-17T09:26:32Z",
  "version": 1
}

[blackpiglet]

Did any GCP object storage API change?
I don't think this error was caused by the Velero or plugin version bump.
Why did this happen recently?

[PrasadJoshi12]

@blackpiglet I think it's changed in v1.51.0
googleapis/google-cloud-go#11519

[kaovilai]

So vmware-tanzu/velero-plugin-for-gcp@0f491aa#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R6

[kaovilai]

cp pr vmware-tanzu/velero-plugin-for-gcp#240

[PrasadJoshi12]

Thanks @kaovilai !

[Lyndon-Li]

Closing this issue as PRs are merged into main and 1.13 for gcp-plugin.

[Kerruba]

Hello, I'm still getting this error while upgrading Velero to 1.17 wit the GCP plugin 1.13.

I see there's a discrepancy between tag 1.13 and branch release-1.13 code:

• tag 1.13 still uses the failing login: https://github.com/vmware-tanzu/velero-plugin-for-gcp/blob/2632dc18e3f014ca9748521730071573b63f1470/velero-plugin-for-gcp/object_store.go#L236
• release1.13 uses the new logic: https://github.com/vmware-tanzu/velero-plugin-for-gcp/blob/edc2907774a3fb8e13791d0d08d5ee0feaa9b762/velero-plugin-for-gcp/object_store.go#L236

I saw the Main CI ran 3-4 days ago (main/release-1.13 branches), but on dockerhub tag v1.13.0 has been created 7 days ago, while tag release-1.13-dev was created 4 days ago.

Does the fix still need to be released?