velero 1.17.0 with gcp plugin 1.13.0 not writing to storage bucket

[Nexus2k]

What steps did you take and what happened:
I've upgraded velero to 1.17.0 and the gcp plugin to 1.13.0 and when I tried to run a backup it immediately failed with an error that the velero-backup.json file can't be found on the storage path.

What did you expect to happen:
I expect that the gcp-plugin works with velero 1.17.0 according to the compability matrix

The following information will help us better understand what's going on:

Reproduction steps:

1. Setup velero 1.17.0 with gcp-plugin 1.13.0 using service accounts for the access to storage bucket
2. Try to start a backup
3. See the error about 404 velero-backup.json not found

Anything else you would like to add:

values.yaml:

initContainers:
  - name: velero-plugin-for-gcp
    image: velero/velero-plugin-for-gcp:v1.12.2
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins

upgradeCRDs: true

configuration:
  backupStorageLocation:
    - provider: "gcp"
      # bucket is the name of the bucket to store backups in. Required.
      bucket: "<my-project-id>-velero-backup"
      prefix: "prod"
      accessMode: ReadWrite
      config:
        serviceAccount: velero-backup@<my-project-id>.iam.gserviceaccount.com

  volumeSnapshotLocation:
    - provider: "gcp"
      config:
        project: <my-project-id>

serviceAccount:
  server:
    annotations:
      iam.gke.io/gcp-service-account: "velero-backup@<my-project-id>.iam.gserviceaccount.com"

credentials:
  useSecret: false

# Backup schedules to create.
schedules:
  daily-30days:
    schedule: "0 0 * * *" # daily at midnight UTC
    template:
      ttl: "720h"
      storageLocation: default
      includedNamespaces:
      - "*"
      excludedNamespaces:
      - kube-*
      - gke-*
      - gmp-public

Environment:

• Velero version (use velero version): 1.17.0
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): v1.33.5-gke.1080000
• Kubernetes installer & version: N/A
• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release): N/A

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[Lyndon-Li]

Please share Velero debug bundle by running velero debug

[LegendSebastiano-L]

Same here, I upgraded k8s to 1.33, so I had to upgrade Velero as well.

Velero chart: 11.1.1
Velero GCP: 1.13.0

In the last backup in k8s, I see this

status:
  completionTimestamp: '2025-10-29T07:16:00Z'
  expiration: '2025-11-05T07:15:56Z'
  failureReason: >-
    error checking if backup already exists in object storage: rpc error: code =
    Unknown desc = storage: object doesn't exist: googleapi: Error 404: No such
    object:
    XX-gke-backups/staging/backups/velero-gkebackup-20251029071556/velero-backup.json,
    notFound
  formatVersion: 1.1.0
  phase: Failed
  startTimestamp: '2025-10-29T07:15:56Z'
  version: 1

And the container is returning these logs (the last 2 logs are constantly returned every minute)

time="2025-10-29T07:15:56Z" level=info msg="Schedule is due, going to submit backup." logSource="pkg/controller/schedule_controller.go:255" schedule=velero/velero-gkebackup

time="2025-10-29T07:15:56Z" level=info msg="No Schedule.template.metadata.labels set - using Schedule.labels for backup object" backup=velero/velero-gkebackup-20251029071556 labels="map[app.kubernetes.io/instance:velero app.kubernetes.io/managed-by:Helm app.kubernetes.io/name:velero environment:staging helm.sh/chart:velero-11.1.1 k8slens-edit-resource-version:v1]"

time="2025-10-29T07:15:57Z" level=info msg="Setting up backup log" backup=velero/velero-gkebackup-20251029071556 logSource="pkg/controller/backup_controller.go:679"

time="2025-10-29T07:15:57Z" level=info msg="Setting up backup temp file" backup=velero/velero-gkebackup-20251029071556 logSource="pkg/controller/backup_controller.go:690"

time="2025-10-29T07:15:57Z" level=info msg="Setting up plugin manager" backup=velero/velero-gkebackup-20251029071556 logSource="pkg/controller/backup_controller.go:697"

time="2025-10-29T07:15:57Z" level=info msg="Getting backup item actions" backup=velero/velero-gkebackup-20251029071556 logSource="pkg/controller/backup_controller.go:701"

time="2025-10-29T07:15:59Z" level=info msg="Getting ItemBlock actions" backup=velero/velero-gkebackup-20251029071556 logSource="pkg/controller/backup_controller.go:706"

time="2025-10-29T07:15:59Z" level=info msg="Setting up backup store to check for backup existence" backup=velero/velero-gkebackup-20251029071556 logSource="pkg/controller/backup_controller.go:711"

time="2025-10-29T07:15:59Z" level=info msg="plugin process exited" backupLocation=velero/global-bucket cmd=/plugins/velero-plugin-for-gcp controller=backup-sync id=181 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-gcp

time="2025-10-29T07:16:00Z" level=info msg="plugin process exited" backup=velero/velero-gkebackup-20251029071556 cmd=/velero id=187 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/velero

time="2025-10-29T07:16:00Z" level=info msg="plugin process exited" backup=velero/velero-gkebackup-20251029071556 cmd=/plugins/velero-plugin-for-gcp id=198 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-gcp

time="2025-10-29T07:16:00Z" level=error msg="backup failed" backuprequest=velero/velero-gkebackup-20251029071556 controller=backup error="error checking if backup already exists in object storage: rpc error: code = Unknown desc = storage: object doesn't exist: googleapi: Error 404: No such object: XX-gke-backups/staging/backups/velero-gkebackup-20251029071556/velero-backup.json, notFound" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_controller.go:722" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupReconciler).runBackup" logSource="pkg/controller/backup_controller.go:317"

time="2025-10-29T07:16:00Z" level=info msg="Updating backup's status" backuprequest=velero/velero-gkebackup-20251029071556 controller=backup logSource="pkg/controller/backup_controller.go:336"
time="2025-10-29T07:16:27Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/global-bucket controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:147"

time="2025-10-29T07:16:27Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/global-bucket controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:132"

time="2025-10-29T07:16:27Z" level=info msg="plugin process exited" backup-storage-location=velero/global-bucket cmd=/plugins/velero-plugin-for-gcp controller=backup-storage-location id=208 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-gcp

It is not a problem with the Service account, because the same is used in production with the old Velero version and is working. I also deleted all backups from K8s, and Velero was able to read that I had backups in the GCP bucket, and it pulled them into k8s.

my value file

  initContainers:
    - name: velero-plugin-for-gcp
      image: velero/velero-plugin-for-gcp:v1.13.0
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - mountPath: /target
          name: plugins

  cleanUpCRDs: false

  rbac:
    create: true

  serviceAccount:
    server:
      create: true

  backupsEnabled: true
  snapshotsEnabled: true

  configuration:
    backupStorageLocation:
      - name: global-bucket
        provider: gcp
        bucket: XX-gke-backups
        prefix: staging
        default: true

    volumeSnapshotLocation:
      - name: global-bucket
         provider: gcp
         default: true

  schedules:
    gkebackup:
      disabled: false
      annotations:
        environment: staging
      labels:
        environment: staging
      schedule: "0 0 * * *"
      useOwnerReferencesInBackup: false
      template:
        ttl: "168h"
        storageLocation: global-bucket

  credentials:
    useSecret: true
    existingSecret: velero-credentials

The only thing I changed from the Velero chart 5.2.0 to 11.1.0 was the velero-plugin-for-gcp from 1.8.2 to 1.13.0

[Nexus2k]

Logs were the same as from @LegendSebastiano-L for me, 404 on check for the .json file. I fixed the issue by downgrading to velero 1.16.0 and gcp-plugin 1.12.2 using the same GCP service-account.

[Nexus2k]

Digged thru the gcp-plugin repo and the issue is solved with vmware-tanzu/velero-plugin-for-gcp#239 which has not been released yet.
Looking at the actual release branch: https://github.com/search?q=repo%3Avmware-tanzu%2Fvelero-plugin-for-gcp%20serviceAccountConfigKey&type=code you can see that somebody only changed the field name in the const but not when they're used leading to the plugin not using the service-account and therefore getting a 404 (it's a bit security by obscurity there by Google, should be a 401 unauthorized)

[Nexus2k]

While this issue here is a duplicate of #9335 I'd like to keep it open until a working gcp-plugin was released.

[kaovilai]

#9335 (comment)
will be fixed in 1.13.1 as noted 2 weeks ago.

[reasonerjt]

While this issue here is a duplicate of #9335 I'd like to keep it open until a working gcp-plugin was released.

Conventionally, when we GA velero all issues with "target/v1.17.1" need to be closed. I'm closing this issue. Please reach out to us via Slack if you don't see the plugin bumped up after velero's patch is released.

[blackpiglet]

@Nexus2k

Velero-plugin-for-gcp v1.13.1 has been released and is available at https://github.com/vmware-tanzu/velero-plugin-for-gcp/releases/tag/v1.13.1.

We understand that closing an issue before a working version is released may seem unreasonable. Our process involves linking issues to specific versions using a label. All related issues are closed before a tag is created for the version. This helps us avoid the cumbersome task of triaging open issues every time, especially with the constant influx of new reports.

Thank you for your patience and effort.