Use GKE workload identity federation in composer

[ohrite]

Description

This PR proposes to stop passing service_account.json as a secret, and instead rely on GKE workload identity federation, just like we do for Github Actions.

Relates to #3780

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

How has this been tested?

terraform plan and test runs on staging

Post-merge follow-ups

• No action required
• Actions required (specified below)

Remove service_acount.json/service-account.json from the jobs-data Kubernetes secret.

[github-actions]

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 3 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer["dags/publish_open_data/publish_california_open_data.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "Wx+C0A==" -> (known after apply)
!~      detect_md5hash      = "Dmn2ccrjkUs7GTsL5zO+rw==" -> "different hash"
!~      generation          = 1751416049286660 -> (known after apply)
        id                  = "calitp-staging-composer-dags/publish_open_data/publish_california_open_data.yml"
!~      md5hash             = "Dmn2ccrjkUs7GTsL5zO+rw==" -> (known after apply)
        name                = "dags/publish_open_data/publish_california_open_data.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "WAOTIw==" -> (known after apply)
!~      detect_md5hash      = "Q2KZHpa145QxEt0qrMVrvQ==" -> "different hash"
!~      generation          = 1751929677147947 -> (known after apply)
        id                  = "calitp-staging-composer-dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"
!~      md5hash             = "Q2KZHpa145QxEt0qrMVrvQ==" -> (known after apply)
        name                = "dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["profiles.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "fcYfVw==" -> (known after apply)
!~      detect_md5hash      = "8XK0sntGMFxIy7Ve/HuJzg==" -> "different hash"
!~      generation          = 1749663111897003 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/profiles.yml"
!~      md5hash             = "8XK0sntGMFxIy7Ve/HuJzg==" -> (known after apply)
        name                = "data/warehouse/profiles.yml"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #724

[github-actions]

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 3 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer["dags/publish_open_data/publish_california_open_data.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "Wx+C0A==" -> (known after apply)
!~      detect_md5hash      = "Dmn2ccrjkUs7GTsL5zO+rw==" -> "different hash"
!~      generation          = 1751416675383240 -> (known after apply)
        id                  = "calitp-composer-dags/publish_open_data/publish_california_open_data.yml"
!~      md5hash             = "Dmn2ccrjkUs7GTsL5zO+rw==" -> (known after apply)
        name                = "dags/publish_open_data/publish_california_open_data.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "WAOTIw==" -> (known after apply)
!~      detect_md5hash      = "Q2KZHpa145QxEt0qrMVrvQ==" -> "different hash"
!~      generation          = 1751416672802943 -> (known after apply)
        id                  = "calitp-composer-dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"
!~      md5hash             = "Q2KZHpa145QxEt0qrMVrvQ==" -> (known after apply)
        name                = "dags/unzip_and_validate_gtfs_schedule_hourly/validate_gtfs_schedule.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["profiles.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "fcYfVw==" -> (known after apply)
!~      detect_md5hash      = "8XK0sntGMFxIy7Ve/HuJzg==" -> "different hash"
!~      generation          = 1751416670054928 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/profiles.yml"
!~      md5hash             = "8XK0sntGMFxIy7Ve/HuJzg==" -> (known after apply)
        name                = "data/warehouse/profiles.yml"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #724

[github-actions]

Terraform plan in iac/cal-itp-data-infra-staging/composer/us

Plan: 0 to add, 1 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # kubernetes_secret.composer will be updated in-place
!~  resource "kubernetes_secret" "composer" {
!~      data                           = (sensitive value)
        id                             = "airflow-jobs/jobs-data"
#        (5 unchanged attributes hidden)

#        (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Terraform Plan #556

[github-actions]

Terraform plan in iac/cal-itp-data-infra/composer/us

Plan: 0 to add, 1 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # kubernetes_secret.composer will be updated in-place
!~  resource "kubernetes_secret" "composer" {
!~      data                           = (sensitive value)
        id                             = "airflow-jobs/jobs-data"
#        (5 unchanged attributes hidden)

#        (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Terraform Plan #556