[feature] restic and azure workload identity

[sybnex]

I was the last weeks hardly working on our backup solution using volsync and restic using azure workload identity in our AKS environment.

I found, that restic can do so since: restic/restic#4029
But there are some missing values in volsync to archive the full flow with managed identites with a AKS setup.

There are three env variables missing:

AZURE_TENANT_ID
AZURE_CLIENT_ID
AZURE_FEDERATED_TOKEN_FILE

AZURE_TENANT_ID and AZURE_CLIENT_ID have to be configured by the user and must be transferred to the scheduling pod from the restic_config secret.

The AZURE_FEDERATED_TOKEN_FILE points to the service account token in the pod and could be fixed in the backup pod template like:

    - name: AZURE_FEDERATED_TOKEN_FILE
      value: "/var/run/secrets/kubernetes.io/serviceaccount/token"

What is the value to the end user? (why is it a priority?)
The main value to use workload identity is, that u don't have any passwords or token in Kubernetes at all. Our CISO would love to see that.

How will we know we have a good solution? (acceptance criteria)
Backup in AKS without credentials

Additional context
I'll try to make a pull request, especially for the azure setup documentation

Workaround

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: inject-env-from-sa-label
spec:
  rules:
  - name: inject-backup-mode
    match:
      any:
      - resources:
          kinds:
          - Pod
          selector:
            matchLabels:
              app.kubernetes.io/created-by: volsync
    context:
    - name: clientID
      apiCall:
        urlPath: "/api/v1/namespaces/{{request.namespace}}/serviceaccounts/backup"
        jmesPath: "metadata.annotations.\"azure.workload.identity/client-id\""
    - name: tenantID
      variable:
        value: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    - name: tokenFile
      variable:
        value: "/var/run/secrets/kubernetes.io/serviceaccount/token"
    mutate:
      patchStrategicMerge:
        spec:
          containers:
          - (name): "*"
            env:
            - name: AZURE_CLIENT_ID
              value: "{{ clientID }}"
            - name: AZURE_TENANT_ID
              value: "{{ tenantID }}"
            - name: AZURE_FEDERATED_TOKEN_FILE
              value: "{{ tokenFile }}"

[tesshuflower]

@sybnex This doesn't appear to be documented in restic, if you can help with updating restic docs (maybe that is what you mean by opening a PR?) that would be great.

It should be simple for us to pass through and AZURE_TENANT_ID if they're specified in the restic config, however when it comes to AZURE_FEDERATED_TOKEN_FILE, I'm not sure when this needs to be setup.

Can we assume if AZURE_CLIENT_ID and AZURE_TENANT_ID are set that AZURE_FEDERATED_TOKEN_FILE should point to /var/run/secrets/kubernetes.io/serviceaccount/token?

[tesshuflower]

Actually if we want to be consistent with the way we've handled other credentials, perhaps it would be better if we model
AZURE_FEDERATED_TOKEN_FILE as we did with GOOGLE_APPLICATION_CREDENTIALS. Essentially we allow setting it as text in the restic secret and volsync will mount this as a file and point the env var to it.

See the note about GOOGLE_APPLICATION_CREDENTIALS here

[sybnex]

Hi,

Restic is using the Azure SDK get a token for the connection. The SDK has like 5 possibilites how archive this. Volsync uses the direct path to the backend storage only and forward this info over the restic-config to the restic pod =>

volsync/internal/controller/mover/restic/mover.go

Line 461 in a4be672

utils.EnvFromSecret(repo.Name, "AZURE_ACCOUNT_NAME", true),

An example secret in my case would look like this:

---
apiVersion: v1
kind: Secret
metadata:
  name: restic-config
type: Opaque
stringData:
  RESTIC_REPOSITORY: azure:volsync-backup:/
  RESTIC_PASSWORD: my-secure-restic-password
  AZURE_ACCOUNT_NAME: xxxxxxxxxxxxxx
  # below variables will not yet be forwarded
  AZURE_TENANT_ID: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  AZURE_CLIENT_ID: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"
  AZURE_FEDERATED_TOKEN_FILE: "/var/run/secrets/kubernetes.io/serviceaccount/token"

AZURE_TENANT_ID and AZURE_CLIENT_ID must be configurable.

But AZURE_FEDERATED_TOKEN_FILE can be fixed by the given path. Azure uses the Kubernetes Service Account to validate the Managed Identities in Azure (https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster?tabs=new-cluster) to get a Token to access the Backend Storage like Azure Blob. Restic is aware of all off those by using the Azure SDK and I tested it successfully with the workaorund above.

PS: The docu in restic is there, but just very small => https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#microsoft-azure-blob-storage

Alternatively, if run on Azure, restic will automatically use service accounts configured via the standard environment variables or Workload / Managed Identities.

Do u need any more informations?

[tesshuflower]

@sybnex I can add these additional AZURE variables, but for AZURE_FEDERATED_TOKEN_FILE we'd like to model it like we did previously for GOOGLE_APPLICATION_CREDENTIALS (see explanation of GOOGLE_APPLICATION_CREDENTIALS in the NOTE in this section: https://volsync.readthedocs.io/en/stable/usage/restic/index.html#specifying-a-repository)

Do you see any issue with that? It means in your restic secret, AZURE_FEDERATED_TOKEN_FILE would be slightly different, you'd need to set it to the string value of the token file itself. VolSync would then write this to a file, and point the env var to that file location (in the mover pod itself). This has the advantage of being consistent with other credentials, and not needing to alter your service account - but perhaps this causes some issue with automatic token renewal?

[sybnex]

I see u point.

But the content from /var/run/secrets/kubernetes.io/serviceaccount/token is automatically generated by kubernetes itself. It is has the token generated from a Kubernetes Service Account which is used by the deployment/pod during the runtime => https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server

So it would be impossible to read it during the run and forward the content at the same time to this variable, as the token is created during the startup of the pod.

Volsync could just use this given string as default to use the already mounted token from kubernetes. This works acutally only in this case together with the azure sdk and managed identites with the federation solution. Check this Article for more details in the fow: https://paulyu.dev/article/cross-tenant-workload-identity-on-aks/

Anyway. It would be great to implement it as far as you can.

[tesshuflower]

@sybnex I think we will just pass through the env vars directly then from the secret, just to keep it simple (and handled the same way as most other env vars).

so it'll be up to you to set AZURE_FEDERATED_TOKEN_FILE: "/var/run/secrets/kubernetes.io/serviceaccount/token" in your secret, as per your example above.