Skip to content

VEP94: add projected volumes in Volume API #95

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

VEP94: add projected volumes in Volume API #95

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 130 additions & 0 deletions veps/sig-compute/projected-volumes-for-service-account-tokens.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# VEP #94: Extend Volume API to Support Projected Volumes for Service Account Tokens

## Release Signoff Checklist
Items marked with (R) are required *prior to targeting to a milestone / release*.
- [X] (R) Enhancement issue created, which links to VEP dir in [kubevirt/enhancements] (not the initial VEP PR)
- [ ] (R) Target version is explicitly mentioned and approved
- [ ] (R) Graduation criteria filled

## Overview
This VEP proposes extending KubeVirt's Volume API to support Projected Volumes,
similar to Kubernetes, enabling flexible mounting of service account
tokens with custom audiences and paths into virtual machines via virtiofs for
token rotations. This addresses an existing limitation in dynamic
authentication scenarios like AWS IRSA.

## Motivation
KubeVirt currently mounts service accounts at a hardcoded path which works fine
for standard Kubernetes API tokens but fails for advanced projections like
those in AWS IRSA. There are mounted at
`/var/run/secrets/eks.amazonaws.com/serviceaccount` with audience
`sts.amazonaws.com`. This prevents VMs from accessing rotated tokens and forces
users to invent custom workarounds like mutation webhooks or sidecars, adding
complexity, overhead and security risks.

KubeVirt exposes virtual machines through a Kubernetes-native API with a
similar look and feel to Pods, simplifying the user experience and making VM
management consistent with existing Kubernetes workloads. Supporting Projected
Volumes extends this approach by enabling declarative and flexible handling of
projected service account tokens inside VMs. This directly benefits use cases
in environments like EKS and other multi-cloud setups.

Discussions in issue #13311 highlight the need for an upstream fix, as
workarounds are temporary and not integrated.

## Goals
- Extend VolumeSource with Projected field, supporting ServiceAccountTokenProjection.
- Enable dynamic token propagation via virtiofs.
- Introduce feature gate for controlled rollout.
- Ensure backward compatibility and integration tests.

## Non Goals
- Full support for all Kubernetes projection types (focus on service accounts).
- Support for non-virtiofs dynamic updates.

## Definition of Users
VM owners and workload developers running KubeVirt in cloud environments, such
as EKS, multi-cloud who need secure, dynamically rotated tokens inside their
VMs for authentication with external services. Cluster administrators may be
indirectly impacted but are not the primary users.

## User Stories
- As a VM owner running workloads on EKS, I want to mount IRSA tokens with
custom audiences into my VM so that applications inside the VM can
authenticate securely without custom webhooks or sidecars.
- As a workload developer, I want projected service account tokens to be
mounted declaratively through the VMI spec, so that I can configure
authentication in a Kubernetes-native way without extra operational overhead.
- As a VM user, I want tokens inside my VM to rotate automatically without
requiring VM restarts, so my applications continue to run securely without
interruption.

## Repos
- kubevirt/kubevirt (core API and virt-launcher)

## Design
Add `Projected` to `VolumeSource`, with `ServiceAccountTokenProjection` for
audience, expiration, and path. Virt-launcher detects projections and shares
via virtiofs. Feature gate: ProjectedVolumes.

## API Examples
```yaml
volumes:
- name: irsa-volume
projected:
sources:
- serviceAccountToken:
audience: "sts.amazonaws.com"
expirationSeconds: 3600
path: "token"
```
Comment on lines +70 to +80
Copy link

@germag germag Nov 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking that maybe the API should be:

volumes:
- name: irsa-volume
  serviceAccount:
    serviceAccountName: service-account-irsa
    serviceAccountToken:
        audience: "sts.amazonaws.com"
        expirationSeconds: 3600
        path: "token"

The only drawback (that I can see) is it will not be integrated with
the rest of projected volume sources (e.g., configMap) if we support
those in the future.

Let me explain:

The current Kubevirt API is:

volumes:
- name: irsa-volume
  serviceAccount:
    - serviceAccountName: service-account-irsa

I don't really like this API because it suggests it can be more
than a single serviceAccount volume, so it's confusing. For instance,
the k8s API is explicit about that:

kind: Pod
...
spec:
  serviceAccountName: build-robot
  automountServiceAccountToken: false

So, if we use the proposed API in this VEP we also must define the
serviceAccountName somewhere, so we have few options:

1- define both the volume and the projection:

volumes:
- name: irsa-volume
  serviceAccount:
    serviceAccountName: service-account-irsa

- name: irsa-volume-projected
  projected:
    sources:
    - serviceAccountToken:
        audience: "sts.amazonaws.com"
        expirationSeconds: 3600
        path: "token"

Even if this option will allow more projected sources in the future, and
is more similar to k8s API, I think it's not an option because both
have volumes.name which one we should mount?, is very confusing.

2- Add serviceAccountName under serviceAccountToken

volumes:
- name: irsa-volume
  projected:
    sources:
    - serviceAccountToken:
        serviceAccountName: service-account-irsa
        audience: "sts.amazonaws.com"
        expirationSeconds: 3600
        path: "token"

I also don't think this is a good API, it's not consistent with the
actual API, so what happens if the user defines this and
volumes.serviceAccount.serviceAccountName, which one should have
priority?. The only "pro" I can see is it will allow sharing other projected
sources in the future (if we want to support them)

That is the reason I think

volumes:
- name: irsa-volume
  serviceAccount:
    serviceAccountName: service-account-irsa
    serviceAccountToken:
        audience: "sts.amazonaws.com"
        expirationSeconds: 3600
        path: "token"

is a bit better, even if it will not be shared with others projected
sources in the future, and is not close to the k8s API. I also have
the advantage that the current admitter code will still prevent
multiple serviceAccount's

@vladikr wdyt?


## Alternatives
- Mutation Webhooks: Inject sidecars or modify VM definitions at admission time
to deliver tokens. While effective, this introduces an external dependency
and is not integrated into core KubeVirt. (based on repo: kubevirt/irsa-mutation-webhook)

- Sidecar Injections (based on PR #14568): Run custom containers inside the
virt-launcher pod to provide token files to the VM. This offers flexibility
but increases pod complexity, operational burden, and maintenance costs and not supported directly supported.

- Recursive Secret Scanning: Extend virt-launcher to recursively scan
/var/run/secrets and expose contents to the VM. This raises significant
security risks by unintentionally exposing unrelated secrets.

- QEMU Guest Agent Extensions: Extend the guest agent protocol to push
projected tokens into the VM filesystem. This avoids virtiofs but requires
custom controllers for token rotation and introduces additional moving parts.

- Userspace NFS Server: Run a lightweight NFS server and mount it into the VM
to deliver projected files. This adds dependencies on NFS, and is not aligned
with Kubernetes-native volume semantics.

These are workarounds; the proposed API extension provides a declarative, upstream solution.

## Scalability
Per-VM handling; scales with existing volume limits. Minimal overhead as
projections are pod-level.

## Update/Rollback Compatibility
- New field is optional; existing VMIs remain unaffected.
- On downgrade, projected volumes fall back to errors or no-op.

## Functional Testing Approach
- Unit tests: API validation, struct marshaling.
- Integration tests: Deploy VMI with projected token, verify mount and rotation in guest.

## Implementation History
October 01, 2025: VEP drafted.

## Graduation Requirements
### Alpha
- Feature gate enabled.
- Initial implementation for service account projections.

### Beta
- Full IRSA support.

### GA
- Documentation complete.
- Featuregate removal.