Add support for Ray token auth #4179

andrewsykim · 2025-11-05T04:34:19Z

Why are these changes needed?

TODO:

unit / integration tests
support for referencing an existing secret

Related issue number

N/A

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

andrewsykim · 2025-11-07T02:12:26Z

ray-operator/controllers/ray/utils/constant.go

 	RAY_ENABLE_AUTOSCALER_V2 = "RAY_enable_autoscaler_v2"

+	// RAY_AUTH_MODE_ENV_VAR is the Ray environment variable for configuring the authentication mode
+	RAY_AUTH_MODE_ENV_VAR = "RAY_auth_mode" // TODO: change to RAY_AUTH_MODE once updated in Ray nightly


TODO after ray-project/ray#58408

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim · 2025-11-07T02:14:28Z

ray-operator/test/e2e/raycluster_auth_test.go

+			verifyAuthTokenEnvVars(t, rayCluster, workerPod)
+		}
+
+		// TODO(andrewsykim): add job submission test with and without token once a Ray version with token support is released.


TODO after ray-project/ray#58408

Signed-off-by: Andrew Sy Kim <[email protected]>

rueian · 2025-11-07T03:57:00Z

ray-operator/controllers/ray/common/pod.go

+		ValueFrom: &corev1.EnvVarSource{
+			SecretKeyRef: &corev1.SecretKeySelector{
+				LocalObjectReference: corev1.LocalObjectReference{Name: secretName},
+				Key:                  "auth_token",


Suggested change

Key: "auth_token",

Key: utils.RAY_AUTH_TOKEN_SECRET_KEY,

fixed, thanks

rueian · 2025-11-07T03:57:51Z

ray-operator/controllers/ray/utils/util.go

 }

+// IsAuthEnabled returns whether Ray auth is enabled.
+func IsAuthEnabled(spec *rayv1.RayClusterSpec) bool {


Suggested change

func IsAuthEnabled(spec *rayv1.RayClusterSpec) bool {

func IsTokenAuthEnabled(spec *rayv1.RayClusterSpec) bool {

Weird. I am still seeing it remains the old IsAuthEnabled.

sorry, I probably forgot to push the latest commits

rueian · 2025-11-07T03:58:50Z

ray-operator/controllers/ray/common/pod.go

 }

+// setAuthEnvVars sets environment variables required for Ray token authentication
+func setAuthEnvVars(clusterName string, podTemplate *corev1.PodTemplateSpec) {


Suggested change

func setAuthEnvVars(clusterName string, podTemplate *corev1.PodTemplateSpec) {

func setTokenAuthEnvVars(clusterName string, podTemplate *corev1.PodTemplateSpec) {

nit: a bit more explicit.

Signed-off-by: Andrew Sy Kim <[email protected]>

Future-Outlier

Would it be possible to include an example (under folder ray-operator/config/samples/) using rayproject/ray:nightly to test this PR prior to merging?

Future-Outlier · 2025-11-10T12:38:48Z

ray-operator/controllers/ray/utils/constant.go

+	RAY_AUTH_TOKEN_SECRET_KEY = "auth_token"
+


I am thinking is ray_auth_token better than auth_token?

Future-Outlier

I use the following example to test this PR.
And I found that the RayCluster's head pod is never READY, do you know why?

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-kuberay-auth-token
spec:
  rayVersion: "2.51.0"
  authOptions:
    mode: "token"
  headGroupSpec:
    # rayStartParams is optional with RayCluster CRD from KubeRay 1.4.0 or later but required in earlier versions.
    rayStartParams: {}
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray:nightly
          resources:
            # limits:
            #   cpu: 1
            #   memory: 2G
            requests:
              cpu: 1
              memory: 2G
          ports:
          - containerPort: 6379
            name: gcs-server
          - containerPort: 8265 # Ray dashboard
            name: dashboard
          - containerPort: 10001
            name: client
  workerGroupSpecs:
  # the pod replicas in this group typed worker
  - replicas: 1
    minReplicas: 1
    maxReplicas: 5
    # logical group name, for this called small-group, also can be functional
    groupName: workergroup
    # rayStartParams is optional with RayCluster CRD from KubeRay 1.4.0 or later but required in earlier versions.
    rayStartParams: {}
    template:
      spec:
        containers:
        - name: ray-worker # must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc'
          image: rayproject/ray:nightly
          resources:
            # limits:
            #   cpu: 1
            #   memory: 1G
            requests:
              cpu: 1
              memory: 1G

Future-Outlier · 2025-11-10T14:15:16Z

ray-operator/controllers/ray/common/pod.go

+		Value: "token",
+	})


Suggested change

Value: "token",

})

Value: rayv1.AuthModeToken,

})

Future-Outlier

I have 2 questions relate to RBAC.

Our controller only calls Get() and Create() on Secrets, never delete, patch, or update. Should we remove these unused verbs from the RBAC definition to follow the principle of least privilege?
Should we add Owns(&corev1.Secret{}) to SetupWithManager? Currently we have list and watch permissions in RBAC, but without Owns(), these permissions aren’t being used.

(but I'm not sure watch and list secret is a good practice or not)

andrewsykim · 2025-11-10T15:21:15Z

We'll need watch/list for informer cache, good call on delete, we probably don't need that as we can rely on garbage collection from owner ref

Future-Outlier · 2025-11-10T15:35:21Z

I might try to build a ray image with this PR and test it tmr.

ray-project/ray#58368

Future-Outlier · 2025-11-10T15:36:49Z

We'll need watch/list for informer cache, good call on delete, we probably don't need that as we can rely on garbage collection from owner ref

sorry I didn't write my comments correctly, I agreed we need watch/list for informer cache, and should we add Owns(&corev1.Secret{}) to SetupWithManager?

andrewsykim · 2025-11-10T15:45:29Z

@Future-Outlier FYI if you're testing, you need to use nightly build and there's a lot of on-going changes for this feature

Future-Outlier · 2025-11-10T15:46:54Z

@Future-Outlier FYI if you're testing, you need to use nightly build and there's a lot of on-going changes for this feature

thanks @andrewsykim
I will test more tmr and will probably provide a video or screenshot

Future-Outlier · 2025-11-12T02:46:46Z

just successfully built the image

cd ray-project/ray
docker build --progress=plain --build-arg BUILD_DATE="$(date +%Y-%m-%d:%H:%M:%S)" -t $RAY_IMAGE -f ./python/ray/tests/kuberay/Dockerfile .

andrewsykim · 2025-11-12T03:13:24Z

ray-operator/controllers/ray/common/pod.go

+		},
+	})
+
+	// Configure auth token for wait-gcs-ready init container if it exists


@sampan-s-nayak is it expected for "ray healh-check` to require auth token?

Is ray autoscaler also require auth token?

Good catch, I'll add this logic shortly

win5923 · 2025-11-12T17:00:20Z

docs/reference/api.md

+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `mode` _[AuthMode](#authmode)_ | Mode specifies the authentication mode.<br />Supported values are "disabled" and "token".<br />Defaults to "token". |  | Enum: [disabled token] <br /> |


I think it’s better to set the default to disabled, as token authentication requires Ray >= 2.51.0 and some users may still be on older pinned versions.

It's worth mentioning the mode only defaults to token when authOptions != nil

win5923 · 2025-11-12T17:33:17Z

ray-operator/controllers/ray/common/pod.go

+		},
+	})
+
+	// Configure auth token for wait-gcs-ready init container if it exists


Is ray autoscaler also require auth token?

Future-Outlier

my current thought:

before this PR get merged
I think we should

test token mode
test k8s mode
test case with 1 and 2 + autoscaler

given that build a ray image is extremely hard, I'm going to wait for the latest ray image release.

andrewsykim · 2025-11-13T22:06:08Z

@Future-Outlier let's keep this PR scoped to token mode only. The k8s auth mode will be undocumented feature for a couple Ray releases since it needs more testing, we can add KubeRay support once the public docs are ready (probably before KubeRay v1.6 though)

ryanaoleary · 2025-11-15T00:13:49Z

ray-operator/controllers/ray/utils/validation.go

 	}
+
+	if IsAuthEnabled(spec) {
+		if spec.RayVersion == "" {


Kinda an out-there edge case but is there any validation that the RayVersion specified matches the image actually being used in the container?

andrewsykim force-pushed the auth-support branch 2 times, most recently from 5f7e0bb to 132bea2 Compare November 6, 2025 23:02

andrewsykim commented Nov 7, 2025

View reviewed changes

Add support for Ray token auth

0b3ce04

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim force-pushed the auth-support branch from 132bea2 to 0b3ce04 Compare November 7, 2025 02:13

andrewsykim commented Nov 7, 2025

View reviewed changes

andrewsykim force-pushed the auth-support branch from 8861359 to e009bc0 Compare November 7, 2025 02:20

add e2e test for Ray cluster auth

8808456

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim force-pushed the auth-support branch from e009bc0 to 8808456 Compare November 7, 2025 02:56

andrewsykim changed the title ~~[WIP] Add support for Ray token auth~~ Add support for Ray token auth Nov 7, 2025

andrewsykim marked this pull request as ready for review November 7, 2025 03:35

andrewsykim requested review from MortalHappiness, kevin85421 and rueian as code owners November 7, 2025 03:35

andrewsykim requested review from ryanaoleary and removed request for MortalHappiness and kevin85421 November 7, 2025 03:39

rueian reviewed Nov 7, 2025

View reviewed changes

Future-Outlier self-assigned this Nov 7, 2025

address nits from Ruiean

e7cae09

Signed-off-by: Andrew Sy Kim <[email protected]>

Future-Outlier reviewed Nov 10, 2025

View reviewed changes

andrewsykim commented Nov 12, 2025

View reviewed changes

Future-Outlier mentioned this pull request Nov 12, 2025

[Docs] How to build ray image when involved frontend, c++, and python ray-project/ray#58567

Open

win5923 reviewed Nov 12, 2025

View reviewed changes

Future-Outlier reviewed Nov 13, 2025

View reviewed changes

rueian approved these changes Nov 14, 2025

View reviewed changes

ryanaoleary reviewed Nov 15, 2025

View reviewed changes

ryanaoleary approved these changes Nov 15, 2025

View reviewed changes

	func IsAuthEnabled(spec *rayv1.RayClusterSpec) bool {
	func IsTokenAuthEnabled(spec *rayv1.RayClusterSpec) bool {

	func setAuthEnvVars(clusterName string, podTemplate *corev1.PodTemplateSpec) {
	func setTokenAuthEnvVars(clusterName string, podTemplate *corev1.PodTemplateSpec) {

Add support for Ray token auth #4179

Are you sure you want to change the base?

Add support for Ray token auth #4179

Conversation

andrewsykim commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

andrewsykim commented Nov 10, 2025

Uh oh!

Future-Outlier commented Nov 10, 2025

Uh oh!

Future-Outlier commented Nov 10, 2025

Uh oh!

andrewsykim commented Nov 10, 2025

Uh oh!

Future-Outlier commented Nov 10, 2025

Uh oh!

Future-Outlier commented Nov 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

andrewsykim commented Nov 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

andrewsykim commented Nov 5, 2025 •

edited

Loading

Future-Outlier Nov 10, 2025 •

edited

Loading