feat(ws): add spec.podTemplate.ports[] to WorkspaceKind #507

harshad16 · 2025-07-29T05:10:57Z

related: #37

This PR adds spec.podTemplate.ports[] to workspaceKind CRD, which lets users include ports httpproxy setting for their workspaces.

WorkspaceKind CRD changes

spec:
  podTemplate:
    ports:
      - portId: "jupyterlab"
        httpProxy: {}

These changes would be consider while setting the routing for proper traffic controller/routing to the pods.

google-oss-prow · 2025-07-29T05:11:01Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign thesuperzapper for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

thesuperzapper · 2025-07-31T16:11:49Z

/ok-to-test

andyatmiami · 2025-08-01T16:14:06Z

workspaces/controller/internal/webhook/suite_test.go

+// NewExampleWorkspaceKindWithEmptyPortsArrayInPodTemplate returns a WorkspaceKind with an empty ports array in podTemplate.ports.
+func NewExampleWorkspaceKindWithEmptyPortsArrayInPodTemplate(name string) *kubefloworgv1beta1.WorkspaceKind {
+	workspaceKind := NewExampleWorkspaceKind(name)
+	workspaceKind.Spec.PodTemplate.Ports = []kubefloworgv1beta1.WorkspaceKindPort{}
+	return workspaceKind
+}
+
+// NewExampleWorkspaceKindWithDuplicatePortsInPodTemplate returns a WorkspaceKind with duplicate ports in podTemplate.ports.
+func NewExampleWorkspaceKindWithDuplicatePortsInPodTemplate(name string) *kubefloworgv1beta1.WorkspaceKind {
+	workspaceKind := NewExampleWorkspaceKind(name)
+	workspaceKind.Spec.PodTemplate.Ports = []kubefloworgv1beta1.WorkspaceKindPort{
+		{
+			PortId: "jupyterlab",
+		},
+		{
+			PortId: "jupyterlab",
+		},
+	}
+	return workspaceKind
+}
+
+// NewExampleWorkspaceKindWithNonExistentPortIdInImageConfig returns a WorkspaceKind with a non-existent portId in imageConfig.ports.
+func NewExampleWorkspaceKindWithNonExistentPortIdInImageConfig(name string) *kubefloworgv1beta1.WorkspaceKind {
+	workspaceKind := NewExampleWorkspaceKind(name)
+	workspaceKind.Spec.PodTemplate.Ports = []kubefloworgv1beta1.WorkspaceKindPort{
+		{
+			PortId: "non-existent-port-id",
+		},
+	}
+	return workspaceKind
+}
+


I wonder if we should create a helper function:

// NewExampleWorkspaceKindWithCustomPorts returns a WorkspaceKind with custom ports in podTemplate.ports. func NewExampleWorkspaceKindWithCustomPorts(name string, ports []kubefloworgv1beta1.WorkspaceKindPort) *kubefloworgv1beta1.WorkspaceKind { workspaceKind := NewExampleWorkspaceKind(name) workspaceKind.Spec.PodTemplate.Ports = ports return workspaceKind }

open question on if this should be an exported function (or not) - but i'd lean towards leaving it exportable for other ad-hoc re-use...

while trying to address this, realized unless we write more elaborate test case, creating helper function would only increase the line of code, by not have any additional effect.
since major bit is already encapsulated in NewExampleWorkspaceKind function.
I couldn't find direct impact of creating a common helper only for port changes.
WDYT ?

andyatmiami

I think we need to update the samples/ workspacekind.yaml to reflect these changes.

https://github.com/harshad16/kf-notebooks/blob/port-in-wsk-podtemplate/workspaces/controller/config/samples/jupyterlab_v1beta1_workspacekind.yaml#L140

workspaces/controller/config/crd/bases/kubeflow.org_workspacekinds.yaml

workspaces/controller/internal/controller/suite_test.go

workspaces/controller/internal/webhook/suite_test.go

andyatmiami · 2025-08-01T16:27:51Z

workspaces/controller/api/v1beta1/workspacekind_types.go

+	// +kubebuilder:validation:MinLength:=1
+	// +kubebuilder:validation:MaxLength:=32


Not a problem - just moreso curiousity...

is this MaxLength tied to any inherent restriction in k8s? just wondering if the 32 length was arbitrary or if it has a real constraint underlying it.

the portId is not related to direct k8s object,
it is more of endpoint that initial application should be served at.
for example: in case of jupyter, jupyterlab id , would mean, when application is served,
the route of service , would be served at {route}/jupyterlab

the validation length needs to be discussed further.

workspaces/controller/api/v1beta1/workspacekind_types.go

andyatmiami · 2025-08-01T17:00:03Z

workspaces/controller/api/v1beta1/workspacekind_types.go

+
+	// the http proxy config for the port
+	// +kubebuilder:validation:Optional
+	HTTPProxy *HTTPProxy `json:"httpProxy,omitempty"`


Out of curiosity - what is the practical purpose of defining a PortId with no HTTProxy ? Is this moreso future-proofing as we anticipate other types of structs to be defined here in future (like SSH, etc)?

andyatmiami · 2025-08-01T17:01:33Z

workspaces/controller/api/v1beta1/workspacekind_types.go

 	// +kubebuilder:validation:Optional
-	HTTPProxy *HTTPProxy `json:"httpProxy,omitempty"`
+	Ports []WorkspaceKindPort `json:"ports,omitempty"`


Out of curiosity - what is the practical purpose of defining a WorkspaceKind with no Ports ? Why would someone want to do that ?

workspaces/controller/internal/webhook/workspacekind_webhook.go

- add validation webhook for podtemplate.ports - update the sample workspacekind with ports reference - referencing same id for portid in imageconfig and podtemplate.ports Signed-off-by: Harshad Reddy Nalla <[email protected]>

andyatmiami · 2025-08-06T20:01:01Z

/ok-to-test

andyatmiami · 2025-08-06T21:06:35Z

/lgtm

testing these changes on a cluster and was able to:

create a workspacekind (using samples/)
create a workspace referencing the workspacekind (using samples/)
view the YAML representation of workspacekind and see the ports: changes

 $ kubectl get workspacekinds.kubeflow.org/jupyterlab -o yaml
apiVersion: kubeflow.org/v1beta1
kind: WorkspaceKind
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kubeflow.org/v1beta1","kind":"WorkspaceKind","metadata":{"annotations":{},"name":"jupyterlab"},"spec":{"podTemplate":{"containerSecurityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"runAsNonRoot":true},"culling":{"activityProbe":{"jupyter":{"lastActivity":true}},"enabled":true,"maxInactiveSeconds":86400},"extraEnv":[{"name":"NB_PREFIX","value":"{{ httpPathPrefix \"jupyterlab\" }}"}],"extraVolumeMounts":[{"mountPath":"/dev/shm","name":"dshm"}],"extraVolumes":[{"emptyDir":{"medium":"Memory"},"name":"dshm"}],"options":{"imageConfig":{"spawner":{"default":"jupyterlab_scipy_190"},"values":[{"id":"jupyterlab_scipy_180","redirect":{"message":{"level":"Info","text":"This update will change..."},"to":"jupyterlab_scipy_190"},"spawner":{"description":"JupyterLab, with SciPy Packages","displayName":"jupyter-scipy:v1.8.0","hidden":true,"labels":[{"key":"python_version","value":"3.11"}]},"spec":{"image":"ghcr.io/kubeflow/kubeflow/notebook-servers/jupyter-scipy:v1.8.0","imagePullPolicy":"IfNotPresent","ports":[{"displayName":"JupyterLab","id":"jupyterlab","port":8888,"protocol":"HTTP"}]}},{"id":"jupyterlab_scipy_190","spawner":{"description":"JupyterLab, with SciPy Packages","displayName":"jupyter-scipy:v1.9.0","labels":[{"key":"python_version","value":"3.11"}]},"spec":{"image":"ghcr.io/kubeflow/kubeflow/notebook-servers/jupyter-scipy:v1.9.0","imagePullPolicy":"IfNotPresent","ports":[{"displayName":"JupyterLab","id":"jupyterlab","port":8888,"protocol":"HTTP"}]}}]},"podConfig":{"spawner":{"default":"tiny_cpu"},"values":[{"id":"tiny_cpu","spawner":{"description":"Pod with 0.1 CPU, 128 Mb RAM","displayName":"Tiny CPU","labels":[{"key":"cpu","value":"100m"},{"key":"memory","value":"128Mi"}]},"spec":{"resources":{"requests":{"cpu":"100m","memory":"128Mi"}}}},{"id":"small_cpu","spawner":{"description":"Pod with 1 CPU, 2 GB RAM","displayName":"Small CPU","hidden":false,"labels":[{"key":"cpu","value":"1000m"},{"key":"memory","value":"2Gi"}]},"spec":{"affinity":{},"nodeSelector":{},"resources":{"requests":{"cpu":"1000m","memory":"2Gi"}},"tolerations":[]}},{"id":"big_gpu","spawner":{"description":"Pod with 4 CPU, 16 GB RAM, and 1 GPU","displayName":"Big GPU","hidden":false,"labels":[{"key":"cpu","value":"4000m"},{"key":"memory","value":"16Gi"},{"key":"gpu","value":"1"}]},"spec":{"affinity":{},"nodeSelector":{},"resources":{"limits":{"nvidia.com/gpu":1},"requests":{"cpu":"4000m","memory":"16Gi"}},"tolerations":[{"effect":"NoSchedule","key":"nvidia.com/gpu","operator":"Exists"}]}}]}},"podMetadata":{"annotations":{"my-workspace-kind-annotation":"my-value"},"labels":{"my-workspace-kind-label":"my-value"}},"ports":[{"httpProxy":{"removePathPrefix":false,"requestHeaders":{}},"portId":"jupyterlab"}],"probes":null,"securityContext":{"fsGroup":100},"serviceAccount":{"name":"default-editor"},"volumeMounts":{"home":"/home/jovyan"}},"spawner":{"deprecated":false,"deprecationMessage":"This WorkspaceKind will be removed on 20XX-XX-XX, please use another WorkspaceKind.","description":"A Workspace which runs JupyterLab in a Pod","displayName":"JupyterLab Notebook","hidden":false,"icon":{"url":"https://jupyter.org/assets/favicons/apple-touch-icon-152x152.png"},"logo":{"url":"https://upload.wikimedia.org/wikipedia/commons/3/38/Jupyter_logo.svg"}}}}
  creationTimestamp: "2025-08-06T21:02:15Z"
  finalizers:
  - notebooks.kubeflow.org/workspacekind-protection
  generation: 2
  name: jupyterlab
  resourceVersion: "31521"
  uid: 22488214-4fa7-410d-b41c-50d8624c5134
spec:
  podTemplate:
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
    culling:
      activityProbe:
        jupyter:
          lastActivity: true
      enabled: true
      maxInactiveSeconds: 86400
    extraEnv:
    - name: NB_PREFIX
      value: '{{ httpPathPrefix "jupyterlab" }}'
    extraVolumeMounts:
    - mountPath: /dev/shm
      name: dshm
    extraVolumes:
    - emptyDir:
        medium: Memory
      name: dshm
    options:
      imageConfig:
        spawner:
          default: jupyterlab_scipy_190
        values:
        - id: jupyterlab_scipy_180
          redirect:
            message:
              level: Info
              text: This update will change...
            to: jupyterlab_scipy_190
          spawner:
            description: JupyterLab, with SciPy Packages
            displayName: jupyter-scipy:v1.8.0
            hidden: true
            labels:
            - key: python_version
              value: "3.11"
          spec:
            image: ghcr.io/kubeflow/kubeflow/notebook-servers/jupyter-scipy:v1.8.0
            imagePullPolicy: IfNotPresent
            ports:
            - displayName: JupyterLab
              id: jupyterlab
              port: 8888
              protocol: HTTP
        - id: jupyterlab_scipy_190
          spawner:
            description: JupyterLab, with SciPy Packages
            displayName: jupyter-scipy:v1.9.0
            hidden: false
            labels:
            - key: python_version
              value: "3.11"
          spec:
            image: ghcr.io/kubeflow/kubeflow/notebook-servers/jupyter-scipy:v1.9.0
            imagePullPolicy: IfNotPresent
            ports:
            - displayName: JupyterLab
              id: jupyterlab
              port: 8888
              protocol: HTTP
      podConfig:
        spawner:
          default: tiny_cpu
        values:
        - id: tiny_cpu
          spawner:
            description: Pod with 0.1 CPU, 128 Mb RAM
            displayName: Tiny CPU
            hidden: false
            labels:
            - key: cpu
              value: 100m
            - key: memory
              value: 128Mi
          spec:
            resources:
              requests:
                cpu: 100m
                memory: 128Mi
        - id: small_cpu
          spawner:
            description: Pod with 1 CPU, 2 GB RAM
            displayName: Small CPU
            hidden: false
            labels:
            - key: cpu
              value: 1000m
            - key: memory
              value: 2Gi
          spec:
            affinity: {}
            resources:
              requests:
                cpu: "1"
                memory: 2Gi
        - id: big_gpu
          spawner:
            description: Pod with 4 CPU, 16 GB RAM, and 1 GPU
            displayName: Big GPU
            hidden: false
            labels:
            - key: cpu
              value: 4000m
            - key: memory
              value: 16Gi
            - key: gpu
              value: "1"
          spec:
            affinity: {}
            resources:
              limits:
                nvidia.com/gpu: "1"
              requests:
                cpu: "4"
                memory: 16Gi
            tolerations:
            - effect: NoSchedule
              key: nvidia.com/gpu
              operator: Exists
    podMetadata:
      annotations:
        my-workspace-kind-annotation: my-value
      labels:
        my-workspace-kind-label: my-value
    ports:
    - httpProxy:
        removePathPrefix: false
        requestHeaders: {}
      portId: jupyterlab
    securityContext:
      fsGroup: 100
    serviceAccount:
      name: default-editor
    volumeMounts:
      home: /home/jovyan
  spawner:
    deprecated: false
    deprecationMessage: This WorkspaceKind will be removed on 20XX-XX-XX, please use
      another WorkspaceKind.
    description: A Workspace which runs JupyterLab in a Pod
    displayName: JupyterLab Notebook
    hidden: false
    icon:
      url: https://jupyter.org/assets/favicons/apple-touch-icon-152x152.png
    logo:
      url: https://upload.wikimedia.org/wikipedia/commons/3/38/Jupyter_logo.svg
status:
  podTemplateOptions:
    imageConfig:
    - id: jupyterlab_scipy_180
      workspaces: 0
    - id: jupyterlab_scipy_190
      workspaces: 1
    podConfig:
    - id: tiny_cpu
      workspaces: 1
    - id: small_cpu
      workspaces: 0
    - id: big_gpu
      workspaces: 0
  workspaces: 1

thesuperzapper · 2025-08-07T18:48:17Z

workspaces/controller/api/v1beta1/workspacekind_types.go

+// PortId represents a port identifier
+//   - this is NOT used as the Container or Service port name, but as part of the HTTP path
+//   - this is used to reference the port in the `imageconfig` ports.[].id
+//   - this is also used to reference the port in the podtemplate ports.[].portId
+//


Suggested change

// PortId represents a port identifier

// - this is NOT used as the Container or Service port name, but as part of the HTTP path

// - this is used to reference the port in the `imageconfig` ports.[].id

// - this is also used to reference the port in the podtemplate ports.[].portId

//

// PortId the id of the port

thesuperzapper · 2025-08-07T18:50:26Z

workspaces/controller/api/v1beta1/workspacekind_types.go

@@ -115,9 +125,9 @@ type WorkspaceKindPodTemplate struct {
 	// volume mount paths
 	VolumeMounts WorkspaceKindVolumeMounts `json:"volumeMounts"`

-	// http proxy configs (MUTABLE)
+	// ports that the container listens on


Suggested change

// ports that the container listens on

// ports that the container listens on

// +listType:="map"

// +listMapKey:="id"

thesuperzapper · 2025-08-07T18:50:57Z

workspaces/controller/api/v1beta1/workspacekind_types.go

+	// the id of the port
+	// - identifier for the port in `imageconfig` ports.[].id
+	// +kubebuilder:example="jupyterlab"
+	PortId PortId `json:"portId"`


Probably lets just make this id, because its kind of implied by the ports list.

thesuperzapper · 2025-08-07T19:18:01Z

workspaces/controller/api/v1beta1/workspacekind_types.go

+	// - identifier for the port in `imageconfig` ports.[].id
+	// +kubebuilder:example="jupyterlab"
+	PortId PortId `json:"portId"`
+


Because all ports of the same ID should have the same protocol, we should remove the Protocol from imageConfigValue to here.

thesuperzapper · 2025-08-07T19:19:56Z

workspaces/controller/api/v1beta1/workspacekind_types.go

+	// - identifier for the port in `imageconfig` ports.[].id
+	// +kubebuilder:example="jupyterlab"
+	PortId PortId `json:"portId"`
+


We should add a DefaultDisplayName field here, and make the one in imageConfigValue optional (pointer) so that people can override it for a specific image config option.

github-project-automation bot added this to Kubeflow Notebooks Jul 29, 2025

google-oss-prow bot added the do-not-merge/work-in-progress label Jul 29, 2025

github-project-automation bot moved this to Needs Triage in Kubeflow Notebooks Jul 29, 2025

google-oss-prow bot added area/controller area - related to controller components area/v2 area - version - kubeflow notebooks v2 labels Jul 29, 2025

google-oss-prow bot requested review from andyatmiami, kimwnasptd and thesuperzapper July 29, 2025 05:11

google-oss-prow bot added the size/L label Jul 29, 2025

harshad16 mentioned this pull request Jul 31, 2025

feat(ws): setup virtualservice to route traffic to relevant workspace #500

Open

harshad16 marked this pull request as ready for review July 31, 2025 05:14

google-oss-prow bot removed the do-not-merge/work-in-progress label Jul 31, 2025

google-oss-prow bot added the ok-to-test label Jul 31, 2025

andyatmiami reviewed Aug 1, 2025

View reviewed changes

andyatmiami suggested changes Aug 1, 2025

View reviewed changes

google-oss-prow bot assigned andyatmiami Aug 1, 2025

andyatmiami reviewed Aug 1, 2025

View reviewed changes

workspaces/controller/config/crd/bases/kubeflow.org_workspacekinds.yaml Outdated Show resolved Hide resolved

andyatmiami reviewed Aug 1, 2025

View reviewed changes

workspaces/controller/internal/controller/suite_test.go Outdated Show resolved Hide resolved

andyatmiami reviewed Aug 1, 2025

View reviewed changes

workspaces/controller/internal/webhook/suite_test.go Outdated Show resolved Hide resolved

andyatmiami reviewed Aug 1, 2025

View reviewed changes

workspaces/controller/api/v1beta1/workspacekind_types.go Outdated Show resolved Hide resolved

andyatmiami reviewed Aug 1, 2025

View reviewed changes

workspaces/controller/internal/webhook/workspacekind_webhook.go Outdated Show resolved Hide resolved

harshad16 force-pushed the port-in-wsk-podtemplate branch from 5170f6e to 438e485 Compare August 6, 2025 18:56

google-oss-prow bot added the area/backend area - related to backend components label Aug 6, 2025

include ports array in podtemplate for httpProxy setting

3239dde

- add validation webhook for podtemplate.ports - update the sample workspacekind with ports reference - referencing same id for portid in imageconfig and podtemplate.ports Signed-off-by: Harshad Reddy Nalla <[email protected]>

harshad16 force-pushed the port-in-wsk-podtemplate branch from 438e485 to 3239dde Compare August 6, 2025 19:40

google-oss-prow bot added the lgtm label Aug 6, 2025

andyatmiami approved these changes Aug 6, 2025

View reviewed changes

thesuperzapper reviewed Aug 7, 2025

View reviewed changes

		// +kubebuilder:validation:MinLength:=1
		// +kubebuilder:validation:MaxLength:=32

feat(ws): add spec.podTemplate.ports[] to WorkspaceKind #507

Are you sure you want to change the base?

feat(ws): add spec.podTemplate.ports[] to WorkspaceKind #507

Uh oh!

Conversation

harshad16 commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-oss-prow bot commented Jul 29, 2025

Uh oh!

thesuperzapper commented Jul 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andyatmiami left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andyatmiami commented Aug 6, 2025

Uh oh!

andyatmiami commented Aug 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

harshad16 commented Jul 29, 2025 •

edited

Loading