[WIP] feat: implement Gateway API routing provider + standalone notebooks - #1301
Draft
aojea wants to merge 14 commits into
Draft
[WIP] feat: implement Gateway API routing provider + standalone notebooks#1301aojea wants to merge 14 commits into
aojea wants to merge 14 commits into
Conversation
google-oss-prow
Bot
added
area/ci
aojea
changed the title
|
/ok-to-test |
|
|
||
| // generateGatewayAPIHTTPRoute generates an HTTPRoute for a Workspace using the Gateway API | ||
| func (r *WorkspaceReconciler) generateGatewayAPIHTTPRoute(workspace *kubefloworgv1beta1.Workspace, workspaceKind *kubefloworgv1beta1.WorkspaceKind, service *corev1.Service, imageConfigSpec kubefloworgv1beta1.ImageConfigSpec) *gatewayv1.HTTPRoute { | ||
| namePrefix := generateNamePrefix(workspace.Name, maxVirtualServiceNameLength) |
There was a problem hiding this comment.
rename maxVirtualServiceNameLength
| } | ||
|
|
||
| // generateGatewayAPIHTTPRoute generates an HTTPRoute for a Workspace using the Gateway API | ||
| func (r *WorkspaceReconciler) generateGatewayAPIHTTPRoute(workspace *kubefloworgv1beta1.Workspace, workspaceKind *kubefloworgv1beta1.WorkspaceKind, service *corev1.Service, imageConfigSpec kubefloworgv1beta1.ImageConfigSpec) *gatewayv1.HTTPRoute { |
There was a problem hiding this comment.
how about moving this into helper package, also generateVirtualServiceHTTPRoute under separate files? so that it is easier to test comprehensively and compare the differences.
|
This is great! |
| skipIstioInstall = os.Getenv("ISTIO_INSTALL_SKIP") == "true" | ||
| isIstioAlreadyInstalled = false | ||
|
|
||
| routingProvider = func() string { |
There was a problem hiding this comment.
set skipIstioInstall = True when ROUTING_PROVIDER != "istio"?
And add a check if gateway-api is installed when ROUTING_PROVIDER == 'gateway-api'
aojea
force-pushed
the
gateway_api
branch
4 times, most recently
from
August 5, 2026 14:25
b455568 to
c91b055
Compare
Member
|
@aojea FYI that your commits are showing up as "unverified" 😅 |
Author
sorry , trying to get it in a better shape for reviewing |
Member
|
/retitle feat: implement Gateway API routing provider |
google-oss-prow
Bot
changed the title
Moves the controller and backend modules to the latest stable Kubernetes client libraries (v0.36.3) with the matching controller-runtime v0.24.1, istio.io/client-go v1.30.3, and envtest 1.36.0. This is a prerequisite for the Gateway API ExternalAuth filter (GEP-1494), which needs sigs.k8s.io/gateway-api v1.4.0 or newer and therefore k8s v0.34 or newer. Breaking changes handled: - headerrequest.New() gained a uidHeaders parameter, so the backend's request header authenticator passes nil for it. - ctrl.NewWebhookManagedBy() is now generic and takes the object it builds for, replacing the separate For() call. The generated WorkspaceKind CRD and OpenAPI spec pick up the k8s 1.36 core API surface, most visibly the new EnvVarSource.fileKeyRef and ResourceClaim.request fields. Two deprecations are suppressed rather than migrated, as both are refactors in their own right: the typed admission.Validator[T] interface requires reworking the validator methods, and apimachinery's runtime.SchemeBuilder requires moving type registration out of the *_types.go files. Signed-off-by: Antonio Ojea <aojea@google.com>
extract VirtualService route generation into separate file Signed-off-by: Antonio Ojea <aojea@google.com>
… backend, and frontend Signed-off-by: Antonio Ojea <aojea@google.com>
…rs and gateway-api Signed-off-by: Antonio Ojea <aojea@google.com>
…up to Tilt workflow Signed-off-by: Antonio Ojea <aojea@google.com>
…kend dependencies Signed-off-by: Antonio Ojea <aojea@google.com>
aojea
changed the title
aojea
marked this pull request as draft
August 18, 2026 17:34
Workspace HTTPRoutes proxy straight to a notebook pod, so nothing authenticates or authorizes a request before it reaches the workspace. The Istio path relies on the per-namespace AuthorizationPolicy that the Kubeflow Profile controller creates; the Gateway API path had no equivalent. Generated routes now carry the ExternalAuth filter from GEP-1494, pointing at an external authorization service. The filter is placed first in each rule's filter list so the authorization service sees the path the client requested, before any rewriting. This is opt-in: an empty EXTERNAL_AUTH_BACKEND_NAME leaves routes unchanged. It is wired up by the new ext-authz component and the gateway-api-ext-authz overlay, kept separate from the gateway-api overlay because ExternalAuth is an Extended, experimental feature and needs the Gateway API experimental channel CRDs. The controller refuses to start on a partial configuration rather than emitting routes the data plane would reject. Signed-off-by: Antonio Ojea <aojea@google.com>
Adds workspaces-authz, the service the ExternalAuth filter calls. It authenticates callers against an OIDC issuer and authorizes them against Kubernetes RBAC, replacing the two things Kubeflow Profiles provided: the identity header injected by oauth2-proxy behind the Istio ingress, and the per-namespace AuthorizationPolicy that bound a workspace to its owner. It speaks both protocols named by the Gateway API ExternalAuth filter, Envoy's ext_authz gRPC service and HTTP forward-auth. Istio's AuthorizationPolicy CUSTOM action calls the same gRPC service, so a single deployment covers both routing providers. Design notes: - The claims-to-username mapping mirrors the API server's OIDC options, so the identity used for the SubjectAccessReview is the identity the API server knows. That is what makes an administrator's RoleBindings meaningful, and it removes the need for the Profile controller to create them. - Authorized requests carry the ID token as a bearer token, so upstream services can revalidate it with a TokenReview instead of trusting a username header. - Identity headers are set with overwrite semantics, and the groups header is set even when empty, so a client-supplied value can never reach an upstream service. This is what the Istio AuthorizationPolicy did by pinning the ingress gateway's mTLS identity. - Workspace connect paths are validated as Kubernetes object names, which rejects traversal and percent-encoded separators without relying on the data plane to normalize the path first. - Errors reaching the issuer or the API server are returned as errors rather than denials, so the data plane fails closed and those failures stay distinguishable from a genuine 403. Signed-off-by: Antonio Ojea <aojea@google.com>
The backend trusted the kubeflow-userid header unconditionally, so its security rested entirely on the network path forcing every request through a proxy that overwrites that header. It also meant the username being authorized with a SubjectAccessReview was one the API server had never seen; it only worked because the Profile controller happened to create RoleBindings for that exact string. When ENABLE_TOKEN_AUTH is set, a request carrying a bearer token is now authenticated with a TokenReview instead. The identity then comes from the API server, so the SubjectAccessReview and an administrator's RoleBindings agree on who the caller is without any Kubeflow-specific component in the middle. Bearer tokens are handled separately from the header authenticator rather than chained through a union, so that a caller cannot present a token that fails review and still be authenticated by a header it controls. Off by default, which leaves the existing header behaviour unchanged. Signed-off-by: Antonio Ojea <aojea@google.com>
…flow Adds a `tilt-up-gateway-api-ext-authz` flavor that runs the workspaces UI behind Dex and the authorization service, so the standalone story can be exercised without installing any Kubeflow component. The RBAC in manifests/dex/rbac.yaml is the whole access model: plain RoleBindings naming the identities Dex issues. No Profile, no KFAM. Also pins the Gateway API CRDs to v1.6.1 to match the version the controller builds against, rather than the v1.1.0 it installed before, and makes the channel selectable. The ext-authz flavor needs the experimental channel because ExternalAuth is an experimental feature. Three things cannot be static manifests and are handled by scripts/setup-ext-authz.sh: the generated cookie key and client secret, the Gateway's self-signed CA which the authorization service must trust to complete OIDC discovery, and a host alias so the issuer URL resolves to the Gateway from inside the cluster as well as from the browser. Signed-off-by: Antonio Ojea <aojea@google.com>
Workspace routes proxy straight to a notebook pod, so a missing ExternalAuth filter means the workspace is reachable by anyone. Nothing caught that: the filter can go missing from a config mistake, and it is silently dropped when the Gateway API standard channel CRDs are installed, because ExternalAuth only exists in the experimental channel. With ENABLE_EXT_AUTHZ=true the suite deploys the gateway-api-ext-authz overlay and asserts that every rule of the generated HTTPRoute starts with an ExternalAuth filter pointing at the authorization service. The ordering matters as much as the presence, since a filter placed after the rewrite would hand the authorization service a path the client never asked for. The suite also fails early with a pointed message when the experimental channel is missing, rather than letting the pruned filter look like a passing run. Signed-off-by: Antonio Ojea <aojea@google.com>
A workspace pod serves an unauthenticated notebook, so any pod that can reach its Service gets a session. Nothing restricted that: the routing layer only sees traffic that arrives through the gateway, and the controller generated no policy of its own. Another tenant's workspace, which by definition runs user-supplied code, could reach ws-<name>.<namespace>.svc directly. Istio deployments are covered by the per-namespace AuthorizationPolicy that the Kubeflow Profile controller creates. This is the equivalent for deployments without Istio and without Profiles. The generated policy selects the workspace pods and allows ingress only from the routing layer. Selecting the pods is what does the work: once any ingress policy selects a pod, traffic that no policy allows is dropped, so no separate default-deny is needed in the user namespace. Enabled by the new workspace-network-policy component, which both gateway-api overlays now include. Disabled by default elsewhere, so existing deployments are unchanged. The field indexers are collapsed onto a shared helper. The five per-type blocks were identical apart from the object, and adding a sixth pushed the function past the cyclomatic complexity limit. Signed-off-by: Antonio Ojea <aojea@google.com>
The gateway-api components allowed the whole kubeflow namespace to reach the backend and frontend. The backend trusts the identity header its caller sets, so any pod scheduled into that namespace could assert any identity. The Istio components pin the ingress gateway's mTLS identity instead; this is the closest equivalent without a mesh. Also adds the policy the authorization service was missing. It decides who a request belongs to, so a direct caller could obtain an allow decision for an identity of its choosing. The peer selectors use matchExpressions rather than matchLabels because kustomize's `labels` transformer runs with includeSelectors and rewrites matchLabels in selector fields, including the peer podSelector of a NetworkPolicy. With matchLabels it injected each component's own labels into the peer, producing a selector that matches nothing and silently dropping all traffic to that component. Signed-off-by: Antonio Ojea <aojea@google.com>
Listing namespaces required the caller to hold cluster-wide `list namespaces`, and returned every namespace the backend ServiceAccount could see. Both are wrong outside the Kubeflow distribution. A normal user does not hold a cluster-wide permission, so the endpoint answered 403 and the namespace picker came up empty. It was not noticed because the Central Dashboard populated the picker from KFAM's bindings API rather than from this endpoint, so nothing standalone exercised it. Anyone who did hold the permission was shown every namespace in the cluster, whether or not they could use it. The endpoint now requires only authentication and filters the result to the namespaces where the caller may list workspaces. There is no Kubernetes API that answers this directly, so each namespace is checked with a SubjectAccessReview. The checks run concurrently with a bound, since each is a round trip, and the authorizer already caches decisions. Signed-off-by: Antonio Ojea <aojea@google.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit introduces Gateway API (v1) support as an alternative to Istio for routing traffic to workspaces.
Key changes:
This allows Kubeflow users to adopt any Gateway API-compliant ingress controller (like Envoy Gateway) instead of being strictly coupled to Istio.
Assisted by AI