feat: stream live Workspace updates to the dashboard via SSE

[christian-heusel]

Note

The idea of this PR is to be a conversation starter and not intended to be merged in its current state!

---

What this does

Replace the Workspaces table fixed interval polling with live updates over Server-Sent Events (SSE). The table now reflects backend changes (state transitions, pause/resume, pods coming up, create/delete) almost instantly, without fixed-interval refetching.

2026-08-28.01-39-50.mp4

Motivation

Today the frontend keeps the Workspaces table fresh by re-fetching the entire list every 30 s via a countdown (RefreshCounter → POLL_INTERVAL). This means up to 30 s of staleness after a workspace changes, and a full list re-fetch on every tick regardless of whether anything changed.

The backend already runs a controller-runtime manager with a shared informer cache on Workspace, and its RBAC already grants watch, so the real-time signals already exists but were never surfaced to the frontend.

How it works

Workspace informer -> change hub (pub/sub) -> SSE handler -> browser EventSource -> table

• A watch query parameter on the existing list routes upgrades the response to an event stream, mirroring the Kubernetes API convention: GET /api/v1/workspaces/{namespace}?watch=true (also watch=1; cluster-wide GET /api/v1/workspaces?watch=true). Same route, same auth as the non-streaming list.
• The stream emits an initial full-list snapshot, then a fresh snapshot on every change. Bursts are debounced (~250 ms) and idle connections get a heartbeat comment every 20 s.
• Sending full snapshots (rather than deltas) keeps the frontend a drop-in and makes reconnects self-healing — any change during a brief disconnect is captured by the next snapshot.

Changes

Backend (workspaces/backend)

• internal/streaming/hub.go: small concurrency-safe pub/sub broadcaster (namespace-filtered, non-blocking, coalescing) with unit tests.
• cmd/main.go: registers a Workspace informer event handler that publishes changes to the hub.
• api/workspaces_stream_handler.go + a watch branch in the existing list handler: the SSE endpoint. Clears the per-connection WriteTimeout via http.ResponseController so long-lived streams aren't torn down; gzip is configured to pass text/event-stream through untouched so flushing works.

Frontend (workspaces/frontend)

• hooks/useWorkspacesLive.ts: SSE client (via @microsoft/fetch-event-source, so it can set the dev kubeflow-userid header the axios layer already uses). Returns a FetchState tuple identical to the polling hook, plus a connection status.
• components/LiveStatusIndicator.tsx: "Live"/"Reconnecting…" indicator that replaces the countdown in live mode (manual refresh preserved).
• pages/Workspaces/Workspaces.tsx: selects the live or polling variant.

Testing

Click to see test report

SSE Live-Update: Test Report

Feature: Live Workspace table updates via Server-Sent Events (branch feat/workspace-live-updates)
Date: 2026-08-28
Target: Kubeflow Notebooks v2, Tilt deployment via Istio gateway (https://localhost:8443), backend behind istio-envoy over HTTP/2.
Endpoint under test: GET /workspaces/api/v1/workspaces/{namespace}?watch=true (and cluster-wide /workspaces?watch=true).

Method

Changes were driven through the BFF's own pause/start action
(POST .../workspaces/{namespace}/{name}/actions/pause), which exercises the
full API → controller → informer → hub → SSE loop end-to-end. Tilt's Kind
cluster was not reachable via kubectl from this shell (only a GKE context was
configured), so backend/pod restart was simulated at the network layer (abrupt
client disconnect + reconnect) rather than by killing the pod.

All local pre-checks passed before live testing: backend go build/go vet/
gofmt/golangci-lint (clean on changed files) + hub unit tests with -race;
frontend test:type-check, test:lint, and test:jest (452 tests).

Results summary

#	Test	Result
T1	Response headers	✅ text/event-stream, cache-control: no-cache, x-accel-buffering: no; HTTP/2 via istio-envoy
T1b	Served uncompressed with Accept-Encoding: gzip (browser-realistic)	✅ No content-encoding: gzip; body is plaintext data:
T2	Missing Kubeflow-Userid → 401	✅ 401 + error envelope
T3	watch=false / absent → normal JSON	✅ application/json, one-shot list
T3b	watch=1 alias	✅ streams
T4	Initial snapshot immediate on connect	✅ full list pushed at once
T5	Snapshot payload == non-streaming list	✅ byte-identical
T6	Namespace scoping (namespaced vs cluster-wide)	✅ both stream correctly
T7	Nonexistent/empty namespace	✅ stream opens, data: {"data":[]}
R1	Self-heal: change during disconnect appears on reconnect	✅ PASS (paused True→False captured, ~300 ms reconnect)
R2	Live-push latency (change → snapshot)	✅ ≈271 ms (debounce-bound)
R3	Concurrent clients fan-out (1 change → N clients)	✅ both received
R4	20× rapid reconnect churn → backend health	✅ list & stream still 200
H	Heartbeat keep-alive	✅ : ping at exactly 20 s idle

Detailed observations

Protocol & headers (T1 / T1b)

HTTP/2 200
content-type: text/event-stream
cache-control: no-cache
x-accel-buffering: no
vary: Accept-Encoding
server: istio-envoy

Critically, even when the client sends Accept-Encoding: gzip (which every
browser does), the response is not gzip-compressed — the gzhttp
ExceptContentTypes("text/event-stream") configuration works, so the browser
EventSource/parser receives readable data: frames.

Live push & lifecycle (T4, R2)

A single pause→unpause→pause sequence produced snapshots tracking the real
controller lifecycle, each pushed within ~1 s of the underlying change:

00:32:44  test.state = Paused       (paused=true)   ← initial snapshot
00:32:47  test.state = Pending      (paused=false)  ← after unpause POST
00:32:48  test.state = Running      (paused=false)  ← pod up
00:32:50  test.state = Terminating  (paused=true)   ← after re-pause POST
00:32:54  test.state = Paused       (paused=true)   ← settled

Precise latency measurement (first snapshot after a change):

POST(unpause) t0
   +271 ms  state=Pending   <-- first post-change snapshot
  +1868 ms  state=Running   (controller starting the pod — real work, not our latency)

Push latency is ≈271 ms, essentially the 250 ms debounce interval plus
network overhead.

Heartbeat (H)

00:33:29  data: {…}     ← initial snapshot
00:33:49  : ping        ← heartbeat, exactly 20 s later

Keeps idle connections alive through proxies/gateways.

Namespace scoping & edge cases (T5–T7)

• Streamed initial snapshot is byte-identical to the non-streaming list endpoint.
• Cluster-wide (/workspaces?watch=true) and namespaced
  (/workspaces/default?watch=true) both stream correctly.
• A nonexistent namespace still opens the stream and emits data: {"data":[]}.

Connection-loss behavior (focus area)

Frontend runtime path (useWorkspacesByNamespaceLive +
@microsoft/fetch-event-source):

1. Drop detected → onerror runs; non-fatal, so the hook sets
   connectionStatus = 'reconnecting' (UI shows "Reconnecting…") and returns
   undefined.
2. Library retries after a fixed 1000 ms (DefaultRetryInterval; no
   exponential backoff — confirmed in the library source fetch.js).
3. Reconnect succeeds → onopen re-validates the text/event-stream
   content type → connectionStatus = 'live', error cleared → onmessage
   delivers a fresh full snapshot.
4. Because every connection re-sends the entire list, any changes that occurred
   during the gap are already present in that first snapshot — no missed
   updates. Proven empirically by R1: the paused flag was flipped while no
   stream was connected, and the reconnected stream's initial snapshot already
   reflected the new value (reconnect completed in ~300 ms).
5. Fatal errors only: a 4xx (e.g. 401/403) is treated as non-retryable →
   connectionStatus = 'error', stopping the reconnect loop (no reconnect
   storm on an auth failure).

Backend robustness: 20 abrupt connect/disconnect cycles left the backend
fully responsive (both the list endpoint and a fresh stream returned 200).
Per-connection resources (goroutine, hub subscription, heartbeat ticker) are
released via defer on r.Context().Done().

Findings & recommendations

1. Fixed 1 s reconnect, no backoff (library default). Ideal for brief blips
   (fast recovery), but during a prolonged outage every open browser tab
   retries at ~1 req/s against the gateway. Consider adding jittered exponential
   backoff (the hook's onerror can return an increasing interval) before a wide
   rollout. Low-effort, non-blocking.
2. Connect-time authorization only (by design, matching the existing model):
   a long-lived stream is authorized once at connect and not re-authorized
   mid-connection (SAR result cached ~10 s). Acceptable for now; note it if
   session-revocation latency ever becomes a requirement.
3. No correctness issues found — all other behavior matches the design.

Verdict

The live-update feature works correctly end-to-end against the live Tilt
deployment, and the connection-loss path is safe and self-healing: brief
drops recover within ~1 s with no lost updates, and the backend stays healthy
under reconnect churn.

Try it out

curl -N -sk -H "Kubeflow-Userid: admin" \
  "https://localhost:8443/workspaces/api/v1/workspaces/default?watch=true"
# expect an initial `data:` snapshot, `: ping` heartbeats, and a new snapshot
# whenever a workspace in the namespace changes.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andyatmiami, ederign for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• workspaces/backend/OWNERS
• workspaces/frontend/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment