configstream consumer never refreshes or re-acquires its RAR session, so any stream interruption is permanent

[lebauce]

Summary

comp/core/configstreamconsumer/impl/consumer.go treats its Remote Agent Registry session_id as immortal. It never refreshes it, and it never re-acquires one when the server rejects it. The result is that a config-stream client (system-probe, trace-agent, process-agent) which loses its stream once never recovers: it retries forever with a dead session_id and stays frozen on its boot-time config snapshot.

Observed on a recent main build in a large internal Kubernetes deployment, as an unbounded loop of:

Config stream error: stream receive error: rpc error: code = PermissionDenied
  desc = session_id '<uuid>' not found: remote agent must register with RAR
  before subscribing to config stream, reconnecting...

at one line per 5s per affected process, indefinitely.

Defect 1 — the session is never refreshed, so RAR always evicts it

helper.RegisterRemoteAgent returns a recommended refresh interval, and the consumer discards it:

// consumer.go:194
sessionID, _, regErr := helper.RegisterRemoteAgent(...)

There is no Refresh call anywhere in consumer.go. Meanwhile RAR's reaper ticks every second (comp/core/remoteagentregistry/impl/registry.go:309) and deletes any session idle beyond remote_agent.registry.idle_timeout — default 30s (pkg/config/setup/all_settings.go:1044-1049), against a recommended refresh interval of 10s (registry.go:322).

So every config-stream consumer's session is evicted 30 seconds after its stream opens, on every deployment. This goes unnoticed because the authorization gate is only evaluated at stream open (comp/core/configstream/server/server.go:64) and never re-checked in the send loop (server.go:75-98). The session becomes a corpse the live stream doesn't notice.

Defect 2 — PermissionDenied is not special-cased, so the failure is permanent

• registerWithBackoff() has exactly one caller: consumer.go:159, inside start(). It runs once per process lifetime.
• streamLoop() (consumer.go:244-269) handles every error identically: log, increment reconnect_count, sleep 5s, retry.
• connectAndStream() (consumer.go:283) re-sends metadata.New(map[string]string{"session_id": c.sessionID}) — the field is written once at consumer.go:200 and never reassigned.
• There is no status.FromError / codes.* inspection in the file, so the consumer cannot even distinguish PermissionDenied from a transient error.

Combined with defect 1, the first stream interruption is fatal. Triggers include a core-agent restart, which wipes the registry entirely — it is in-memory only (registry.go:198) and rebuilt empty on start (registry.go:71).

Consequence beyond the log noise

Because disk load is skipped while the stream is active:

// comp/core/config/config.go:88-96
if deps.Cfgstream != nil && deps.Cfgstream.IsActive() {
    // Snapshot already in the global builder; skip disk load to avoid
    // clobbering streamed values via same-source last-write-wins.

a stranded client is permanently pinned to whatever it received at boot and silently ignores every later config change. The code already anticipates the unrecoverable case — consumer.go:326-327 logs "the core agent may have restarted. This sub-process must be restarted to accept a new configuration" — but that path only runs if a snapshot arrives, which PermissionDenied prevents.

The sibling client gets this right

comp/core/remoteagent/helper/serverhelper.go:204-246 handles it correctly: on refresh failure it logs "entering registration loop", clears s.sessionID (:240), and re-registers via registerWithAgent() (:217). The configstream consumer bypasses that helper and never inherited the behaviour.

Suggested fix

Both parts are needed; part 1 alone stops the log storm but not the staleness.

1. Re-register on authorization failure. In streamLoop/connectAndStream, inspect the status and on codes.PermissionDenied (and codes.Unauthenticated, for server.go:56/:61) call registerWithBackoff() to mint a fresh session_id before retrying — mirroring serverhelper.go:238-241. Note this also requires handling the sequence-ID reset: applySnapshot (consumer.go:325-329) currently rejects the post-restart snapshot as stale, so re-registration alone would fix the spam but leave the client on stale config.
2. Keep the session alive. Either tick RefreshRemoteAgent at the recommended_refresh_interval already returned and discarded at consumer.go:194, or treat an open config stream as liveness server-side (e.g. have the send loop in server.go:75-98 bump LastSeen). Otherwise the 30s idle_timeout keeps evicting every consumer regardless of part 1.

Provenance

Introduced by #51026, which rewrote consumer.go into the current register-once + stream-forever split. The PermissionDenied gate in server.go predates it (#46206, relanded #50385). #51480 ("Add RAR API to refresh the configuration") was closed unmerged.

Unrelated but adjacent: #55310 touches configstreambootstrap/nodetreemodel for env-var-sourced settings and does not affect session lifecycle or re-registration.

[lebauce]

cc @rahulkaukuntla