Skip to content

fix(ws): recover broken chat connections and subscriptions - #1301

Closed
materemias wants to merge 1 commit into
siteboon:mainfrom
materemias:fix/ws-channel-recovery
Closed

materemias wants to merge 1 commit into
siteboon:mainfrom
materemias:fix/ws-channel-recovery

Conversation

@materemias

@materemias materemias commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Purpose

Supersedes #1283 with one commit based on upstream main at 5e73a49b. This carries the reviewed reconnect fix and adds recovery when a replacement socket never acknowledges its chat subscription.

Restore the shared /ws chat connection when it stops working, then restore the selected chat's subscription and persisted history. Provider startup, abort behavior, defaults, authentication modes, /shell, and plugin WebSockets are unchanged.

Behavior

  • Send a nonce-bearing chat.ping 30 seconds after connection or a matching pong. If no matching pong arrives within 3 seconds, retire the socket and retry after the existing 3-second delay. A handshake stuck for 30 seconds also retries. The gateway rejects missing or non-string nonces with protocol_error / INVALID_NONCE, without emitting a pong or closing the socket.
  • Keep visible-page and pageshow probes as faster triggers, but do not depend on those events. Use one transport timer and reject stale socket callbacks; token changes, logout, and unmount clean up the old connection.
  • Publish only open sockets to chat. After replacement, subscribe first and wait for the matching chat_subscribed acknowledgement before refreshing history. Synthetic reconnect notifications never enter transcript storage.
  • Bound the reconnect acknowledgement wait to 10 seconds. If it expires, close that socket and let the existing transport reconnect and resubscribe, even when heartbeats remain healthy. A timer-only HTTP fallback would not restore live subscription, so history still waits for a successful acknowledgement. Persistent server errors keep retrying rather than being treated as success.
  • Cancel the acknowledgement deadline on acknowledgement, socket/session change, or unmount. Preserve initial history loading and hidden-chat HTTP gating; visibility-only changes do not resubscribe.

The server sends replay after the acknowledgement, so replay and the history request may overlap. Browser timer suspension/throttling can delay recovery. Failed application messages are not automatically resent. Retry backoff and provider-run interruption changes are outside this PR.

Verification

Native Chromium fixtures, using actual client hooks and gateway code:

  • A TCP-blackholed socket on upstream 5e73a49b stayed apparently connected with no replacement or new message after 51 seconds. The carried transport fix made exactly one replacement by the 40-second observation and received the message, without visibility/page-show events.
  • For the new failure path, forced the gateway's pending-approval lookup to throw during reconnect subscription while heartbeats continued working. The previous PR left history blocked. This revision closed the unacknowledged socket, received a fresh subscription acknowledgement, refreshed persisted history, and received a live message through the run registry.
  • The new visible/hidden missing-ack regressions fail against the original PR hook and pass with this revision. Tests also cover acknowledgement/deadline cleanup, auth/token rotation, stale callbacks, healthy sockets, initial loading, and hidden-session activation. The session-switch test also fires the old deadline after the new session renders but before passive-effect cleanup; it fails without the session-identity guard and passes with it.

These are isolated fixtures with stubbed authentication/provider execution, not a physical laptop-suspend test or a provider-backed conversation.

The real-socket gateway regression rejects a malformed nonce without a pong, then confirms valid pings still work on the same socket without provider calls or broadcasts. It fails before nonce validation and passes afterward.

Commands on this branch:

  • npm run test:client: 424 passed.
  • npm test: 416 passed, one Codex fixture-dependent skip.
  • npm run typecheck: passed.
  • npm run build: passed, with CSS/bundle-size warnings.
  • npm run lint: successful exit, 132 warnings and no errors.

Summary by CodeRabbit

  • New Features

    • Added WebSocket health checks to detect unavailable connections and recover automatically.
    • Added support for resuming chat sessions after reconnecting, including subscription confirmation and message catch-up.
    • Added reconnection handling when returning to an active browser tab or window.
    • Added support for chat.ping requests with corresponding pong responses.
  • Bug Fixes

    • Improved handling of connection timeouts, failed sends, interrupted handshakes, and stale connections.
    • Prevented reconnect notifications from appearing as chat messages.
    • Improved authentication and logout handling during WebSocket recovery.

Copilot AI lite review requested due to automatic review settings September 9, 2026 15:45
@coderabbitai

coderabbitai Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 6a6d49ab-bcba-4c5d-bed4-4821070e3364

📥 Commits

Reviewing files that changed from the base of the PR and between 0f8be0a and 5085474.

📒 Files selected for processing (4)
  • server/modules/websocket/README.md
  • server/modules/websocket/services/chat-websocket.service.ts
  • server/modules/websocket/tests/chat-ping.test.ts
  • src/modules/chat/hooks/useChatSessionState.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • server/modules/websocket/README.md

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

Changes

The WebSocket server now handles chat.ping and returns pong. The client adds heartbeat probes, handshake timeouts, retry handling, and resume probing. Chat reconnect recovery waits for subscription acknowledgement before refreshing session history.

WebSocket protocol

Layer / File(s) Summary
Ping/pong protocol
server/modules/websocket/README.md, server/modules/websocket/services/chat-websocket.service.ts, server/modules/websocket/tests/chat-ping.test.ts, server/shared/types.ts, src/shared/types.ts
The protocol supports chat.ping and nonce-validated pong frames. Tests verify direct replies without runtime calls or broadcasts.
Transport health and retry
src/shared/context/WebSocketContext.tsx, src/shared/tests/webSocketContext.test.tsx
The provider manages connection deadlines, heartbeat probes, failed sends, socket replacement, visibility events, authentication gates, and retries.
Acknowledged chat resubscription
src/modules/chat/ChatInterface.tsx, src/modules/chat/hooks/*, src/modules/chat/tests/*
useChatSessionState owns reconnect catch-up. It waits for matching chat_subscribed events, delays refreshes, closes sockets after 10 seconds without acknowledgement, and preserves cleanup behavior.

Sequence Diagram(s)

sequenceDiagram
  participant WebSocketProvider
  participant useChatSessionState
  participant ChatServer
  participant SessionMessagesAPI
  WebSocketProvider-->>useChatSessionState: websocket_reconnected
  useChatSessionState->>ChatServer: chat.subscribe
  ChatServer-->>useChatSessionState: chat_subscribed
  useChatSessionState->>SessionMessagesAPI: request latest messages
Loading

Suggested reviewers: blackmammoth

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 50854

The WebSocket recovery changes have no remaining verified merge-blocking risk.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 23.08% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 13 functions across 11 files. (1 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: recovery for broken WebSocket chat connections and subscriptions.
Full details: Docstring Coverage

Explanation

Docstring coverage is 23.08% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 13 functions across 11 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit reads each line,
The patch grows clear beneath the moon,
Small changes hop in place,
Tests guard the garden path,
Reviews bloom before the dawn.

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new chat.ping handler can emit pong frames with a null nonce on invalid input, which breaks the documented nonce-matching contract and should be corrected before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens the chat /ws transport against half-open connections by adding an application-level ping/pong probe + bounded reconnect handshakes, and updates chat session logic to only refresh persisted history after a replacement socket has explicitly re-acknowledged its subscription.

Changes:

  • Add periodic chat.ping probes with pong deadlines + connect deadlines to retire stuck sockets and trigger reconnects.
  • Ensure chat subscription recovery waits for a matching chat_subscribed acknowledgement (with a 10s bound) before refreshing persisted history.
  • Add targeted frontend and backend tests plus protocol documentation updates for the new heartbeat and resubscribe behavior.
File summaries
File Description
src/shared/types.ts Update shared transport documentation to include pong as a gateway frame kind.
src/shared/context/WebSocketContext.tsx Implement ping/pong probing, connect timeout, and safer reconnect/retirement logic for the shared chat socket.
src/shared/tests/webSocketContext.test.tsx Add coverage for half-open recovery, pong matching, stale callbacks, token rotation, and cleanup behaviors.
src/modules/chat/hooks/useChatSessionState.ts Add reconnect acknowledgement gating + deadline to prevent history refresh before a replacement subscription is confirmed.
src/modules/chat/hooks/useChatRealtimeHandlers.ts Remove reconnect callback path and ensure reconnect notifications never become transcript rows.
src/modules/chat/ChatInterface.tsx Wire subscribe into session state and remove the prior reconnect handler wiring.
src/modules/chat/tests/transcriptScrollOwnership.test.tsx Update test harness to satisfy the new subscribe requirement.
src/modules/chat/tests/reconnectSubscription.test.tsx Add coverage for resubscribe ack gating, hidden behavior, timeouts, and stale listener protection.
server/shared/types.ts Add pong to gateway event kind union.
server/modules/websocket/services/chat-websocket.service.ts Add chat.ping handling that replies with pong.
server/modules/websocket/tests/chat-ping.test.ts Add a backend test ensuring pings only reply to the origin socket and do not touch providers/broadcast.
server/modules/websocket/README.md Document the broken-channel recovery mechanism and the resubscribe-ack gating.
Review details
  • Files reviewed: 12/12 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +640 to +645
case 'chat.ping':
sendJson(ws, {
kind: 'pong',
nonce: typeof data.nonce === 'string' ? data.nonce : null,
});
return;

import { api } from '@/shared/api';
import type { MarkSessionIdle, SessionActivityMap,Project,ProjectSession,LLMProvider,NormalizedMessage,ChatMessage,DiffCalculator } from '@/shared/types';
import type { MarkSessionIdle, SessionActivityMap,Project,ProjectSession,LLMProvider,NormalizedMessage,ChatMessage,DiffCalculator,ServerEvent } from '@/shared/types';
@materemias
materemias force-pushed the fix/ws-channel-recovery branch from 6678985 to 0f8be0a Compare September 9, 2026 15:52
@materemias
materemias force-pushed the fix/ws-channel-recovery branch from 0f8be0a to 5085474 Compare September 9, 2026 20:10
@blackmammoth

Copy link
Copy Markdown
Member

@materemias please break up the PR into small pieces with a reproduction step of the error for each.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants