[CAE-1088] & [CAE-1072] feat: RESP3 notifications support & Hitless notifications handling #3418

ndyakov · 2025-06-26T17:44:01Z

This PR Includes both RESP3 notifications and Hitless Upgrade handlers.

- Add PushNotificationRegistry for managing notification handlers - Add PushNotificationProcessor for processing RESP3 push notifications - Add client methods for registering push notification handlers - Add PubSub integration for handling generic push notifications - Add comprehensive test suite with 100% coverage - Add push notification demo example This system allows handling any arbitrary RESP3 push notification with registered handlers, not just specific notification types.

- Change PushNotificationRegistry to allow only one handler per command - RegisterHandler methods now return error if handler already exists - Update UnregisterHandler to remove handler by command only - Update all client methods to return errors for duplicate registrations - Update comprehensive test suite to verify single handler behavior - Add specific test for duplicate handler error scenarios This prevents handler conflicts and ensures predictable notification routing with clear error handling for registration conflicts.

- Remove all global push notification handler functionality - Simplify registry to support only single handler per notification type - Enable push notifications by default for RESP3 connections - Update comprehensive test suite to remove global handler tests - Update demo to show multiple specific handlers instead of global handlers - Always respect custom processors regardless of PushNotifications flag Push notifications are now automatically enabled for RESP3 and each notification type has a single dedicated handler for predictable behavior.

- Add PushNotificationProcessor field to pool.Conn for connection-level processing - Modify connection pool Put() and isHealthyConn() to handle push notifications - Process pending push notifications before discarding connections - Pass push notification processor to connections during creation - Update connection pool options to include push notification processor - Add comprehensive test for connection health check integration This prevents connections with buffered push notification data from being incorrectly discarded by the connection health check, ensuring push notifications are properly processed and connections are reused.

…d clients - Remove unused Timestamp and Source fields from PushNotificationInfo - Add pushProcessor to newConn function to ensure Conn instances have push notifications - Add push notification methods to Conn type for consistency - Ensure cloned clients and Conn instances preserve push notification functionality This fixes issues where: 1. PushNotificationInfo had unused fields causing confusion 2. Conn instances created via client.Conn() lacked push notification support 3. All client types now consistently support push notifications

- Add 10 new unit tests covering all previously untested code paths - Test connection pool integration with push notifications - Test connection health check integration - Test Conn type push notification methods - Test cloned client push notification preservation - Test PushNotificationInfo structure validation - Test edge cases and error scenarios - Test custom processor integration - Test disabled push notification scenarios Total coverage now includes: - 20 existing push notification tests - 10 new comprehensive coverage tests - All new code paths from connection pool integration - All Conn methods and cloning functionality - Edge cases and error handling scenarios

- Remove RegisterPushNotificationHandlerFunc methods from all types - Remove PushNotificationHandlerFunc type adapter - Keep only RegisterPushNotificationHandler method for cleaner interface - Remove unnecessary push notification constants (keep only Redis Cluster ones) - Update all tests to use simplified interface with direct handler implementations Benefits: - Cleaner, simpler API with single registration method - Reduced code complexity and maintenance burden - Focus on essential Redis Cluster push notifications only - Users implement PushNotificationHandler interface directly - No functional changes, just interface simplification

- Add sync.RWMutex to PushNotificationProcessor struct - Protect enabled field access with read/write locks in IsEnabled() and SetEnabled() - Use thread-safe IsEnabled() method in ProcessPendingNotifications() - Fix concurrent access to enabled field that was causing data races This resolves the race condition between goroutines calling IsEnabled() and SetEnabled() concurrently, ensuring thread-safe access to the enabled field.

…ationName - Add protected flag to RegisterHandler methods across all types - Protected handlers cannot be unregistered, UnregisterHandler returns error - Rename 'command' parameter to 'pushNotificationName' for clarity - Update PushNotificationInfo.Command field to Name field - Add comprehensive test for protected handler functionality - Update all existing tests to use new protected parameter (false by default) - Improve error messages to use 'push notification' terminology Benefits: - Critical handlers can be protected from accidental unregistration - Clearer naming reflects that these are notification names, not commands - Better error handling with informative error messages - Backward compatible (existing handlers work with protected=false)

- Add VoidPushNotificationProcessor that reads and discards push notifications - Create PushNotificationProcessorInterface for consistent behavior - Always provide a processor (real or void) instead of nil - VoidPushNotificationProcessor properly cleans RESP3 push notifications from buffer - Remove all nil checks throughout codebase for cleaner, safer code - Update tests to expect VoidPushNotificationProcessor when disabled Benefits: - Eliminates nil pointer risks throughout the codebase - Follows null object pattern for safer operation - Properly handles RESP3 push notifications even when disabled - Consistent interface regardless of push notification settings - Cleaner code without defensive nil checks everywhere

…ethods - Remove enabled field from PushNotificationProcessor struct - Remove IsEnabled() and SetEnabled() methods from processor interface - Remove enabled parameter from NewPushNotificationProcessor() - Update all interfaces in pool package to remove IsEnabled requirement - Simplify processor logic - if processor exists, it works - VoidPushNotificationProcessor handles disabled case by discarding notifications - Update all tests to use simplified interface without enable/disable logic Benefits: - Simpler, cleaner interface with less complexity - No unnecessary state management for enabled/disabled - VoidPushNotificationProcessor pattern handles disabled case elegantly - Reduced cognitive overhead - processors just work when set - Eliminates redundant enabled checks throughout codebase - More predictable behavior - set processor = it works

- Add nil check in newConn to create VoidPushNotificationProcessor when needed - Fix tests to use Protocol 2 for disabled push notification scenarios - Prevent nil pointer dereference in transaction and connection contexts - Ensure consistent behavior across all connection creation paths The panic was occurring because newConn could create connections with nil pushProcessor when options didn't have a processor set. Now we always ensure a processor exists (real or void) to maintain the 'never nil' guarantee.

- Add push processor initialization to NewSentinelClient to prevent nil pointer dereference - Add GetPushNotificationProcessor and RegisterPushNotificationHandler methods to SentinelClient - Use VoidPushNotificationProcessor for Sentinel (typically doesn't need push notifications) - Ensure consistent behavior across all client types that inherit from baseClient This fixes the panic that was occurring in Sentinel contexts where the pushProcessor field was nil, causing segmentation violations when processing commands.

- Copy pushProcessor from parent client to transaction in newTx() - Ensure transactions inherit push notification processor from parent client - Prevent nil pointer dereference in transaction contexts (Watch, Unwatch, etc.) - Maintain consistent push notification behavior across all Redis operations This fixes the panic that was occurring in transaction examples where the transaction's baseClient had a nil pushProcessor field, causing segmentation violations during transaction operations like Watch and Unwatch.

- Add push processor initialization to NewFailoverClient to prevent nil pointer dereference - Use VoidPushNotificationProcessor for failover clients (typically don't need push notifications) - Ensure consistent behavior across all client creation paths including failover scenarios - Complete the coverage of all client types that inherit from baseClient This fixes the final nil pointer dereference that was occurring in failover client contexts where the pushProcessor field was nil, causing segmentation violations during Redis operations in sentinel-managed failover scenarios.

push_notifications.go

…lation - Add GetHandler() method to PushNotificationProcessorInterface for better encapsulation - Add GetPushNotificationHandler() convenience method to Client and SentinelClient - Remove HasHandlers() check from ProcessPendingNotifications to ensure notifications are always consumed - Use PushNotificationProcessorInterface in internal pool package for proper abstraction - Maintain GetRegistry() for backward compatibility and testing - Update pubsub to use GetHandler() instead of GetRegistry() for cleaner code Benefits: - Better API encapsulation - no need to expose entire registry - Cleaner interface - direct access to specific handlers - Always consume push notifications from reader regardless of handler presence - Proper abstraction in internal pool package - Backward compatibility maintained - Consistent behavior across all processor types

… FailoverClient - Add PushNotifications field to FailoverOptions struct - Update clientOptions() to pass PushNotifications field to Options - Change SentinelClient and FailoverClient initialization to use same logic as regular Client - Both clients now support real push notification processors when enabled - Both clients use void processors only when explicitly disabled - Consistent behavior across all client types (Client, SentinelClient, FailoverClient) Benefits: - SentinelClient and FailoverClient can now fully utilize push notifications - Consistent API across all client types - Real processors when push notifications are enabled - Void processors only when explicitly disabled - Equal push notification capabilities for all Redis client types

…better encapsulation - Remove GetRegistry() method from PushNotificationProcessorInterface - Enforce use of GetHandler() method for cleaner API design - Add GetRegistryForTesting() method for test access only - Update all tests to use new testing helper methods - Maintain clean separation between public API and internal implementation Benefits: - Better encapsulation - no direct registry access from public interface - Cleaner API - forces use of GetHandler() for specific handler access - Consistent interface design across all processor types - Internal registry access only available for testing purposes - Prevents misuse of registry in production code

…anic - Add nil check for proto.Reader parameter in both PushNotificationProcessor and VoidPushNotificationProcessor - Prevent segmentation violation when ProcessPendingNotifications is called with nil reader - Return early with nil error when reader is nil (graceful handling) - Fix panic in TestProcessPendingNotificationsEdgeCases test This addresses the runtime panic that occurred when rd.Buffered() was called on a nil reader, ensuring robust error handling in edge cases where the reader might not be properly initialized.

- Remove unnecessary handlerWrapper complexity from push notifications - Use separate maps for handlers and protection status in registry - Store handlers directly without indirection layer - Maintain same instance identity for registered/retrieved handlers - Preserve all protected handler functionality with cleaner implementation Changes: - internal/pushnotif/registry.go: Use separate handlers and protected maps - push_notifications.go: Remove handlerWrapper, store handlers directly - Maintain thread-safe operations with simplified code structure Benefits: - Reduced memory overhead (no wrapper objects) - Direct handler storage without type conversion - Cleaner, more maintainable code - Same functionality with better performance - Eliminated unnecessary complexity layer - Preserved all existing behavior and safety guarantees

- TestClientWithoutPushNotifications: Now expects error instead of nil - TestClientPushNotificationEdgeCases: Now expects error instead of nil

…tions - Fix TestConnWithoutPushNotifications to expect errors instead of nil - Update test to verify error messages contain helpful information - Add strings import for error message validation - Maintain consistency with improved developer experience approach The test now correctly expects errors when trying to register handlers on connections with disabled push notifications, providing immediate feedback to developers about configuration issues rather than silent failures. This aligns with the improved developer experience where VoidProcessor returns descriptive errors instead of silently ignoring registrations.

Optimize the peeking on newly acquired connection on *unix. Use syscall to peek on the socket instead of blocking for a fixed amount of time. This won't work on Windows, hence the `MaybeHasData` will always return true on Windows and the client will have to block for a given time to actually peek on the socket. *Time to complete N HSET operations (individual commands)* | Batch Size | Before (total sec) | After (total sec) | Time Saved | % Faster | |------------|-------------------|------------------|------------|----------| | 100 ops | 0.0172 | 0.0133 | 0.0038 | **22.4%** | | 1K ops | 0.178 | 0.133 | 0.045 | **25.3%** | | 10K ops | 1.72 | 1.28 | 0.44 | **25.6%** | | 100K ops | 17.1 | 13.4 | 3.7 | **22.0%** |

internal/pool/conn.go

htemelski-redis

Looks good, couldn't find any glaring issues
One minor nitpick

push/errors.go

…sp3-notification-handlers

* feat(hitless): Introduce handlers for hitless upgrades This commit includes all the work on hitless upgrades with the addition of: - Pubsub Pool - Examples - Refactor of push - Refactor of pool (using atomics for most things) - Introducing of hooks in pool --------- Co-authored-by: Copilot <[email protected]>

Copilot

Pull Request Overview

This PR implements RESP3 push notifications support and hitless upgrade handling for the Redis client. It adds infrastructure for processing push notifications from Redis and manages connection state transitions during server upgrades while maintaining API compatibility.

Adds RESP3 push notification processing system with configurable handlers
Implements hitless upgrade support for maintaining connections during server maintenance
Extends pool and PubSub functionality to handle connection handoffs and notifications

Reviewed Changes

Copilot reviewed 67 out of 69 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
universal.go	Adds HitlessUpgradeConfig support to universal options
tx.go	Implements connection cloning and push processor sharing for transactions
sentinel.go	Adds push notification support and pool management for sentinel clients
redis_test.go	Removes extraneous blank line
redis.go	Core implementation of push notification processing and hitless upgrade management
push_notifications.go	Factory functions for creating push notification processors
push/registry.go	Registry for managing push notification handlers
push/push_test.go	Comprehensive test suite for push notification system
push/push.go	Package documentation for push notification API
push/processor_unit_test.go	Unit tests for notification processors
push/processor.go	Core push notification processor implementations
push/handler_context.go	Context structure for notification handlers
push/handler.go	Interface definition for notification handlers
push/errors.go	Error types and helper functions for push notifications
pubsub.go	Updates PubSub to handle new connection patterns and push notifications
pool_pubsub_bench_test.go	Performance benchmarks for pool and PubSub operations
osscluster.go	Cluster client integration with push notifications and hitless upgrades
options.go	Configuration options for hitless upgrades and push notifications
main_test.go	Test setup with logging configuration
logging/logging_test.go	Tests for logging level functionality
logging/logging.go	Logging utilities and level management
internal_test.go	Test context variable addition
internal/util/math.go	Math utility functions
internal/util/convert.go	Safe integer conversion functions
internal/redis.go	Redis null constant definition
internal/proto/reader.go	RESP3 push notification parsing functionality

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

redis.go

push/push_test.go

redis.go

Copilot · 2025-09-03T17:46:20Z

options.go

@@ -253,7 +285,13 @@ func (opt *Options) init() {
 		opt.Protocol = 3
 	}
 	if opt.DialTimeout == 0 {


The default dial timeout has been changed from 5 seconds to 10 seconds. This change affects connection establishment performance and should be documented as a breaking change.

We should return the dial timeout to the previous value, I agree.

sentinel.go

push/processor.go

ndyakov requested a review from Copilot June 26, 2025 17:44

This comment was marked as outdated.

Sign in to view

ndyakov added 13 commits June 27, 2025 01:36

remove: push-notification-demo

79f6df2

ndyakov force-pushed the ndyakov/CAE-1088-resp3-notification-handlers branch from 8bb0e59 to 8006fab Compare June 26, 2025 22:36

ndyakov added 3 commits June 27, 2025 01:47

ndyakov commented Jun 27, 2025

View reviewed changes

push_notifications.go Outdated Show resolved Hide resolved

ndyakov added 4 commits June 27, 2025 13:59

ndyakov force-pushed the ndyakov/CAE-1088-resp3-notification-handlers branch from 4e35707 to 9a7a5c8 Compare June 27, 2025 12:05

ndyakov added 4 commits June 27, 2025 16:27

refactor: move push notification logic to pusnotif package

ada72ce

Fixes tests:

e31987f

- TestClientWithoutPushNotifications: Now expects error instead of nil - TestClientPushNotificationEdgeCases: Now expects error instead of nil

ndyakov marked this pull request as ready for review June 27, 2025 14:49

Merge branch 'master' into ndyakov/CAE-1088-resp3-notification-handlers

72c24d5

ndyakov dismissed htemelski-redis’s stale review via 72c24d5 July 22, 2025 11:07

ndyakov and others added 2 commits July 24, 2025 11:48

Merge branch 'master' into ndyakov/CAE-1088-resp3-notification-handlers

5d1d063

ndyakov commented Jul 24, 2025

View reviewed changes

internal/pool/conn.go Outdated Show resolved Hide resolved

ndyakov commented Jul 24, 2025

View reviewed changes

internal/pool/conn.go Outdated Show resolved Hide resolved

add comments

25bde87

ofekshenawa previously approved these changes Jul 30, 2025

View reviewed changes

ndyakov added 2 commits August 1, 2025 15:42

Merge branch 'master' into ndyakov/CAE-1088-resp3-notification-handlers

92b2861

Merge branch 'master' into ndyakov/CAE-1088-resp3-notification-handlers

5767a31

ndyakov dismissed ofekshenawa’s stale review via 5767a31 August 4, 2025 07:56

Merge branch 'master' into ndyakov/CAE-1088-resp3-notification-handlers

c7ffeff

htemelski-redis previously approved these changes Aug 4, 2025

View reviewed changes

push/errors.go Outdated Show resolved Hide resolved

chore(pr): address pr comments

4dc03fc

ndyakov dismissed htemelski-redis’s stale review via 4dc03fc August 4, 2025 12:33

chore(log): add note related to logging and simple void logger

b62fe92

htemelski-redis previously approved these changes Aug 4, 2025

View reviewed changes

Merge remote-tracking branch 'origin/master' into ndyakov/CAE-1088-re…

422779b

…sp3-notification-handlers

ndyakov dismissed htemelski-redis’s stale review via 422779b August 12, 2025 16:38

Merge branch 'master' into ndyakov/CAE-1088-resp3-notification-handlers

36f9f58

htemelski-redis previously approved these changes Aug 27, 2025

View reviewed changes

ndyakov changed the title ~~[CAE-1088] feat: RESP3 notifications support~~ [CAE-1088] & [CAE-1072] feat: RESP3 notifications support & Hitless notifications handling Sep 3, 2025

ndyakov dismissed htemelski-redis’s stale review via cb3af08 September 3, 2025 11:49

ndyakov and others added 2 commits September 3, 2025 14:59

Merge branch 'master' into ndyakov/CAE-1088-resp3-notification-handlers

f380b4f

handle panic in background workers

8aecdb8

ndyakov requested a review from Copilot September 3, 2025 17:43

Copilot AI reviewed Sep 3, 2025

View reviewed changes

ndyakov mentioned this pull request Sep 4, 2025

Feat/improve success rate of new connections #3502

Open

DialTimeout defaults back to 5 seconds

0196b9a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CAE-1088] & [CAE-1072] feat: RESP3 notifications support & Hitless notifications handling #3418

[CAE-1088] & [CAE-1072] feat: RESP3 notifications support & Hitless notifications handling #3418

Uh oh!

ndyakov commented Jun 26, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htemelski-redis left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Sep 3, 2025

Uh oh!

ndyakov Sep 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[CAE-1088] & [CAE-1072] feat: RESP3 notifications support & Hitless notifications handling #3418

Are you sure you want to change the base?

[CAE-1088] & [CAE-1072] feat: RESP3 notifications support & Hitless notifications handling #3418

Uh oh!

Conversation

ndyakov commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htemelski-redis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

ndyakov Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ndyakov commented Jun 26, 2025 •

edited

Loading