Optimize Stream Sync Peer Discovery: Reduce Startup Time to Seconds#4937
Draft
GheisMohammadi wants to merge 4 commits intodevfrom
Draft
Optimize Stream Sync Peer Discovery: Reduce Startup Time to Seconds#4937GheisMohammadi wants to merge 4 commits intodevfrom
GheisMohammadi wants to merge 4 commits intodevfrom
Conversation
GheisMohammadi
changed the title
- Add SetEnoughStreamsCallback method to Operator interface - Implement SetEnoughStreamsCallback in streamManager struct - Add enoughStreamsCallback field to streamManager - Call callback when softHaveEnoughStreams() is true - Enables early exit from startup mode when enough streams are found
- Add startup mode with faster timings for initial peer discovery - Centralize all timing constants in const.go - Add network-specific DHT request limits and target valid peer counts - Align target valid peers with stream sync configuration requirements - Improve adaptive sleep logic with SleepDecreaseRatio - Add comprehensive timing constants for startup vs normal modes
- Add startup mode with faster timings for initial peer discovery - Implement early exit from startup mode when enough streams found - Add network-specific peer discovery limits and validation - Improve advertise() function with better error handling and logging - Add isValidPeer() stub for future peer validation logic - Use configurable DHT request limits instead of hardcoded values - Add target valid peer counting and filtering - Improve advertiseLoop() with adaptive timing based on discovery success
- Add TestProtocol_Advertise with mock discovery and proper setup - Add TestProtocol_AdvertiseLoop with timing validation - Add TestProtocol_ExitStartupMode for startup mode functionality - Add TestProtocol_GetDHTRequestLimit for network-specific limits - Add TestProtocol_GetTargetValidPeers for target peer validation - Fix interface compilation errors by adding SetEnoughStreamsCallback stubs - Add mockDiscovery struct to properly mock discovery interface - Update tests to handle nil pointer scenarios and proper initialization
GheisMohammadi
force-pushed
the
improve/discovery
branch
from
38a7674 to
2e8d474
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR addresses the critical issue where stream sync takes ~15 minutes to find enough peers on node startup, while restarting the service allows much faster peer discovery. The solution implements a comprehensive peer discovery optimization strategy with startup mode, network-aware limits, and improved timing algorithms.
Problem Statement
Issue: Stream sync on node startup takes approximately 15 minutes to find minimum streams and begin syncing, while restarting the service enables much faster peer discovery.
Root Causes:
Solution Overview
1. Startup Mode Implementation
2. Network-Aware Peer Discovery
3. Improved Timing Algorithms
const.go/= 2Configuration Alignment