add swip-25: pullsync protocol improvement #66

zelig · 2025-02-24T11:34:58Z

Submitted a preliminary SWIP for pullsync changes

istae · 2025-02-25T13:00:35Z

Adding more context to this; when a peer pullsync with another peer, it will first request chunk addresses from a particular bin and will only then request the actual chunk data for the chunks it does not have locally. This is done for every bin >= storage depth and with every neighbor peer, and this is where the inefficiency comes in. One might argue that because it's only the chunk address request that is replicated by a factor of the size of neighborhood, the inefficiency is somewhat tolerable.

Another detail is that only reachable nodes will store the chunk and terminate a pushsync request. Then all peers in the neighborhood, reachable or not, will pullsync the chunk. I wonder how this will play out with the more efficient syncing strategy proposed by the SWIP? I guess since the new strategy only reduces the replicated requests of the same chunk, and not favor efficiency over replication reliability or something like that, it should be ok.

nugaon · 2025-02-26T15:01:52Z

SWIPs/swip-pullsync.md

+
+## Specification
+<!--The technical specification should describe the syntax and semantics of any new feature. The specification should be detailed enough to allow competing, interoperable implementations for the current Swarm platform and future client implementations.-->
+Each peer takes all their neighbours they are allowed to synchronise with (have full node ambitions): p_0, p_1, ..., p_n. For each peer, they decide their uniquness depth, i.e., the PO, within which they are the only peer in the set: `UD_i, UD_1, ... UD_n`. Now for each peer `p_i` we start subscribing to all POs greater or equal to `UD_i`. Note that unlike the earlier algorithm, this one is extremely sensitive to the changing peerset, so every single time there is a change in the neighbours, pullsync stretegy needs to be reevaluated. In addition to `po>=UD_i`, our pivot peer needs to sync the PO corresponding to their PO with the peer in order to get all the chunks that they are closer to than their peer. To sum up, for any pivot peer P:


for me it is a bit unclear by what rule the peers get their uniqueness depth. maybe, some examples could resolve my issue but from the text it is a bit hard for me to catch it.

"uniquness depth, i.e., the PO, within which they are the only peer in the set"

the depth of your exclusive neighbourhood

zelig · 2025-03-08T19:55:20Z

SWIPs/swip-pullsync.md

+
+## Specification
+<!--The technical specification should describe the syntax and semantics of any new feature. The specification should be detailed enough to allow competing, interoperable implementations for the current Swarm platform and future client implementations.-->
+Each peer takes all their neighbours they are allowed to synchronise with (have full node ambitions): p_0, p_1, ..., p_n. For each peer, they decide their uniquness depth, i.e., the PO, within which they are the only peer in the set: `UD_i, UD_1, ... UD_n`. Now for each peer `p_i` we start subscribing to all POs greater or equal to `UD_i`. Note that unlike the earlier algorithm, this one is extremely sensitive to the changing peerset, so every single time there is a change in the neighbours, pullsync stretegy needs to be reevaluated. In addition to `po>=UD_i`, our pivot peer needs to sync the PO corresponding to their PO with the peer in order to get all the chunks that they are closer to than their peer. To sum up, for any pivot peer P:


"uniquness depth, i.e., the PO, within which they are the only peer in the set"

the depth of your exclusive neighbourhood

zelig · 2025-03-08T19:56:47Z

SWIPs/swip-pullsync.md

+
+## Test Cases
+<!--Test cases for an implementation are mandatory for SWIPs that are affecting changes to data and message formats. Other SWIPs can choose to include links to test cases if applicable.-->
+Thorough testing is neeeded, cos this can produce inconsistencies in the localstore and has major impact for retrievebility.


I don't get this @nugaon

significance · 2025-04-14T11:25:18Z

i think it is important that we thoroughly consider and explore all possible edge cases especially during network merges and splits and also when this is concurrent with the various configurations of unbalanced neighbourhoods

nugaon · 2025-06-11T09:37:12Z

#66 (comment)

I just removed original comment but it did not close the whole tab... (I mistakenly proposed the current workflow to ensure certainty in reserve sync.)

The rationale could include explanation how this pull sync strategy syncs the whole reserve.

You detailed that how respective bin X of peers are distinct from each other, but the subscriptions for those bins do not change from the current workflow. The proposal states that it is enough to additionally sync the PO(p, P) bin at each p peer to keep the full reserve but that is not elaborated why it is enough.

nugaon · 2025-06-11T09:43:19Z

SWIPs/swip-pullsync.md

+<!--The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale may also provide evidence of consensus within the community, and should discuss important objections or concerns raised during discussion.-->
+
+One can see that each chunk is taken from its most immediate neighbourhood only. So depending on to what extent the peer addresses are balanced we save a lot on not taking anything twice. Imagine a peer with neighbourhood depth `d`, and in the hood 3 neighbours each having a different 2 bit prefix within the neighbourhood. Then `UD_i=d+3` for each peer, so we synchronise PO=d+3,d+4,d+5,etc. from each peer.
+this is exactly 16 times less chunks than what we need to syncronise with the current process. Also we need to synchronise PO=d+2 chunks from each peer. 


how is it 16 times less and is it together with d+2 syncs?

which node will synchronize bin 0 if storageRadius is 0 and the neighbor nodes are 1111, 1100, 1000 and the pivot node is 1101.

nugaon · 2025-06-11T09:50:18Z

SWIPs/swip-pullsync.md

+One can see that each chunk is taken from its most immediate neighbourhood only. So depending on to what extent the peer addresses are balanced we save a lot on not taking anything twice. Imagine a peer with neighbourhood depth `d`, and in the hood 3 neighbours each having a different 2 bit prefix within the neighbourhood. Then `UD_i=d+3` for each peer, so we synchronise PO=d+3,d+4,d+5,etc. from each peer.
+this is exactly 16 times less chunks than what we need to syncronise with the current process. Also we need to synchronise PO=d+2 chunks from each peer. 
+
+One potential caveat is that if a peer quits or is no longer contactable before the pivot finished syncing with them, then another peer needs to start the process.


it needs recalculating all UDs and maybe add or drop bin subscriptions at each peer in my understanding.
It is a bit vague what process the another peer should start.

what I meant is that then we would need to start the same sync process with another peer and that will be from the start if the peer is new.

* bin sync with binary tree * compactible node sync * comments

Refine glossary terms related to pull-sync protocol and neighbourhood depth for clarity and precision.

Copilot

Pull Request Overview

This PR introduces SWIP-25, proposing a more efficient pullsync protocol for synchronizing content between peers in the same neighborhood. The improvement aims to reduce redundant chunk exchanges by having each chunk synchronized from only its closest peer.

Key Changes:

Introduces a new pullsync strategy using a leaf-compacted binary tree to identify which peer should sync each chunk
Reduces synchronization overhead from N*S chunk hashes (where N is neighborhood size and S is reserve size) by eliminating redundant exchanges
Maintains backward compatibility by keeping the subscription request wire protocol unchanged

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-27T05:50:37Z

SWIPs/swip-pullsync.md

+## Motivation
+<!--The motivation is critical for SWIPs that want to change the Swarm protocol. It should clearly explain why the existing protocol specification is inadequate to address the problem that the SWIP solves. SWIP submissions without sufficient motivation may be rejected outright.-->
+Imagine, that a naive peer joins a neighbourhood, then they will 'subscribe to' each 
+depth of their peers within the neighbourhood. As they are receiving new chunks of course these are offering it too back to the peer they got it from. Plus they try to synchronise from each peer the entire reserve, not just part, which means a naive node's synchronisation involves exchange of `N*S` chunk hashrd where N is the neighbourhood size and S is the size of the reserve. This is hugely inefficient. 


Corrected spelling of 'hashrd' to 'hashes'.

Suggested change

depth of their peers within the neighbourhood. As they are receiving new chunks of course these are offering it too back to the peer they got it from. Plus they try to synchronise from each peer the entire reserve, not just part, which means a naive node's synchronisation involves exchange of `N*S` chunk hashrd where N is the neighbourhood size and S is the size of the reserve. This is hugely inefficient.

depth of their peers within the neighbourhood. As they are receiving new chunks of course these are offering it too back to the peer they got it from. Plus they try to synchronise from each peer the entire reserve, not just part, which means a naive node's synchronisation involves exchange of `N*S` chunk hashes where N is the neighbourhood size and S is the size of the reserve. This is hugely inefficient.

Copilot · 2025-10-27T05:50:37Z

SWIPs/swip-pullsync.md

+All chunks need to be syncronized only once.
+How about we syncronize each chunks from its closest peer among the neighborhood peers.


Corrected spelling of 'syncronized' to 'synchronized' and 'syncronize' to 'synchronize'.

Suggested change

All chunks need to be syncronized only once.

How about we syncronize each chunks from its closest peer among the neighborhood peers.

All chunks need to be synchronized only once.

How about we synchronize each chunk from its closest peer among the neighborhood peers.

Copilot · 2025-10-27T05:50:37Z

SWIPs/swip-pullsync.md

+
+If all the peers we synced from are finished, the respective nodes reserve for any depth equal or higher to storage radius will be the same. 
+
+Unlike the earlier algorithm, this one is extremely sensitive to the changing peerset, so every single time there is a change in the neighbours, pullsync stretegy needs to be reevaluated. 


Corrected spelling of 'stretegy' to 'strategy'.

Suggested change

Unlike the earlier algorithm, this one is extremely sensitive to the changing peerset, so every single time there is a change in the neighbours, pullsync stretegy needs to be reevaluated.

Unlike the earlier algorithm, this one is extremely sensitive to the changing peerset, so every single time there is a change in the neighbours, pullsync strategy needs to be reevaluated.

Copilot · 2025-10-27T05:50:37Z

SWIPs/swip-pullsync.md

+
+## Test Cases
+<!--Test cases for an implementation are mandatory for SWIPs that are affecting changes to data and message formats. Other SWIPs can choose to include links to test cases if applicable.-->
+Thorough testing is neeeded, cos this can produce inconsistencies in the localstore and has major impact for retrievebility.


Corrected spelling of 'neeeded' to 'needed' and 'retrievebility' to 'retrievability'. Additionally, 'cos' should be 'because' in formal documentation.

Suggested change

Thorough testing is neeeded, cos this can produce inconsistencies in the localstore and has major impact for retrievebility.

Thorough testing is needed, because this can produce inconsistencies in the localstore and has major impact for retrievability.

Copilot · 2025-10-27T05:50:38Z

SWIPs/swip-pullsync.md

+Each compactible node (i.e. that has one child) is the indication that all the chunks on the missing branch has no single closest peer and are equidistant from two or more peers on the existing branch.
+
+Ideally To sync all the chunks we need to cover all the branches of the trie:
+- all chunks of leaf nodes must be syncronized from its stored peer.


Corrected spelling of 'syncronized' to 'synchronized'.

Suggested change

- all chunks of leaf nodes must be syncronized from its stored peer.

- all chunks of leaf nodes must be synchronized from its stored peer.

Copilot · 2025-10-27T05:50:38Z

SWIPs/swip-pullsync.md

+- as secondary ordering within a bin is based on first time of storage.
+- the chronology makes it possible to have live (during session) and historical syncing.
+
+## Copyright/


Remove trailing slash from 'Copyright/' header.

Suggested change

## Copyright/

## Copyright

add swip pullsync

168b3da

zelig added the protocol describes a process every swarm node must implement and adhere to label Feb 24, 2025

zelig self-assigned this Feb 24, 2025

zelig requested a review from istae February 24, 2025 11:36

zelig changed the title ~~add swip pullsync protocol improvement~~ add swip-25: pullsync protocol improvement Feb 24, 2025

nugaon reviewed Feb 26, 2025

View reviewed changes

zelig commented Mar 8, 2025

View reviewed changes

nugaon reviewed Jun 11, 2025

View reviewed changes

update: swip-25 (#78)

9f23cdf

* bin sync with binary tree * compactible node sync * comments

nugaon mentioned this pull request Sep 8, 2025

feat(swip25): pull syncing optimization ethersphere/bee#5194

Open

4 tasks

Update glossary for pull-sync protocol in SWIP

c05e338

Refine glossary terms related to pull-sync protocol and neighbourhood depth for clarity and precision.

Copilot AI review requested due to automatic review settings October 27, 2025 05:49

Copilot AI reviewed Oct 27, 2025

View reviewed changes

		All chunks need to be syncronized only once.
		How about we syncronize each chunks from its closest peer among the neighborhood peers.


		If all the peers we synced from are finished, the respective nodes reserve for any depth equal or higher to storage radius will be the same.

		Unlike the earlier algorithm, this one is extremely sensitive to the changing peerset, so every single time there is a change in the neighbours, pullsync stretegy needs to be reevaluated.

	Thorough testing is neeeded, cos this can produce inconsistencies in the localstore and has major impact for retrievebility.
	Thorough testing is needed, because this can produce inconsistencies in the localstore and has major impact for retrievability.

	- all chunks of leaf nodes must be syncronized from its stored peer.
	- all chunks of leaf nodes must be synchronized from its stored peer.

Uh oh!

add swip-25: pullsync protocol improvement #66

Are you sure you want to change the base?

add swip-25: pullsync protocol improvement #66

Uh oh!

Conversation

zelig commented Feb 24, 2025

Uh oh!

istae commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

significance commented Apr 14, 2025

Uh oh!

nugaon commented Jun 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

istae commented Feb 25, 2025 •

edited

Loading