Self rate limiting due to number of concurrent requests for `data_column_sidecars_by_range`

[jimmygchen]

When trying to sync on a PeerDAS-enabled devnet, we see a large number of self rate limiting logs. Range sync only sends up to 5 batches concurrently, but this changes with PeerDAS, as we may send more than 1 data column requests per batch - so might be worth investigating and see if the current limit makes sense for data column requests, as this will slow down sync:

Jul 05 18:08:02.446 DEBUG Self rate limiting due to the number of concurrent requests  peer_id: 16Uiu2HAmACmiJYvqiMWDiVN8tbwTpHqFWxj3E9bPvnRq1jeokGhH, protocol: data_column_sidecars_by_range, service: "network"
Jul 05 18:08:02.549 DEBUG Self rate limiting due to the number of concurrent requests  peer_id: 16Uiu2HAmACmiJYvqiMWDiVN8tbwTpHqFWxj3E9bPvnRq1jeokGhH, protocol: data_column_sidecars_by_range, service: "network"
Jul 05 18:08:02.651 DEBUG Self rate limiting due to the number of concurrent requests  peer_id: 16Uiu2HAmACmiJYvqiMWDiVN8tbwTpHqFWxj3E9bPvnRq1jeokGhH, protocol: data_column_sidecars_by_range, service: "network"
Jul 05 18:08:02.753 DEBUG Self rate limiting due to the number of concurrent requests  peer_id: 16Uiu2HAmACmiJYvqiMWDiVN8tbwTpHqFWxj3E9bPvnRq1jeokGhH, protocol: data_column_sidecars_by_range, service: "network"

[jimmygchen]

@pawanjay176 wonder if you are aware of this - given you were able to sync with up to 20 slots /s on your branch?

[dapplion]

We could address this by prioritizing peers with no active requests on the same protocol. With a similar function to this one but only for that specific topic

lighthouse/beacon_node/network/src/sync/network_context.rs

Line 406 in b9c1a2b

fn active_request_count_by_peer(&self) -> HashMap<PeerId, usize> {