Perf: Eliminate Random CAS contention in TransactionBroadcaster

[rakshaak29]

Steps to Reproduce

1. Start a Besu node on a highly active network or under a load testing environment.
2. Subject the node to a high-volume load test of inbound transactions via multiple concurrent RPC clients or extensive P2P gossip.
3. Profile the node's CPU usage; specifically look for thread contention inside java.util.Random.next() invoked from TransactionBroadcaster.onTransactionsAdded().

Expected behavior:
Peer list shuffling inside the TransactionBroadcaster should be non-blocking and use thread-local randomness under high concurrency to avoid Compare-And-Swap (CAS) contention overhead.

Actual behavior:
TransactionBroadcaster.java currently uses a single, shared instance of java.util.Random across all threads to shuffle the peer list (Collections.shuffle(peers, random)). java.util.Random relies on an internal AtomicLong and a CAS loop to update the seed. In a highly concurrent environment handling thousands of incoming transactions, multiple threads hit Collections.shuffle simultaneously and contend for this same CAS loop, leading to heavy CPU overhead and degraded transaction broadcasting throughput.

Frequency:
100% of the time under high transaction concurrency.

Logs

(No specific logs are printed for this issue as it is a CPU/concurrency bottleneck visible only through a JVM profiler like async-profiler or JFR.)

Versions

• Software version: main branch (latest)
• Java version: openjdk version "21.0.11"
• OS Name & Version: N/A (Cross-platform JVM concurrency issue)
• Kernel Version: N/A
• Virtual Machine software & version: N/A
• Docker Version: N/A
• Cloud VM, type, size: N/A
• Consensus Client & Version if using Proof of Stake: N/A

Smart contract information (If you're reporting an issue arising from deploying or calling a smart contract, please supply related information)

N/A

Additional Information

Suggested Fix:
Modify the code to use Collections.shuffle(peers) (without explicitly passing the random instance). Java inherently delegates parameter-less shuffling to ThreadLocalRandom.current(), bypassing the CAS bottleneck entirely. The shared Random instance should only be injected and used when a deterministic seed is strictly necessary (e.g., in tests).

[rakshaak29]

Hello @siladu and @macfarla ,
I would like to work on this issue, it would be great if you assign this to me.

[macfarla]

assigned to you @rakshaak29

I removed the bug label because this would be more an improvement / better-practice issue

• onTransactionsAdded receives a batch of transactions — the shuffle happens once per batch, not once per transaction
• This callback almost certainly runs on a single scheduler thread (the EthContext scheduler), not from hundreds of concurrent threads simultaneously
• The CAS contention in Random only becomes meaningful when dozens or hundreds of threads hit it concurrently — which isn't this usage pattern