Finalizing blocks appears to be slow when checkpointing large blocks

[jvff]

Motivation

The block synchronizer sometimes restarts when it all of the fanned out extend tip requests time out. When that happens, it can take a few minutes (5-15 minutes) for new blocks to start being downloaded because the request to obtain the block locator from takes a while to be processed by the state service. This happens because the state is busy committing checkpointed finalized blocks.

It would be nice to see if this can be improved to improve the synchronization performance.

Related Work

PRs:

• temporary fix: fix(network): increase state concurrency and syncer lookahead #3455

Tickets:

• Fix "synced block height too far ahead of the tip: dropped downloaded block" #3603
• Fix slowness when syncing near the tip #3375
• Zebra validation can be very slow on macOS, and cause very slow syncs #3042

[teor2345]

If the state is busy committing blocks, then we should speed up the state commit code.

(If the syncer pauses while this happens, this is correct behaviour, and it shows that our backpressure is actually working.)

[ftm1000]

Hey team! Please add your planning poker estimate with ZenHub @conradoplg @dconnolly @gustavovalverde @jvff @mpguerra @oxarbitrage @upbqdn

[ftm1000]

Please add your planning poker estimate with ZenHub @teor2345

[teor2345]

This isn't causing us any issues right now.