How do we want to address "light" blocks after epoch crossing?

[bw-solana]

The first several blocks after an epoch crossing tend to be very lightly filled with transactions (both vote and non-vote). For example, see https://explorer.solana.com/block/327024000

A large part of this is the extra work that happens when creating the first descendant bank crossing into a new epoch: https://github.com/anza-xyz/agave/blob/master/runtime/src/bank.rs#L1415-L1429

This process takes ~2.5 seconds, with variability across nodes. I suspect this leads to things like:

1. Spending so much time processing epoch that we tick through the first slot
2. We are forking and spending so many resources on everything else going on during that time that leader just can't pack very well.
3. We are forking, so leader targeting is messed up.
4. We process epoch boundary multiple times because the forking causes multiple leaders to try and build off the last slot in the previous epoch. (e.g. we can see 327024000 and 327024004 both tried to build off 327023999 and would have triggered the calculation)

Some ideas that could be considered:

1. Re-using the rewards calculation from the first time we computed it
2. Partitioning the rewards calculation to split up the 2.5s time (spread the pain across multiple slots).
3. Delaying the calculation a few slots because boundary crossing involves other work as well.
4. Reduce complexity of the calculation (see below)

From @jstarry:

we already know in epoch stakes how much stake each vote has and how many credits it earned. Calculate the rewards "points" from those values, calculate commission, and update the 1500 vote accounts and be done (distribution stays the same). Cut out a few things like:

1. calculating the commission split individually for each stake account rather than once per vote account
2. oh you didn't get a fractional lamport reward for your micro-stake? let's hold until next epoch boundary and we will try to reward you on the past two epochs instead. dumb.

It will require a feature gate but it's much simpler and faster and doesn't require any iteration over all the stake accounts. We will still need to hash the active stake accounts into distribution buckets but that can be done in advance.
We iterate over all stake accounts at least 4 times in the calculation step right now.
and we iterate over all stake accounts twice for the epoch stakes update too

[bw-solana]

Some example data to highlight the problem:

slot	parent slot	non-vote tx	vote tx	leader	notes
327023999	327023998	376	1.3k	3fpcsJ2WPdrmpjVVA5Lmv4m9Qx5xqND1xJhFprDSjWJ4	Last slot of epoch 756
327024000	327023999	0	0	BUv44cVtsdvU9z2BfFGk6s5JZZWrmVnq5qCaii5ARyyB	First slot of epoch 757
327024001	327024000	31	3	BUv44cVtsdvU9z2BfFGk6s5JZZWrmVnq5qCaii5ARyyB
327024002	327024001	44	4	BUv44cVtsdvU9z2BfFGk6s5JZZWrmVnq5qCaii5ARyyB
327024003	327024002	74	5	BUv44cVtsdvU9z2BfFGk6s5JZZWrmVnq5qCaii5ARyyB
327024004	327023999	N/A	N/A	HEL1USMZKAL2odpNBj2oCjffnFGaYwmbGmyewGv1e2TU	Trying to skip 327024000, necessitates processing epoch crossing again. Got skipped
327024005	N/A	N/A	N/A	HEL1USMZKAL2odpNBj2oCjffnFGaYwmbGmyewGv1e2TU	Skipped. Doesn't appear we attempted to build
327024006	327024003	510	~1k	HEL1USMZKAL2odpNBj2oCjffnFGaYwmbGmyewGv1e2TU
327024007	327024006	586	~1.2k	HEL1USMZKAL2odpNBj2oCjffnFGaYwmbGmyewGv1e2TU
327024008	327024007	N/A	N/A	GSTampk6BJRKSDkzhaMM49R7qRx98MTPYYWvKbp83XKc	Skipped because 2GUnfxZavKoPfS9s3VSEjaWDzB3vNf5RojUhprCS1rSx got 327024012 out faster
327024009	N/A	N/A	N/A	GSTampk6BJRKSDkzhaMM49R7qRx98MTPYYWvKbp83XKc	Didn't attempt to build because cluster moved onto 327024012
327024010	N/A	N/A	N/A	GSTampk6BJRKSDkzhaMM49R7qRx98MTPYYWvKbp83XKc	Didn't attempt to build because cluster moved onto 327024012
327024011	N/A	N/A	N/A	GSTampk6BJRKSDkzhaMM49R7qRx98MTPYYWvKbp83XKc	Didn't attempt to build because cluster moved onto 327024012
327024012	327024007	50	~1.2k	2GUnfxZavKoPfS9s3VSEjaWDzB3vNf5RojUhprCS1rSx	This node was faster and was able to tick through previous leader and publish before them

[t-nelson]

did 004 fully broadcast or was it aborted? that we didn't attempt 005 is interesting. everything else seems to follow

[bw-solana]

Yes, looks like 004 was fully broadcast and replayed.