Use the events api to attest on a head event

[pawanjay176]

Issue

Currently, we always attest to the slot by requesting the attestation_data 4 seconds into the slot. See

lighthouse/validator_client/validator_services/src/attestation_service.rs

Lines 163 to 169 in 134039d

	sleep(duration_to_next_slot + slot_duration / 3).await;
	if let Err(e) = self.spawn_attestation_tasks(slot_duration) {
	crit!(error = e, "Failed to spawn attestation tasks")
	} else {
	trace!("Spawned attestation tasks");
	}

This isn't strictly following the spec as it mentions here https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/validator.md#attesting

A validator should create and broadcast the attestation to the associated attestation subnet when either (a) the validator has received a valid block from the expected block proposer for the assigned slot or (b) 1 / INTERVALS_PER_SLOT of the slot has transpired (SECONDS_PER_SLOT / INTERVALS_PER_SLOT seconds after the start of slot) -- whichever comes first.

We skip (a) here and always attest 4 seconds into the slot.

We should change this to become spec compliant. This also has benefits of spreading the attestation load throughout the slot instead of bursty traffic around 4seconds.

Proposed solution

We currently have the sse api on the http api which emits a head event.

Instead of waiting 4 seconds into the slot, we should essentially do something like this in the attestation service here https://github.com/sigp/lighthouse/blob/unstable/validator_client/validator_services/src/attestation_service.rs#L163

tokio::select!{
    _ => sleep(Duration::from_secs(4)),
    _ => poll_head_event_on_all_beacon_nodes(&self.beacon_nodes),
)

The poll_head_event_on_all_beacon_nodes would be the meat of the change imo. The function could use select_all
to wait on the results of the event stream from all beacon nodes in https://github.com/sigp/lighthouse/blob/unstable/validator_client/validator_services/src/attestation_service.rs#L108 and return the first returned head event.

Testing

Best way to test this imo would be to run a kurtosis network and check that the attestations arrive at gossip earlier than 4 seconds into the slot.
We can check the VC log INFO Successfully published attestations type: unaggregated to check that the blocks were published right after the head was updated on the BN.

We could also add some custom logs/metrics to test attestation arrivals over gossip. Not sure if any such metrics already exist cc @michaelsproul

Additional stability

Prysm claimed that the lighthouse behaviour of publishing 4 seconds into the slot was causing some instability when they were publishing as soon as they saw head. I'm not completely sure why that would be the case because we use the work_reprocessing_queue.rs to queue attestations that we haven't seen blocks for yet. To test that this doesn't cause instability, a baseline check we could do is a kurtosis network with a 50:50 split of stable v/s this version and check that the network is stable.

[PoulavBhowmick03]

can i take this up?

[michaelsproul]

I think @hopinheimer might have wanted to tackle this

[hopinheimer]

started work on this. a PR should be up soon.