CL/EL: consider caching requests for certain routes

[iurii-ssv]

While adding missing metrics for EL (and logs for EL/CL) I've noticed that for some routes (AttestationData fetching, for example) we seem to have quite high request rate per second (10+),

we might want investigate this further to see if some of the routes can use caching (similar to the kind we do for AttestationData CL calls), especially if request takes > ~10-100ms (but even if it takes less - the sheer number of them can be problematic as well).

We can find all the relevant CL/EL requests by querying for the following log-lines (also note, some of these might be actually throttled to 1% or 10% of their actual occurrence simply to keep the logs growth at bay):

CL single-client request done
CL multi-client request done
EL single-client request done

Also note, these logs (and corresponding metrics) ^ are supposed to track only "real requests made", (I guess "real" here means a call into 3rd-party client library, not necessarily an HTTP/RPC call, since we don't have any control of what that code in there does - they might be doing their own caching, as seems to be the case for CL "Domain" route)

We might also consider pre-fetching some data, eg. "Domain" call seems like a good candidate for that.

[iurii-ssv]

For "Domain" call there is a caching implemented by go-eth2-client (their multi-client Service struct), but it seems we are fetching it lazily - here is an example of ~10 calls blocking & waiting on what I assume is a network-request (because takes > 1s, and it finishes successfully):

yet we are requesting this "Domain" call straight during duty-execution (via GoClient.DomainData) in multiple duties ... so, it add a potentially critical delays for duty execution. We might be able to pre-fetch the data 1 epoch prior to the target epoch "Domain" call accepts as parameter. Pre-fetching might also be relevant for other routes as well.

[iurii-ssv]

Seems like we are sending ~10 NodeSyncing requests per slot, if these aren't cached already - it's a good candidate for optimization

[github-actions]

This issue has been marked as stale due to 60 days of inactivity. It will be closed in 30 days if there are no updates. Please comment if you would like to keep it open.