Best practices for performant eth_call?

[MartinquaXD]

We rely heavily on eth_call requests to query the current state of the chain.
Recently we switched our implementation for fetching effective balances to what we though was a more reasonable implementation only to see that this degraded the whole reth node's performance.

We tried 3 different approaches:

1. eth_call is a contract creation request where we do our logic in the constructor. The data we are interested in effectively gets returned as if it was the byte code of the newly created contract (see Magic: StorageAccessbile Based Simulations cowprotocol/services#1831 for more context)
2. instead of fake deploying a helper contract via the constructor hack we simply fake deploy it using state overrides
3. actually deploy the helper contract to no longer need any state overrides or constructor shenanigans

To my surprise 1 had by far the best latency while 2 and 3 needed significantly less memory.

Here are some of our metrics where you can clearly see how they changed after we switched from 1 to 2. Switching from 2 to 3 had no visible effect.

Do these results make sense to you?
Intuitively I would think that option 1 has the most overhead (we need to pass the constructor byte code of the helper contract in the calldata and we have to run it to even fake-deploy the contract). And I expected 3 to be the fastest because the requests are as small as possible and the node already has the contract as part of the general blockchain state.

Are there recommendations to write the most performant eth_calls?

[mattsse]

hmm, these are the database transaction commit times, mind sharing the rpc related metrics as well?

how many ethcall requests are we talking?

also which version is this?

the drop in memory most likely looks mmap cache invalidation related.

for actual allocated memory you can take a look at the left graph

and for rpc latency

EDIT:

degraded the whole reth node's performance

in terms of rpc latency?

[MartinquaXD]

hmm, these are the database transaction commit times, mind sharing the rpc related metrics as well?

how many ethcall requests are we talking?

I do see ~20/s. However, our system automatically batches multiple requests into RPC batch calls (where you have an array of calls in a single request). Not sure if reth counts the individual sub calls but I guess it does.

the drop in memory most likely looks mmap cache invalidation related.
for actual allocated memory you can take a look at the left graph

Ahh, didn't know that. The other graph doesn't look like it changed much.

in terms of rpc latency?

Yes. I already shared the latency graph of reth above but for context here is how it manifests in our application. The stage of fetching balances went from 500ms - 700ms to all over the place.

also which version is this?

1.4.8 on ethereum mainnet
Edit: we switched to 1.6.0 this morning and the performance remained unchanged.

[MartinquaXD]

I reverted what appeared to be the offending PR but the metrics did not return to normal. Will investigate further it's possible that this was just an unlucky coincidence.

Is it possible that the huge mmap cache invalidation could be related to the performance degradation?