simulators/rpc: PoC of new simulator rpc-compat-v2, early draft for feedback collection

[felix314159]

Why?

rpc-compat is a good simulator sir, but its current design makes it hard to define new test modes for finer granularity testing. i drafted a v2, where it is easy to define different test modes.

What is a test mode?

a test mode is the logic used to determine whether a test is passed or failed, given the actual & expected outcome. in v1 there seems to be exact matching (e.g. diff-based approach) and specsonly (only type of response matters so it is more forgiving, correct me if i am wrong). since i felt it was difficult to extend rpc-compat with more test modes i decided to draft rpc-compat-v2 where i keep some of v1's logic and make changes where i feel it helps with extensibility (addition of additional test modes).

Which test modes would we want to have?

i do not have knowledge about the history of rpc testing in hive (e.g. what might have been tried in the past), i thought the following test modes might be useful. we can go through the entire list of rpc's we want to test and map them one-by-one to one of these test modes (sorted by hard-to-easy-to-pass):

• exact match (give this exact response or you will fail. we never care about the order of a json object's field tho). basically what v1 rpc-compat does
• ranged match (this might be quite some effort, but we could manually specify acceptable response ranges (e.g. for eth_estimateGas anything in [a,b] passed). i can add a function to dynamically generate io files given any running client with any blockchain state. this way we can agree on some formula per rpc, that does not care about which testnet will be used, to dynamically determine ranges that decide pass/failed and dynamically generate new io files whenever we decide to change our test chain
• type match (like specsonly): we don't care what the clients respond as long as they respond sth with a certain data type and do not throw an error where we do not expect it. it makes sense e.g. for a future rpc where the client just tells you which routines it as of right now is internally running

the entire point of this draft PR is to collect early feedback from people that have more experience than me with rpc testing. the current code only contains logic for 'exact match' tests because i want to collect feedback on the proposed modes before i/we implement them. if you have ideas for more test modes, or don't like the proposed test modes feel free to comment. nothing is set in stone, and v2 is specifically designed for easy removal/addition of test modes.

How to run this?

i just use:
go run . --sim=ethereum/rpc-compat-v2 --client=go-ethereum --docker.buildoutput --loglevel=5
if you want to get spammed with actual responses add --docker.output. I modified the dockerfile so that you only toggle a bool to tell it to either use a locally existing test folder or fetch the io files from execution-apis.

i only tested it with geth, and the only test mode that exists right now (exact match) almost behaves the same as v1: when i tried it in v2 there are 48 failed tests (rest passed), and in v1 there are 49 failed tests (rest passed). the logic is for checking actual responses vs expected responses is completely different so i guess i will find out which test behaves differently than before, if people feel that v2 in general makes sense and is worth looking into. at the end of main.go i added some thoughts / comments i have. it was important to me to not use any 3rd party dependencies to keep it simple. if you want to we can also add performance measuring later (e.g. how long does it take clients to respond to any rpc and whether that should affect fail/pass) but actually hive might already have sth like this (i am not too familiar with all functionality that exists already). i also am unsure about what needs to be formally specced vs what we can should put in the source code, especially in regards to the range-test-mode i proposed.

all in all, i think when you look at the code it should be clear what i am trying to build. it should allow finer granularity testing (in the sense that we can be more forgiving with certain RPC responses to reduce failure noise and motivate client teams to actually care about the failures they see on the dashboard) and since v2 is still in a very early stage feel free to provide any kind of feedback. i can just close the issue if people feel like this is not worth looking into, i spent less than a week on this so far and enjoyed not having to write python all the time lol. let me know what you think / whether this is worth expanding
@fjl @kclowes @marioevz

[fjl]

The main point of this PR is adding the 'range-match' thing. I think it's something we can implement in the existing simulator.