[tool] Improve benchmark tool

[siyuanfoundation]

Background

The current tools/benchmark in etcd is a low-level tool that directly uses the gRPC client to generate load and report performance with limited measuring dimension. However, it has several limitations that make it difficult to use for comprehensive performance analysis:

• Lack of Time-Series Data: It primarily outputs a single summary number at the end of the run (e.g., average latency). This hides performance spikes, degradation over time, or warming-up effects.
• No Resource Monitoring: It does not monitor or correlate etcd server resource usage (specifically memory footprint) during the benchmark run.
• Lack of Guidance: There is no standard guidance or set of profiles defined for what tests to run to measure performance consistently, making it hard to compare results across different setups.
• Manual Comparison: Comparing results across different branches or commits requires manual checkout, build, and execution, making regression testing tedious.
• Environment Setup: Running tests in a clean, isolated environment requires manual setup of etcd instances.

Proposal

We could improve the tool in the following areas:

• Create Docker setup for single-node etcd and benchmark client.
• Implement time-series data collection for latency and resource usage (including etcd memory monitoring).
• Implement branch/commit comparison script to automate checkout, build, and diff.
• Define and implement standardized benchmark test profiles ("ReadHeavy", etc.).
• Implement visualization (e.g., terminal ASCII or static image generation).

[siyuanfoundation]

/cc @serathius @ronaldngounou @kishen-v

[serathius]

While all think you listed would be a good improvement, what's the end goal?

For example #16467 focuses on building e2e integration with K8s scalability tools so that we can run periodic benchmark and request /test benchmark-put on PR.

[siyuanfoundation]

This would be part of #16467.
The end goal would be:

• better documentation and clarity for users to run one off but meaningful benchmarks
• integrate into the benchmark CI workflows, like the standardized benchmark test profiles