Seata Benchmark CLI - Design Discussion

[WangzJi]

Status: Draft
Related Issue: #7646

Table of Contents

• 1. Background & Motivation
• 2. Requirements
• 3. User Interface Design
• 4. Technical Design
• 5. Open Questions
• 6. Implementation Roadmap

---

1. Background & Motivation

1.1 Current Situation

Seata currently lacks an official command-line benchmark tool for performance testing. Users who want to evaluate Seata's performance need to:

• Write custom test code
• Set up complex test environments
• Manually collect and analyze metrics
• Lack standardized benchmarking methodology

This creates barriers for:

• New users evaluating Seata performance
• Performance regression testing in CI/CD
• Comparing different transaction modes (AT/TCC/XA/SAGA)
• Tuning Seata Server configuration

1.2 Goals

Provide a simple, standardized, command-line benchmark tool that:

1. Ease of Use: Minimal parameters to run benchmarks
2. Standardized Metrics: Consistent output format (RT, QPS, success rate, latency percentiles)
3. Multiple Modes: Support AT and TCC transaction modes initially
4. No Database Dependency: Focus on empty transactions to test Seata Server capacity
5. Extensible: Design for future enhancements (more modes, real scenarios, distributed testing)

---

2. Requirements

2.1 Functional Requirements

Must Have (v1.0):

• Support AT mode transaction benchmarking
• Support TCC mode transaction benchmarking
• Accept minimal parameters: Seata Server address, mode, TPS, threads, duration
• Output core metrics: Total transactions, success rate, average TPS, RT, P95/P99 latency
• Empty transaction scenario (no business database required)

Should Have (v1.0):

• Warmup support to exclude initial ramp-up from statistics
• Latency sampling to prevent OOM on large-scale tests
• CSV export for post-analysis

Could Have (Future):

• Real-time TUI (Terminal User Interface) display
• Real business scenarios (account transfer, order creation)
• XA/SAGA mode support
• Distributed benchmark coordination

2.2 Non-Functional Requirements

• Usability: One-line command to run basic benchmark
• Performance: Support high TPS testing (1000+ TPS)
• Portability: Single executable JAR, JDK 8+ compatible
• Observability: Clear progress indication and error reporting
• Extensibility: Pluggable executor design for adding new modes

---

3. User Interface Design

3.1 Command-Line Interface

Basic Usage:

# AT mode benchmark
java -jar seata-benchmark-cli.jar \
  --server 127.0.0.1:8091 \
  --mode AT \
  --tps 100 \
  --duration 60

# TCC mode benchmark
java -jar seata-benchmark-cli.jar \
  --server 127.0.0.1:8091 \
  --mode TCC \
  --tps 200 \
  --threads 20 \
  --duration 120

Advanced Options:

java -jar seata-benchmark-cli.jar \
  --server 127.0.0.1:8091 \
  --mode AT \
  --tps 500 \
  --threads 50 \
  --duration 300 \
  --warmup-duration 30 \
  --export-csv results.csv \
  --application-id benchmark-app \
  --tx-service-group default_tx_group

3.2 Parameters

Parameter	Short	Type	Default	Description
--server	-s	String	Required	Seata Server address (host:port)
--mode	-m	String	Required	Transaction mode: AT, TCC
--tps	-t	Integer	100	Target transactions per second
--threads		Integer	10	Number of concurrent threads
--duration	-d	Integer	60	Benchmark duration in seconds
--warmup-duration		Integer	0	Warmup period in seconds (excluded from final stats)
--export-csv		String	-	Export metrics to CSV file
--application-id		String	benchmark-app	Seata application ID
--tx-service-group		String	default_tx_group	Seata transaction service group
--help	-h	-	-	Show help message
--version	-V	-	-	Show version information

3.3 Output Format

Console Output (Real-time Progress):

[00:05] 500 txns, 100.2 txns/sec, 98.0% success, avg 12ms
[00:10] 1000 txns, 100.1 txns/sec, 99.0% success, avg 11ms
[00:15] 1500 txns, 99.8 txns/sec, 99.2% success, avg 13ms
...

Final Report:

===================================================
      Seata Benchmark Final Report (AT Mode)
===================================================
Total Transactions:    6,000
Success Count:         5,940
Failed Count:          60
Success Rate:          99.00%
Average TPS:           100.2
Elapsed Time:          60 seconds

Latency Statistics:
  P50:                 12 ms
  P95:                 45 ms
  P99:                 89 ms
  Max:                 230 ms

Warmup Statistics (excluded):
  Warmup Transactions: 500
  Warmup Duration:     5 seconds
===================================================

CSV Export Format:

Metric,Value
Total Transactions,6000
Success Count,5940
Failed Count,60
Success Rate (%),99.00
Average TPS,100.2
Elapsed Time (s),60
Latency P50 (ms),12
Latency P95 (ms),45
Latency P99 (ms),89
Latency Max (ms),230
Warmup Transactions,500
Export Timestamp,2025-12-01T10:30:45Z

---

4. Technical Design

4.1 Architecture

┌─────────────────────────────────────────────┐
│         BenchmarkApplication (CLI)          │
│              (PicoCLI Framework)            │
└──────────────────┬──────────────────────────┘
                   │
        ┌──────────┴──────────┐
        │                     │
   ┌────▼─────┐        ┌──────▼──────┐
   │  Config  │        │  Seata TM   │
   │  Loader  │        │   Client    │
   └────┬─────┘        └──────┬──────┘
        │                     │
        └──────────┬──────────┘
                   │
        ┌──────────▼──────────────┐
        │  TransactionExecutor    │
        │   (Strategy Pattern)    │
        ├─────────────────────────┤
        │  - ATModeExecutor       │
        │  - TCCModeExecutor      │
        └──────────┬──────────────┘
                   │
        ┌──────────▼──────────────┐
        │   WorkloadGenerator     │
        │  (Guava RateLimiter)    │
        └──────────┬──────────────┘
                   │
        ┌──────────▼──────────────┐
        │   MetricsCollector      │
        │  (Latency Sampling)     │
        └──────────┬──────────────┘
                   │
        ┌──────────▼──────────────┐
        │      Reporter           │
        │  (Console + CSV)        │
        └─────────────────────────┘

4.2 Core Components

4.2.1 BenchmarkApplication (Main Entry)

• CLI framework: PicoCLI (annotation-driven, modern)
• Responsibilities:
  • Parse command-line arguments
  • Initialize Seata TM client
  • Orchestrate benchmark execution
  • Handle graceful shutdown

4.2.2 TransactionExecutor (Strategy Pattern)

Interface:

public interface TransactionExecutor {
    void init(BenchmarkConfig config);
    TransactionRecord execute();
    void destroy();
}

Implementations:

• ATModeExecutor: Empty AT transaction (begin -> commit, no SQL)
• TCCModeExecutor: Mock TCC transaction (empty try/confirm/cancel)

4.2.3 WorkloadGenerator

• TPS control: Guava RateLimiter
• Multi-threaded execution: Fixed thread pool
• Graceful termination on duration timeout
• Pause/resume support (optional)

4.2.4 MetricsCollector

Key Metrics:

• Total transactions (counter)
• Success/failure counts (atomic counters)
• Latency tracking with sampling (inspired by Kafka)
  • Sample every Nth transaction to limit memory
  • Max 500,000 samples stored
• Current and average TPS calculation
• Percentile calculation: P50, P95, P99

Sampling Strategy:

int sampling = totalTransactions / Math.min(totalTransactions, 500_000);
if (currentTransaction % sampling == 0) {
    storeLatency(latency);
}

4.2.5 Reporter

• Console progress updates (window-based, every 5 seconds)
• Final report formatter
• CSV exporter (optional)

4.3 Empty Transaction Implementation

AT Mode Empty Transaction

public TransactionRecord execute() {
    GlobalTransaction tx = GlobalTransactionContext.getCurrentOrCreate();
    long startTime = System.currentTimeMillis();

    try {
        tx.begin(30000, "benchmark-at-tx");
        String xid = tx.getXid();

        // No SQL operations - pure Seata protocol overhead

        tx.commit();
        long duration = System.currentTimeMillis() - startTime;
        return new TransactionRecord(xid, "Committed", duration, true);

    } catch (TransactionException e) {
        tx.rollback();
        long duration = System.currentTimeMillis() - startTime;
        return new TransactionRecord(null, "Failed", duration, false);
    }
}

TCC Mode Mock Transaction

@LocalTCC
public interface BenchmarkTCCAction {
    @TwoPhaseBusinessAction(name = "benchmarkAction",
                           commitMethod = "commit",
                           rollbackMethod = "rollback")
    boolean prepare(BusinessActionContext context);

    boolean commit(BusinessActionContext context);

    boolean rollback(BusinessActionContext context);
}

// Implementation: All methods return true (no-op)
public class BenchmarkTCCActionImpl implements BenchmarkTCCAction {
    public boolean prepare(BusinessActionContext context) {
        return true; // Mock success
    }

    public boolean commit(BusinessActionContext context) {
        return true; // Mock success
    }

    public boolean rollback(BusinessActionContext context) {
        return true; // Mock success
    }
}

4.4 Technology Stack

Component	Technology	Rationale
CLI Framework	PicoCLI 4.7.5	Modern, annotation-based, extensive validation
Rate Limiting	Guava RateLimiter	Proven, accurate TPS control
Seata Client	seata-all (latest)	Official Seata SDK
Logging	SLF4J + Logback	Standard Java logging
Build Tool	Maven	Consistent with Seata project
Packaging	Maven Shade Plugin	Single executable JAR

---

5. Open Questions

Please provide your feedback on the following design decisions:

Q1: TUI (Terminal User Interface) Support?

Options:

• A: Plain text output only (like Kafka - simple, scriptable)
• B: TUI with real-time updates (like htop - visual, intuitive)
• C: Both supported via --tui flag (flexible, more complexity)

Recommendation: Start with A (plain text), add B in v2.0 if community requests

Q2: TCC Mode Implementation?

Options:

• A: Mock TCC (empty try/confirm/cancel, no database)
• B: Real TCC scenario (account freeze/unfreeze, requires database)
• C: Start with A, add B as --scenario option in future

Recommendation: A for v1.0 (aligns with "no database dependency" goal)

Q3: Monitoring global_table/branch_table?

Options:

• A: No monitoring (avoids exposing Seata Server DB credentials)
• B: Optional monitoring via --monitor-db parameter
• C: AT mode only (TCC has no branch_table to monitor)

Recommendation: A for v1.0 (keep it simple)

Q4: Result Export Formats?

Options:

• A: Console output only
• B: CSV export via --export-csv
• C: JSON export via --export-json
• D: B + C (multiple formats)

Recommendation: B (CSV is sufficient for most analysis needs)

Q5: CLI Framework Choice?

Options:

• A: PicoCLI (modern, annotation-based, good validation)
• B: argparse4j (Kafka's choice, Python-style)
• C: joptsimple (lightweight, Java-native)

Recommendation: A (aligns with modern Java practices)

Q6: XA Mode Support?

Options:

• A: Not in v1.0 (focus on AT/TCC as per community meeting)
• B: Support AT/TCC/XA in v1.0

Recommendation: A (community meeting prioritized AT/TCC)

Q7: Warmup Support?

Options:

• A: Must have in v1.0 (Kafka best practice)
• B: Optional in v1.0 via --warmup-duration
• C: Defer to v2.0

Recommendation: B (important for accurate benchmarks, but optional parameter)

Q8: Latency Percentiles?

Options:

• A: P50, P95, P99 only
• B: Add P99.9 (important for tail latency analysis)
• C: Configurable percentiles via parameter

Recommendation: B (P99.9 is valuable with minimal cost)

---

6. Implementation Roadmap

Phase 1: MVP (v0.1)

Scope:

• AT mode empty transaction executor
• TCC mode mock transaction executor
• Basic CLI (server, mode, tps, threads, duration)
• Plain text progress and final report
• Core metrics: total txns, success rate, avg TPS, P50/P95/P99 latency

Deliverables:

• Executable JAR
• README with usage examples
• Basic test coverage

Phase 2: Enhancement (v0.2)

Scope:

• Latency sampling (prevent OOM)
• Warmup support (--warmup-duration)
• CSV export (--export-csv)
• P99.9 percentile
• Window-based progress reporting
• Comprehensive error handling

Deliverables:

• Enhanced JAR with all features
• User guide documentation
• Performance test report (validate tool can handle 1000+ TPS)

Phase 3: Advanced Features (v1.0) - TBD

Scope (based on community feedback):

• TUI real-time display (optional)
• Real business scenarios (configurable via --scenario)
• XA/SAGA mode support
• Monitoring integration (global_table/branch_table)
• Distributed benchmark coordination

Deliverables:

• Production-ready v1.0 release
• Complete documentation

---

7. Community Feedback

Please share your thoughts on:

1. Scope: Is the v1.0 scope appropriate? Too ambitious or too limited?
2. Parameters: Are the proposed CLI parameters sufficient? Any missing?
3. Output: Is the console + CSV output adequate, or should we prioritize TUI?
4. Open Questions: Your preferences on Q1-Q8 above
5. Roadmap: Any changes to the proposed timeline?

How to Provide Feedback:

• Comment on this discussion thread
• Email the dev mailing list
• Join the community meeting

---

Thank you for your valuable input!

[WangzJi]

[funky-eyes]

I propose the following features:

1. Add a configuration option to set the number of branch transactions (configurable branch count).
2. Support an “empty” mode that does not open any branch transactions (i.e., no branch tx are started).
3. Implement a real AT mode for testing: for example, automatically create mock schemas/tables at startup using an embedded database such as H2.
4. Support rollback-performance testing — exercise the full two‑phase rollback path so both phases end up rolling back.
5. Support fault‑injection with configurable error ratios (e.g., simulate a 99.99% probability of rollback).
6. Support annotation‑driven Saga performance testing (ability to run performance tests against Saga flows via annotations).

These items can be scheduled across development phases, but I recommend implementing items 4, 5 and 6 in the first phase.

[WangzJi]

Thank you for the valuable suggestions!
I’ve received your feedback and will take these ideas into account in our upcoming design work.
The related features will be planned and implemented in phases, and I’ll also provide the corresponding documentation along the way.
Thanks again for your input and support!