ERPbench

A benchmark for evaluating LLM multi-agent systems in long-horizon business decision-making, built on the ERPsim simulation platform.

Overview

ERPbench tests how teams of LLM agents (CEO, CFO, CMO) collaborate to make pricing, production, and procurement decisions across 6 rounds of a competitive market simulation. The benchmark supports:

• Solo mode: Single agent (CEO) makes all decisions
• Arena mode: Head-to-head competition between teams
• Construct diversity: Homogeneous vs. heterogeneous agent teams
• Difficulty levels: Easy, Medium, Hard, Random-Hard

Quick Start

Prerequisites

• Python 3.12+
• uv (recommended) or pip
• Node.js 20+ (for frontend)

Installation

# Clone the repository
git clone https://github.com/your-org/erp-bench.git
cd erp-bench

# Install backend dependencies
uv sync

# Install frontend dependencies (optional)
cd frontend && npm install

Configure API Keys

cp .env.example .env
# Edit .env with your LLM API keys

Run a Quick Test

# Test LLM connectivity
uv run python -m orgconstruct test-llm

# Run a single solo experiment (1 team, 6 rounds)
uv run python -m orgconstruct run-erpsim --teams-solo 1 --rounds 6 --seed 42

# Run a competitive arena (2 vs 2 teams)
uv run python -m orgconstruct run-erpsim --teams-homo 2 --teams-hetero 2 --rounds 6 --seed 42

Project Structure

.
├── backend/src/orgconstruct/    # Core simulation + agents
│   ├── simulation/              # ERPsim engine (demand, finance, supply chain, etc.)
│   ├── agents/                  # LLM agents (CEO, CFO, CMO) + prompts
│   ├── llm/                     # LLM client factory (OpenAI-compatible)
│   ├── memory/                  # Long-term memory + reflection
│   ├── kelly/                   # Kelly Personal Construct Theory engine
│   ├── platform/                # Web platform (auth, matchmaking)
│   ├── api/                     # FastAPI REST endpoints
│   └── cli/                     # Command-line interface
├── frontend/                    # React 19 + TypeScript + Tailwind
├── backend/tests/               # Pytest test suite
├── data/                        # Benchmark results (CSV/JSON)
└── config/                      # YAML configs (agents, LLM providers, simulation)

Benchmark Data

The data/ directory contains aggregated results from our experiments:

File	Description
solo_detailed_results.csv	Per-round results for solo mode
solo_overall_stats.csv	Aggregate statistics by model
solo_by_difficulty_stats.csv	Performance breakdown by difficulty
arena_36_complete.csv	Complete arena competition results
decision_traces.json	Full decision traces for analysis
benchmark_all_results.json	Unified benchmark scores

See data/README.md for detailed field descriptions.

Experiment Conditions

The benchmark supports a 2×2+2 factorial design:

	Homogeneous	Heterogeneous
Stable Market	stable_homogeneous	stable_heterogeneous
Dynamic Market	dynamic_homogeneous	dynamic_heterogeneous
Kelly Constructs	kelly_homogeneous	kelly_heterogeneous

Plus Solo mode as an ablation baseline.

Testing

# Run all tests
uv run pytest

# Run a specific test file
uv run pytest backend/tests/test_demand.py

# Run with coverage
uv run pytest --cov=orgconstruct

Citation

If you use ERPbench in your research, please cite:

@inproceedings{erpbench2026,
  title={{ERPbench}: Benchmarking Multi-Agent Systems in Long-Horizon Business Decision-Making},
  author={Zhang, Xinran and Lu, Pengrui and Liu, Pengfei},
  booktitle={Proceedings of the Fortieth AAAI Conference on Artificial Intelligence},
  year={2026}
}

License

This project is licensed under the MIT License. See LICENSE for details.

Acknowledgments

• ERPsim platform by HEC Montréal
• Market-Share Analysis (MCI) demand model by Cooper & Nakanishi (1988)