Sockfish — Adaptive RL Chess Engine

A chess engine that learns entirely through reinforcement learning, using Stockfish as an adaptive sparring partner rather than a static oracle, wrapped in a live multi-threaded training dashboard.

Quick start (macOS)

# 1. Stockfish (already satisfied if `which stockfish` prints a path)
brew install stockfish

# 2. Python environment
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

# 3. Launch the dashboard
.venv/bin/python main.py

# Optional: headless smoke test (no GUI)
.venv/bin/python main.py --selftest

Press Start in the dashboard. Training runs continuously in the background until you press Pause or Stop (Stop joins all threads and checkpoints the model safely).

How it works

Phase 0 — Self-play bootstrap with an Elo gate

Training begins as self-play against frozen snapshots of the agent's own past weights. Every 5th game is a gate game against a ~Elo-700 Stockfish; once 100 of the last 500 gate games are wins, the engine graduates to the ladder.

Phase 1 — Adversarial Ladder (dynamic bootstrapping)

Each worker thread owns a private Stockfish process spoken to over the UCI protocol. The ladder starts at Elo 800; when the agent sustains a >50% win rate (wins only) over a 40-game window (min 30 games at the rung), the pipeline issues UCI commands to promote Stockfish by 100 Elo.

Stockfish 16+ only accepts UCI_Elo ≥ 1320, so rungs 800–1300 are approximated with Skill Level 0–5 plus tight node budgets; from 1320 upward the genuine UCI_LimitStrength/UCI_Elo mechanism is used.

Phase 2 — Pure self-play

After conquering the Elo 1400 rung, Stockfish is removed entirely. Opponents become frozen snapshots of the agent's own past weights (a pool of up to 10, refreshed every 50 games).

The reward design

• Outcomes: win +1 plus a speed bonus up to +0.2 for fast wins; repetition / fifty-move / ply-cap draws −0.8 (nearly all draws are shuffle loops); stalemate-style draws −0.5; loss −1.
• Potential-based material shaping: each move earns scale·(γ·Φ(s′) − Φ(s)) where Φ is the material balance — dense credit for winning material long before full wins are convertible, and provably policy-invariant (Ng et al., 1999).
• Repetition step penalty: −0.05 the moment the agent's own move repeats a position, attacking shuffle loops at the source.

The learning algorithm

• Policy network: numpy MLP, 774 features (12 piece planes + state bits) → 512 → 256 → 4096 from/to move logits, illegal moves masked.
• Perspective-canonical encoding: positions are always presented from the side-to-move's point of view (rank-flipped for Black, own/enemy piece planes), so everything learned as White transfers to Black — without this the network must learn chess twice. Checkpoints carry an encoding version; incompatible ones are archived automatically.
• REINFORCE with a running-mean baseline, Adam, gradient clipping, batched updates (4 games per gradient step) and per-batch advantage normalization for low-variance, consistently scaled steps.
• Epsilon-greedy exploration: 10% of moves are uniformly random legal moves, breaking exploitation loops (exploratory moves shape outcomes but are excluded from the gradient since the policy didn't pick them).
• Adaptive entropy regularization: the entropy coefficient is steered so average policy entropy tracks a target (1.8 nats) — pressure rises when the policy collapses toward certainty too early, and relaxes once exploration is healthy. The Docs tab plots this curve live.

In-app documentation tab

The Docs tab (header, top right) maps the model's behaviour as live, timestamped data tables: run & session state (uptime, games, throughput, gate/ladder status), training & model internals (parameters, updates, entropy, adaptive beta, baseline, checkpoint size and save time), the full milestone-tournament history, "estimated Elo after N games" checkpoints, and the phase-change timeline — plus the Est. Elo and policy-entropy charts.

Dashboard (2:3 split, per the Criterion B wireframe)

• Dark-blue rounded-card theme (custom canvas-drawn cards, buttons, chips and slider — the only sharp corners left are the chessboards), with a header strip of live stat chips (phase, ladder Elo, total games, games/min throughput, active workers).
• Left: the Highlight Board replays the most recently finished game at one move per second with sliding piece animation — preferring wins, then draws, then losses — with a colored result pill, and only swaps games when the replay ends. Beneath it, the timestamped, colorized Live Terminal Feed.
• Right: 3×3 grid of nine mini-boards, each replaying a random recently finished game at one move per second (a visualization of the model — actual training runs headless, up to 100 concurrent games via the slider), plus the controls ribbon (wins-only winrate over the past 1000 games with a W/D/L breakdown) and the dynamic Est. ELO vs Games Played chart: binned, EMA-smoothed, Retina-crisp native canvas, with dashed markers at the gate pass, every ladder promotion and the Phase 2 transition.

Analytics

• Every game → logs/training_log.csv (buffered writes; high concurrency can finish hundreds of games per second).
• Every 1000 games training pauses for an isolated 6-game test tournament against a fixed baseline (current rung in Phase 1, Elo 2000 in Phase 2); the performance-Elo estimate lands in logs/milestones.csv and on the graph, which resumes across sessions.
• Checkpoints: models/sockfish_model.npz + models/training_state.json (auto-saved every 60 s, on Stop, and auto-resumed on launch).

Performance notes

• Policy inference is lock-free: the optimizer publishes each update as a whole-dict atomic swap, so 100 worker threads never serialize on a lock.
• Each Stockfish process runs with 1 thread / 8 MB hash; expect roughly 20–40 MB RSS per concurrent game at the top of the slider.

Project layout

main.py                     entry point (GUI or --selftest)
sockfish/
  config.py                 every tunable constant
  engine/stockfish_engine.py  UCI wrapper + adaptive Elo control
  rl/encoding.py            board → 774-vector, 4096-way move index
  rl/network.py             numpy policy net, REINFORCE backprop, Adam
  rl/agent.py               epsilon-greedy agent + frozen opponents
  rl/trainer.py             returns, baseline, entropy schedule
  training/ladder.py        dynamic Elo ladder / Phase 2 trigger
  training/pipeline.py      worker threads, trainer thread, milestones
  analytics/logger.py       CSV logs + Elo estimation
  gui/theme.py              shared dark-blue palette
  gui/widgets.py            rounded cards, buttons, pills, chips, slider
  gui/chart.py              native canvas Est. ELO chart (EMA-smoothed)
  gui/board_widgets.py      FEN-rendering Tkinter canvases
  gui/dashboard.py          the 2:3 dashboard + replay controller