Skip to content

fix: decouple warmup_minutes from HLCV cache hash for better reuse#562

Closed
enarjord wants to merge 10 commits intomasterfrom
fix/hlcv-cache-warmup-reuse
Closed

fix: decouple warmup_minutes from HLCV cache hash for better reuse#562
enarjord wants to merge 10 commits intomasterfrom
fix/hlcv-cache-warmup-reuse

Conversation

@enarjord
Copy link
Owner

Summary

  • Configs that differ only in trading parameters (EMA spans, warmup ratio) now share the same backtest HLCV cache slot, avoiding redundant re-downloads of candlestick data
  • The cache uses a ratchet-up strategy: warmup sufficiency is checked at load time, and the cache is overwritten only when a larger warmup is needed
  • Backward-compatible with existing caches (missing cache_meta.json treated as warmup=0)

Problem

When backtesting two configs that share identical backtest.*, live.approved_coins, and exchange settings but differ in bot.long (e.g., different EMA spans), the HLCV data was re-downloaded from scratch. This happened because warmup_minutes—derived from the max EMA span—was included in the cache hash. Different EMA spans → different warmup → different hash → cache miss → full re-download.

Example with two real configs:

  • template_candidate.json: entry_volatility_ema_span_hours=672 → warmup=36,288 min (25.2 days)
  • template_current.json: entry_volatility_ema_span_hours=2490 → warmup=44,820 min (31.1 days)
  • Same coins, same dates, same exchange → but produced different cache hashes

Solution

  1. get_cache_hash() — removed warmup_minutes from the hash. The key now depends only on data-defining parameters (coins, dates, exchange, coin age, gap tolerance, sources).

  2. load_coins_hlcvs_from_cache() — new warmup_minutes parameter. Reads cache_meta.json and checks cached_warmup >= needed_warmup. Returns None if insufficient.

  3. save_coins_hlcvs_to_cache() — new warmup_minutes parameter. Writes cache_meta.json. Removed the early-return guard so that re-fetches with larger warmup can overwrite the same cache slot.

  4. prepare_hlcvs_mss() — threads backtest_warmup_minutes through to both load and save.

Ratchet behavior

Run Config warmup Cached warmup Result
1 36,288 Miss → fetch → save (cached=36,288)
2 36,288 36,288 Hit
3 44,820 36,288 Miss (insufficient) → fetch → save (cached=44,820)
4 36,288 44,820 Hit (44,820 >= 36,288)
5 44,820 44,820 Hit

Test plan

  • 16 new tests in tests/test_cache_warmup_reuse.py covering:
    • Cache hash independence from EMA spans and warmup ratio
    • Non-warmup fields (coins, dates) still differentiate hashes
    • Warmup sufficiency gate (hit/miss/boundary)
    • Legacy cache compatibility (no cache_meta.json)
    • Corrupt cache_meta.json fallback
    • Save persists cache_meta.json (uncompressed + compressed)
    • Overwrite behavior (ratchet-up)
    • Full end-to-end ratchet scenario
  • Full test suite: 744 passed, 118 skipped, 0 failures

🤖 Generated with Claude Code

enarjord and others added 10 commits February 10, 2026 08:58
Wire Hyperliquid builder codes through CCXT's built-in infrastructure.
Overrides CCXT's default builder address with Passivbot's own in
broker_codes.hjson (ref, builder address, fee rate, fee int).

On by default: CCXT auto-attaches builder info to orders when approved.
Adds a periodic nag banner (every 30 min) encouraging users to approve
the builder fee via their main wallet, modeled on Binance's
print_new_user_suggestion pattern. Bot continues working without
approval - users are never blocked.

Default fee: 2 bps (0.02%), configurable via broker_codes.hjson.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update builder code implementation to the full three-path design:

Path A (already approved): Modest thank-you at startup, builder
  codes active on all orders.

Path B (not approved + main wallet): Auto-approve via API, print
  notice banner explaining what happened, then thank-you.

Path C (not approved + agent wallet): Force-enable builder codes
  so the first order fails with approval error. Catch the error,
  show nag banner with approval instructions, temporarily disable.
  Re-enable every ~30 min to re-nag or detect external approval.
  "Free version with ads" pattern.

Also: suppress CCXT's default auto-approval (we handle it ourselves),
add real builder address 0x5e20A6D7e11366390Fde63EA3d0A026903359a74,
rewrite plan doc to reflect all three paths and early-warning strategy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adopt improvements from Codex's branch comparison:
- Add CHANGELOG.md entry for builder codes
- Add docs/hyperliquid_guide.md builder codes section
- Add builder_fee toggle in broker_codes.hjson
- Add robust _is_positive_fee_value() and _extract_max_builder_fee()
- Add input validation with fallback for fee_int
- Add class constants for timing intervals
- Add status check caching to prevent endpoint hammering
- Add 13 unit tests covering static methods, config, async paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Initialize all builder state (_builder_initialized, _builder_pending_approval,
  _builder_disabled_ts) in __init__ instead of hasattr/getattr fallbacks
- _has_builder_config() now respects builder_fee toggle, so _init_builder_codes
  skips entirely when disabled
- _maybe_reenable_builder_codes() uses BUILDER_NAG_INTERVAL_MS class constant
  instead of hardcoded value
- Support both camelCase (feeRate, feeInt, builderFee) and snake_case
  (fee_rate, fee_int, builder_fee) key names in broker_codes.hjson
- execute_order uses logging.error for the failure + logging.warning for
  the fallback action (proper severity levels)
- Add builder codes comment to api-keys.json.example
- Add 4 new tests: camelCase aliases, toggle in _has_builder_config,
  re-enable cycle (still not approved), re-enable cycle (now approved)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Incorporate improvements from feature/hl_builder_codes_codex_gpt5:
- Add _normalize_builder_settings() to centralize key alias resolution
  and cache in _builder_settings (eliminates duplicate parsing)
- Add _builder_thank_you_printed dedup flag (prevents repeated banners)
- Add dedicated _print_builder_thank_you_banner and
  _print_builder_approved_notice_banner methods
- Add cca None guards in _check_builder_fee_approved, _set_builder_approved,
  _init_builder_codes
- Fix disable bug: ref is now applied even when builder_fee=false

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Standalone async script that lets agent-wallet users approve the builder
fee from their main wallet without modifying api-keys.json. Referenced
by the nag banner and docs/hyperliquid_guide.md.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Guard getpass with TTY check to prevent plaintext key echo
- Zero main_key reference after use
- Never log exception messages (use type name only) to avoid leaking key material
- Use logging.error instead of logging.exception to suppress tracebacks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Configs that differ only in trading parameters (EMA spans, warmup ratio)
now share the same HLCV cache slot instead of triggering redundant
re-downloads. The cache uses a ratchet-up strategy: warmup sufficiency
is checked at load time via cache_meta.json, and the cache is
overwritten only when a larger warmup is needed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@enarjord
Copy link
Owner Author

Superseded by #564, which contains only the targeted HLCV cache warmup reuse changes on top of current master.

@enarjord enarjord closed this Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant