Skip to content

Conversation

@myungjin
Copy link
Contributor

@myungjin myungjin commented Feb 3, 2026

Description

Replace seqno-offset cache cleanup with activity-based approach using exponential moving average (EMA) to prevent memory leaks and premature evictions in pipelined execution with replicas.

Key changes:

  • Track last access timestamp per cache instead of seqno ordering
  • Adaptive timeout: EMA × multiplier × max_inflight for pipeline-aware cleanup
  • Handles out-of-order completion and replica stages correctly
  • Silent operation with zero logging overhead

Configuration:

  • ema_alpha=0.5, timeout_multiplier=10
  • min_timeout=0.5s, max_timeout=30.0s

Type of Change

  • Bug Fix
  • New Feature
  • Breaking Change
  • Refactor
  • Documentation
  • Other (please describe)

Checklist

  • I have read the contributing guidelines
  • Existing issues have been referenced (where applicable)
  • I have verified this change is not present in other open pull requests
  • Functionality is documented
  • All code style checks pass
  • New code contribution is covered by automated tests
  • All new and existing tests pass

Replace seqno-offset cache cleanup with activity-based approach using
exponential moving average (EMA) to prevent memory leaks and premature
evictions in pipelined execution with replicas.

Key changes:
- Track last access timestamp per cache instead of seqno ordering
- Adaptive timeout: EMA × multiplier × max_inflight for pipeline-aware cleanup
- Handles out-of-order completion and replica stages correctly
- Silent operation with zero logging overhead

Configuration:
- ema_alpha=0.5, timeout_multiplier=10
- min_timeout=0.5s, max_timeout=30.0s
@myungjin myungjin merged commit abd004d into cisco-open:main Feb 3, 2026
1 check passed
@myungjin myungjin deleted the fix_lazy_kv_cache_cleanup branch February 3, 2026 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant