What is the bug?
GET /_plugins/_replication/follower_stats reports a stale follower_checkpoint for an idle-but-fully-synced follower shard, diverging from GET /_plugins/_replication/<index>/_status, which is always correct. The divergence does not self-correct while the index stays idle — monitoring built on follower_stats sees a permanent phantom lag on a healthy, fully-synced index.
How can one reproduce the bug?
- Start replication on an index and let it fully catch up (leader and follower checkpoints equal).
- Stop writing to the leader index so the follower shard goes idle (no new ops for the follower's
GetChanges long-poll to return).
- Compare
follower_stats's index_stats.<index>.follower_checkpoint against <index>/_status's syncing_details.follower_checkpoint while idle.
follower_stats reports a checkpoint behind _status's live value, and the gap persists indefinitely until the leader receives new writes again or the shard replication task restarts.
What is the expected behavior?
Both endpoints should report the same follower_checkpoint for a fully-synced index, and follower_stats should not report a permanent lag on a healthy index.
What is your host/environment?
- Plugin: cross-cluster-replication
- Affected APIs:
follower_stats, _status
Do you have any additional context?
Root cause: follower_stats's follower_checkpoint (FollowerShardMetric.followerCheckpoint) is a push-updated cache, refreshed only after a successful replay batch in TranslogSequencer, or once at task start in ShardReplicationTask. When the leader goes idle, getChanges() in ShardReplicationTask.replicate() long-polls and eventually throws a timeout exception; that path never triggers a replay, so the cached checkpoint is never refreshed and freezes at its last value. _status has no such cache — it reads the live shard checkpoint on every call, so it's always accurate.
This looks like a gap reintroduced by #438 (fix for #431). That fix correctly removed a per-loop-iteration checkpoint update that was reading a premature value (lastSyncedGlobalCheckpoint) before writes had landed, and replaced it with an update-only-after-a-successful-write. That fixed the original premature-read bug, but as a side effect it removed the only mechanism that refreshed the checkpoint while idle, so staleness resurfaces via a different trigger (idle timeout instead of premature read).
What is the bug?
GET /_plugins/_replication/follower_statsreports a stalefollower_checkpointfor an idle-but-fully-synced follower shard, diverging fromGET /_plugins/_replication/<index>/_status, which is always correct. The divergence does not self-correct while the index stays idle — monitoring built onfollower_statssees a permanent phantom lag on a healthy, fully-synced index.How can one reproduce the bug?
GetChangeslong-poll to return).follower_stats'sindex_stats.<index>.follower_checkpointagainst<index>/_status'ssyncing_details.follower_checkpointwhile idle.follower_statsreports a checkpoint behind_status's live value, and the gap persists indefinitely until the leader receives new writes again or the shard replication task restarts.What is the expected behavior?
Both endpoints should report the same
follower_checkpointfor a fully-synced index, andfollower_statsshould not report a permanent lag on a healthy index.What is your host/environment?
follower_stats,_statusDo you have any additional context?
Root cause:
follower_stats'sfollower_checkpoint(FollowerShardMetric.followerCheckpoint) is a push-updated cache, refreshed only after a successful replay batch inTranslogSequencer, or once at task start inShardReplicationTask. When the leader goes idle,getChanges()inShardReplicationTask.replicate()long-polls and eventually throws a timeout exception; that path never triggers a replay, so the cached checkpoint is never refreshed and freezes at its last value._statushas no such cache — it reads the live shard checkpoint on every call, so it's always accurate.This looks like a gap reintroduced by #438 (fix for #431). That fix correctly removed a per-loop-iteration checkpoint update that was reading a premature value (
lastSyncedGlobalCheckpoint) before writes had landed, and replaced it with an update-only-after-a-successful-write. That fixed the original premature-read bug, but as a side effect it removed the only mechanism that refreshed the checkpoint while idle, so staleness resurfaces via a different trigger (idle timeout instead of premature read).