Describe the blog post

Segment replication is a powerful index replication strategy for high QPS workloads that allows physical isolation and decoupling of indexing and search workloads, rapid replica failovers, and seamless point in time restores. It leverages Lucene's write once architecture indexing documents only once on indexer nodes (primary shards), and replicating built segments across all searcher nodes (replica shards).

However, the beauty of decoupled indexing and search comes with its own unique challenges. The system can accummulate large replication checkpoints during high indexing bursts, or if replication gets delayed due to network issues. Searchers (replica shards) struggle to absorb these checkpoints experiencing high page faults, thrashing for in-flight queries, and degraded search performance. To address these issues, we've recently added hooks in open-source Lucene that enable searchers to perform an "Adaptive Refresh". Instead of refreshing to the latest commit for a large replication payload, searchers can intelligently select the commit point that they can safely absorb. By processing through multiple such points, searchers can eventually get to the latest commit, without incurring too many page faults.

This blog dives into the searcher refresh mechanism in Lucene, how adaptive refresh works, and the strategies to implementing it successfully in a highly replicated search engine.

Expected Title

Adaptive Refresh in Lucene for Resilient Segment Replication

Authors Name

Anoushka Gade, Vigya Sharma

Authors Email

[email protected], [email protected] (vigyasharma on GitHub)

Target Draft Date

10/20/2025

Blog Post Category

technical

Target Publication Date

10/29/2025

Additional Info

This will be a Lucene specific post aimed at the "Lucene" tag on the blog.