Index-Scoped Configuration for Partial Search

[Roaster05]

This PR introduces an index-based configurable setting for partial search results in Elasticsearch. The primary objective is to enable a boolean variable (isPartialSearchAllowed) at the index level, which dictates whether partial search results are allowed for queries involving multiple indices.

Key Changes

1. New Index Metadata Field

• Index Metadata Enhancement: Added a new boolean field partial_search_allowed to the index metadata. This field is dynamically set based on a cluster-wide setting and can be updated independently for each index.

2. Index Setting Update

• Dynamic Setting: Implemented a new index based setting, indices.partial_search.total.use_index_based_setting, to control the index-based partial search behavior.
  • Default value: false
  • This setting is dynamic and scoped to the index triggered.

3. Modified Partial Search Result Limit

• Type: String
• Setting: action.search.modified_partial_search_result_limit
• Default Value: "" (empty string)
• Description: Specifies the age threshold for indexes to allow partial search. Format: XyrXmoXdaXhrXmiXse (years, months, days, hours, minutes, seconds)
• Properties: Dynamic, NodeScope

4. Index Metadata Builders

• Metadata Builders: Introduced builders (IndexMetadata.Builder) to construct and update index metadata with the partial_search_allowed field.
  • Builders facilitate the creation of updated index metadata instances reflecting changes in the partial_search_allowed setting.

5. Python Automation Script

• Automated Index Update Script: Developed a Python script (AutomatedIndexUpdateScript.py) that periodically checks indices based on their creation date.
  • If an index's age exceeds a specified threshold (e.g., 5 minutes), the script sets partial_search_allowed to true, utilizing Elasticsearch's Python client (elasticsearch-py).

6. Integration with Search Request Handling

• Search Request Handling: Integrated logic (TransportSearchAction.java) to evaluate the partial_search_allowed flag for each index involved in a search request.
  • Determines whether to allow partial search results based on the collective state of partial_search_allowed flags across indices in the search request.

Configuration for Index-Based Partial Search Setting

To configure the index.partial_search_allowed setting for a specific index, use the following configuration in Kibana or through the Elasticsearch API:

Enable Partial Search for a Specific Index

Replace {index_name} with the name of your index.

PUT /{index_name}/_settings
{
  "index": {
    "partial_search_allowed": true
  }
}

Setting the Age Configuration Limit

PUT /_cluster/_setting
{
    "action.search.modified_partial_search_result_limit":"3yr5mo6da7mi8se"
}