Only store single reference to mapping for indices that share the same mappings in the cluster state

[cwperks]

Description

This PR contains an optimization for the cluster state that would particularly be beneficial in cases of index rollover where many indices in the cluster share the same mappings.

This is another (and probably the largest) optimization along with 2 other related PRs:

• Add circuit breaking logic for GET _mappings to estimate the size of the mappings before executing a request #19857
• Prevent calling filterFields multiple times on indices with the same mappings in GET _mapping #19914

For this optimization, it works at the time that the new Metadata instance is built to try de-duplicating the mappings if multiple indices share the same mappings.

Related Issues

Related to:

• [BUG] Autocomplete feature in dev console can affect cluster availability OpenSearch-Dashboards#10591
• Autocomplete refresh performance issue in devtools OpenSearch-Dashboards#10047
• Devtools should call <index>/_mappings rather than _mappings API OpenSearch-Dashboards#10550
• [BUG] Dev Tools tries to load every mapping in the cluster when running any command OpenSearch-Dashboards#5654
• [BUG] Indices with large mappings can grind Opensearch Dashboards to a halt OpenSearch-Dashboards#8161
• [BUG] Mapping size is larger than 10MB - Why is it hardcoded!? OpenSearch-Dashboards#2340
• [BUG] Spec testing for GET /_mapping runs out of memory opensearch-api-specification#384

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cwperks]

This seems like low hanging fruit to optimize cluster state heap utilization esp in clusters with many rolled over indices. Is there anything I'm missing with this?

[github-actions]

❌ Gradle check result for 66fe01d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[cwperks]

MappingMetadata can only have a single type which is _doc. See https://stackoverflow.com/a/53688626 for more details.

Single mapping type is declared here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/mapper/MapperService.java#L126

[github-actions]

❌ Gradle check result for 4937d4a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 4937d4a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[kaushalmahi12]

I don't see a case where we are deleting a index, because it can happen that the mappings object we are holding in memory is deleted as part of the delete index call, then all other indices with same mapping pointing to null effectively.

Is that not the case ?

[cwperks]

The logic added here is within the metadata builder so the indices object in this builder would be only the present indices. I need to check, but on index deletion it would create a new version of the metadata using this builder with all of the indices except the one that has been deleted. This function is then called in build() to de-dup the mappings so that we don't hold multiple versions of the same mappings in-memory for the new in-memory Metadata.

[cwperks]

Relevant code: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/cluster/metadata/MetadataDeleteIndexService.java#L159-L169