ES index mode & codec in metricbeat by 3kt · Pull Request #49237 · elastic/beats

3kt · 2026-03-03T22:53:20Z

Type of change: Enhancement

Proposed commit message

Add mode and codec fields to the elasticsearch.index metricset

WHAT:

Add two optional fields to the Metricbeat Elasticsearch index metricset: elasticsearch.index.mode (e.g. time_series) and elasticsearch.index.codec (e.g. default, best_compression, none), sourced from the index /_settings API.
Extend the existing index-settings flow (GetIndexSettings + addIndexSettings) with filter_path entries for *.settings.index.mode and *.settings.index.codec, and populate the new fields in addIndexSettings only when present.
Add mode and codec to the Index struct (omitempty), to the index metricset fields.yml (keyword), to the sample data.json, and to CHANGELOG.asciidoc.

WHY:

Enables monitoring and filtering of index mode (e.g. time-series vs standard) and compression codec from Metricbeat, reusing the same _settings-based scaffolding used for tier_preference, creation_date, and version (PR Additional stats fields for Elasticsearch #41944). Both fields are optional so missing settings do not break event collection.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. The change only adds optional fields. Existing dashboards and queries are unchanged; new fields appear in events when the Elasticsearch index exposes index.mode or index.codec in /_settings.

Author's Checklist

Verified new fields are populated when index has mode/codec in _settings
Verified behaviour unchanged when mode/codec are missing (fields omitted, no error)
Regenerate field docs with mage FieldsDocs (or make update) if required by project workflow

How to test this PR locally

Build and run Metricbeat with the Elasticsearch module and index metricset enabled against a cluster that has indices (some with index.mode or index.codec set if possible).
Ingest events and confirm elasticsearch.index.mode and/or elasticsearch.index.codec appear when the index settings are present.
Confirm events for indices without these settings still publish and do not include the new fields (or include them empty/omitted).

Related issues

Relates to the approach in Additional stats fields for Elasticsearch #41944 (tier_preference, creation_date, version).

Use cases

As an operator, I can filter or aggregate Metricbeat index metrics by index mode (e.g. time_series) or codec (e.g. best_compression) in Kibana or downstream systems.
As an operator, I can see index mode and codec in the same index metricset events as tier_preference, creation_date, and version, without changing existing behaviour when those settings are absent.

Screenshots

N/A

Logs

N/A

…in Metricbeat. Update index settings retrieval and mapping to include these new fields.

github-actions · 2026-03-03T22:53:29Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2026-03-03T22:53:59Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @3kt? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

github-actions · 2026-03-03T22:54:52Z

✅ Vale Linting Results

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

github-actions · 2026-03-03T22:55:10Z

🔍 Preview links for changed docs

consulthys

LGT Stack Monitoring

coderabbitai · 2026-03-05T11:13:07Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1eb7cbcc-4e43-46e1-b05c-fa79c0a81851

📥 Commits

Reviewing files that changed from the base of the PR and between d98cf3b and a4e2c63.

📒 Files selected for processing (8)

CHANGELOG.asciidoc
changelog/fragments/1772635870-es-index-mode-&-codec-in-metricbeat.yaml
docs/reference/metricbeat/exported-fields-elasticsearch.md
docs/reference/metricbeat/metricbeat-metricset-elasticsearch-index.md
metricbeat/module/elasticsearch/fields.go
metricbeat/module/elasticsearch/index/_meta/data.json
metricbeat/module/elasticsearch/index/_meta/fields.yml
metricbeat/module/elasticsearch/index/data.go

📝 Walkthrough

Walkthrough

This pull request adds two new fields—mode and codec—to the Elasticsearch index metricset in Metricbeat. The changes include extending the Index struct in data.go with the new optional fields, updating the collection logic to fetch these values from Elasticsearch index settings, and adding corresponding schema definitions in fields.yml. Documentation files were updated with field definitions and example values (mode: "time_series", codec: "default"), and a changelog fragment was created to document this feature addition.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

🛠️ Update Documentation: Commit on current branch
🛠️ Update Documentation: Create PR

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-03-06T14:19:54Z

@Mergifyio backport 8.19 9.2 9.3

mergify · 2026-03-06T14:20:05Z

backport 8.19 9.2 9.3

✅ Backports have been created

Details

#49316 [8.19](backport #49237) ES index mode & codec in metricbeat has been created for branch 8.19 but encountered conflicts

Cherry-pick of 7dc72ea has failed:

On branch mergify/bp/8.19/pr-49237
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 7dc72eae2.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.asciidoc
	new file:   changelog/fragments/1772635870-es-index-mode-&-codec-in-metricbeat.yaml
	modified:   metricbeat/module/elasticsearch/fields.go
	modified:   metricbeat/module/elasticsearch/index/_meta/data.json
	modified:   metricbeat/module/elasticsearch/index/_meta/fields.yml
	modified:   metricbeat/module/elasticsearch/index/data.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   docs/reference/metricbeat/exported-fields-elasticsearch.md
	deleted by us:   docs/reference/metricbeat/metricbeat-metricset-elasticsearch-index.md

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

#49317 [9.2](backport #49237) ES index mode & codec in metricbeat has been created for branch 9.2 but encountered conflicts

Cherry-pick of 7dc72ea has failed:

On branch mergify/bp/9.2/pr-49237
Your branch is up to date with 'origin/9.2'.

You are currently cherry-picking commit 7dc72eae2.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1772635870-es-index-mode-&-codec-in-metricbeat.yaml
	modified:   docs/reference/metricbeat/exported-fields-elasticsearch.md
	modified:   docs/reference/metricbeat/metricbeat-metricset-elasticsearch-index.md
	modified:   metricbeat/module/elasticsearch/fields.go
	modified:   metricbeat/module/elasticsearch/index/_meta/data.json
	modified:   metricbeat/module/elasticsearch/index/_meta/fields.yml
	modified:   metricbeat/module/elasticsearch/index/data.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   CHANGELOG.asciidoc

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

#49318 [9.3](backport #49237) ES index mode & codec in metricbeat has been created for branch 9.3 but encountered conflicts

Cherry-pick of 7dc72ea has failed:

On branch mergify/bp/9.3/pr-49237
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit 7dc72eae2.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1772635870-es-index-mode-&-codec-in-metricbeat.yaml
	modified:   docs/reference/metricbeat/exported-fields-elasticsearch.md
	modified:   docs/reference/metricbeat/metricbeat-metricset-elasticsearch-index.md
	modified:   metricbeat/module/elasticsearch/fields.go
	modified:   metricbeat/module/elasticsearch/index/_meta/data.json
	modified:   metricbeat/module/elasticsearch/index/_meta/fields.yml
	modified:   metricbeat/module/elasticsearch/index/data.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   CHANGELOG.asciidoc

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

* Add `mode` and `codec` fields to the `elasticsearch.index` metricset in Metricbeat. Update index settings retrieval and mapping to include these new fields. * Add new exported fields for Elasticsearch in Metricbeat documentation * Make update * Added fragment (cherry picked from commit 7dc72ea) # Conflicts: # docs/reference/metricbeat/exported-fields-elasticsearch.md # docs/reference/metricbeat/metricbeat-metricset-elasticsearch-index.md

* Add `mode` and `codec` fields to the `elasticsearch.index` metricset in Metricbeat. Update index settings retrieval and mapping to include these new fields. * Add new exported fields for Elasticsearch in Metricbeat documentation * Make update * Added fragment (cherry picked from commit 7dc72ea) # Conflicts: # CHANGELOG.asciidoc

* Add `mode` and `codec` fields to the `elasticsearch.index` metricset in Metricbeat. Update index settings retrieval and mapping to include these new fields. * Add new exported fields for Elasticsearch in Metricbeat documentation * Make update * Added fragment (cherry picked from commit 7dc72ea) # Conflicts: # CHANGELOG.asciidoc Co-authored-by: Alexis Charveriat <alcharveriat@gmail.com>

* ES index mode & codec in metricbeat (#49237) * Add `mode` and `codec` fields to the `elasticsearch.index` metricset in Metricbeat. Update index settings retrieval and mapping to include these new fields. * Add new exported fields for Elasticsearch in Metricbeat documentation * Make update * Added fragment (cherry picked from commit 7dc72ea) # Conflicts: # CHANGELOG.asciidoc * Fix 9.3 backport: remove stray CHANGELOG.asciidoc from cherry-pick The cherry-pick from main brought over CHANGELOG.asciidoc which does not exist on the 9.3 branch. Remove it to fix the backport. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* ES index mode & codec in metricbeat (#49237) * Add `mode` and `codec` fields to the `elasticsearch.index` metricset in Metricbeat. Update index settings retrieval and mapping to include these new fields. * Add new exported fields for Elasticsearch in Metricbeat documentation * Make update * Added fragment (cherry picked from commit 7dc72ea) # Conflicts: # docs/reference/metricbeat/exported-fields-elasticsearch.md # docs/reference/metricbeat/metricbeat-metricset-elasticsearch-index.md * Fix 8.19 backport: remove markdown docs and regenerate asciidoc fields The cherry-pick from main brought over markdown doc files that don't exist on 8.19, which still uses asciidoc. Remove the erroneous markdown files and regenerate metricbeat/docs/fields.asciidoc to include the new elasticsearch.index.mode and elasticsearch.index.codec field entries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

3kt added 2 commits March 3, 2026 23:39

Add mode and codec fields to the elasticsearch.index metricset …

c0c78f4

…in Metricbeat. Update index settings retrieval and mapping to include these new fields.

Add new exported fields for Elasticsearch in Metricbeat documentation

a13d129

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 3, 2026

mergify bot assigned 3kt Mar 3, 2026

consulthys approved these changes Mar 4, 2026

View reviewed changes

Make update

c9b1811

consulthys added Feature:Stack Monitoring Team:Monitoring Stack Monitoring team labels Mar 4, 2026

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 4, 2026

consulthys added the backport-active-all Automated backport with mergify to all the active branches label Mar 4, 2026

3kt added 2 commits March 4, 2026 15:05

Merge branch 'main' into es-index-mode-codec

09f21cb

Added fragment

ff33232

3kt marked this pull request as ready for review March 5, 2026 11:09

3kt requested review from a team as code owners March 5, 2026 11:09

Merge branch 'main' into es-index-mode-codec

a4e2c63

theletterf approved these changes Mar 5, 2026

View reviewed changes

This was referenced Mar 5, 2026

Adding 'mode' and 'codec' fields to ES monitoring template elastic/elasticsearch#143673

Merged

Added codec and mode fields in Elasticsearch integration elastic/integrations#17686

Merged

consulthys merged commit 7dc72ea into elastic:main Mar 6, 2026
46 checks passed

mergify bot mentioned this pull request Mar 6, 2026

[8.19](backport #49237) ES index mode & codec in metricbeat #49316

Open

9 tasks

mergify bot mentioned this pull request Mar 6, 2026

[9.2](backport #49237) ES index mode & codec in metricbeat #49317

Open

9 tasks

mergify bot mentioned this pull request Mar 6, 2026

[9.3](backport #49237) ES index mode & codec in metricbeat #49318

Merged

9 tasks

This was referenced Mar 10, 2026

[8.19](backport #49237) ES index mode & codec in metricbeat #49384

Merged

[9.3](backport #49237) ES index mode & codec in metricbeat #49385

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ES index mode & codec in metricbeat#49237

ES index mode & codec in metricbeat#49237
consulthys merged 6 commits intoelastic:mainfrom
3kt:es-index-mode-codec

3kt commented Mar 3, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 3, 2026

Uh oh!

mergify bot commented Mar 3, 2026

Uh oh!

github-actions bot commented Mar 3, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 3, 2026 •

edited

Loading

Uh oh!

consulthys left a comment

Uh oh!

coderabbitai bot commented Mar 5, 2026

Walkthrough

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2026

Uh oh!

mergify bot commented Mar 6, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

3kt commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Disruptive User Impact

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

github-actions bot commented Mar 3, 2026

🤖 GitHub comments

Uh oh!

mergify bot commented Mar 3, 2026

Uh oh!

github-actions bot commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Vale Linting Results

Uh oh!

github-actions bot commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

consulthys left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot commented Mar 5, 2026

Walkthrough

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2026

Uh oh!

mergify bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

3kt commented Mar 3, 2026 •

edited

Loading

github-actions bot commented Mar 3, 2026 •

edited

Loading

github-actions bot commented Mar 3, 2026 •

edited

Loading

mergify bot commented Mar 6, 2026 •

edited

Loading