Add Azure Entra ID federated authentication for AWS via STS AssumeRoleWithWebIdentity

[agithomas]

• Enhancement

Proposed commit message

This change adds support for authenticating to AWS using Azure Entra ID (formerly Azure AD) as an OIDC identity provider, via the STS AssumeRoleWithWebIdentity API.

WHAT:

• New package x-pack/libbeat/common/azure/identity/ with a TokenProvider that obtains OAuth2 tokens from Azure Entra ID using the Azure SDK's ClientSecretCredential (with fallback to DefaultAzureCredential for managed identity and other implicit credential sources).
• TokenProvider implements the AWS SDK's stscreds.IdentityTokenRetriever interface, bridging Azure AD tokens into the AWS credential chain.
• New addAzureADWebIdentityCredentials function in x-pack/libbeat/common/aws/ wires the token provider into an STS WebIdentityRoleProvider with CredentialsCache support.
• ConfigAWS gains an azure_ad config block. When azure_ad.tenant_id and azure_ad.client_id are set alongside role_arn, the Azure AD federated flow is used instead of the standard AssumeRole flow.

WHY:
Customers operating in hybrid Azure/AWS environments need to authenticate Beats to AWS without managing long-lived AWS access keys. By federating through Azure Entra ID, temporary AWS credentials are obtained automatically using the organisation's existing identity infrastructure. This is implemented at the libbeat level so all Beats (Metricbeat, Filebeat, etc.) benefit without per-Beat changes.

Example configuration:

  azure_ad:
    tenant_id: "<azure-tenant-id>"
    client_id: "<azure-client-id>"
    client_secret: "<azure-client-secret>"
    scope: "api://<azure-client-id>/.default"
  role_arn: "arn:aws:iam::<account-id>:role/<oidc-role>"

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Prerequisites

• Go 1.25+ installed
• An Azure Entra ID (Azure AD) app registration with:
  • tenant_id, client_id, client_secret
  • An OAuth2 scope (e.g. api:///.default)
• An AWS IAM role configured with an OIDC trust policy for the Azure AD app
• AWS account with EC2 instances (or other resources) emitting CloudWatch metrics
• Elasticsearch running locally (e.g. via elastic-package stack up)

Steps

1. Build x-pack/metricbeat:

cd x-pack/metricbeat
go build -o metricbeat .

2. Create a test config (e.g. metricbeat.azure-ad-test.yml):

metricbeat.modules:
- module: aws
  period: 300s
  metricsets:
    - ec2
  regions:
    - us-east-1
  role_arn: "arn:aws:iam::<account-id>:role/<oidc-role>"
  default_region: "us-east-1"
  azure_ad:
    tenant_id: "<azure-tenant-id>"
    client_id: "<azure-client-id>"
    client_secret: "<azure-client-secret>"
    scope: "api://<azure-client-id>/.default"

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "elastic"
  password: "changeme"
  ssl.verification_mode: none

logging.level: debug
logging.to_stderr: true

3. Run metricbeat:

./metricbeat -e -c metricbeat.azure-ad-test.yml

4. Verify the authentication flow in debug logs -- look for:

"Switching credentials provider to Azure AD web identity"
"AWS Credentials belong to monitoring account ID: <account-id>"
"Collecting metrics from AWS region us-east-1"
"Collected number of metrics = ..."

5. Verify metrics landed in Elasticsearch:

curl -sk -u elastic:changeme 'https://localhost:9200/metricbeat-*/_count'
curl -sk -u elastic:changeme 'https://localhost:9200/metricbeat-*/_search?size=1&pretty' \
  -H 'Content-Type: application/json' \
  -d '{"query":{"match":{"metricset.name":"ec2"}}}'

Related issues

Use cases

Screenshots

Logs

[botelastic]

This pull request doesn't have a Team:<team> label.

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @agithomas? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[cla-checker-service]

💚 CLA has been signed