Skip to content

Metrics tab of MSK broker results in a 404 error instead of metrics gathered from open monitoring endpoint #1630

@hogarth-sv

Description

@hogarth-sv

Issue submitter TODO list

  • I've looked up my issue in FAQ
  • I've searched for an already existing issues here
  • I've tried running main-labeled docker image and the issue still persists there
  • I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

MSK cluster configured with Open Monitoring enabled.

Kafbat deployed in an EKS cluster with a config that works to see brokers, topics, consumers, etc etc

From the pod I can use wget and see the prometheus data from http://brokernode:11001/metrics

The logs from kafbat do not show any errors getting metrics:

2026-01-09T19:08:51.801Z DEBUG 1 --- [or-http-epoll-6] i.k.u.s.ClustersStatisticsScheduler      : Metrics updated for cluster: cloud
2026-01-09T19:09:07.124Z DEBUG 1 --- [     parallel-8] i.k.u.s.m.s.i.InferredMetricsScraper     : 16 metric families inferred from cluster state
2026-01-09T19:09:07.124Z DEBUG 1 --- [     parallel-8] i.k.u.s.ClustersStatisticsScheduler      : Metrics updated for cluster: onprem
2026-01-09T19:09:18.878Z DEBUG 1 --- [    parallel-13] i.k.u.s.ClustersStatisticsScheduler      : Start getting metrics for kafkaCluster: cloud
2026-01-09T19:09:18.878Z DEBUG 1 --- [    parallel-14] i.k.u.s.ClustersStatisticsScheduler      : Start getting metrics for kafkaCluster: onprem
2026-01-09T19:09:19.159Z DEBUG 1 --- [    parallel-16] i.k.u.s.m.s.i.InferredMetricsScraper     : 18 metric families inferred from cluster state
2026-01-09T19:09:19.160Z DEBUG 1 --- [    parallel-16] i.k.u.s.m.s.p.PrometheusMetricsRetriever : Retrieving metrics from prometheus endpoint: broker-1:11001
2026-01-09T19:09:19.160Z DEBUG 1 --- [    parallel-16] i.k.u.s.m.s.p.PrometheusMetricsRetriever : Retrieving metrics from prometheus endpoint: broker-2:11001
2026-01-09T19:09:19.160Z DEBUG 1 --- [    parallel-16] i.k.u.s.m.s.p.PrometheusMetricsRetriever : Retrieving metrics from prometheus endpoint: broker-3:11001
2026-01-09T19:09:22.556Z DEBUG 1 --- [or-http-epoll-7] i.k.u.s.ClustersStatisticsScheduler      : Metrics updated for cluster: cloud
2026-01-09T19:09:36.133Z DEBUG 1 --- [     parallel-6] i.k.u.s.m.s.i.InferredMetricsScraper     : 16 metric families inferred from cluster state
2026-01-09T19:09:36.134Z DEBUG 1 --- [     parallel-6] i.k.u.s.ClustersStatisticsScheduler      : Metrics updated for cluster: onprem

Following the UI through to a broker and clicking on the metrics tab just results in a 404 being raised.

Is this meant to actually work to expose broker metrics, or is this a WIP known issue with it being incomplete at this time ... or do I have configuration incorrect?

Configuration for the MSK cluster for metrics is:

      metrics:
        type: PROMETHEUS
        port: 11001
        ssl: false

Expected behavior

Metrics tab of broker details actually displays metrics gathered from the broker.

Your installation details

Tested with both latest (dfa5a7e v1.4.2) and with main (db7cc80 09/01/2026, 11:17:25)

MSK is running 3.7.x with SASL_SCRAM (metrics endpoint tested unauthenticated and with plain http via wget).

Steps to reproduce

  1. Setup an MSK cluster with Open Monitoring enabled with the JMX Broker stats enabled and security group allowing access to 11001
  2. Verify that prometheus compatible data is being exported via wget -O - http://brokernode:11001/metrics
  3. Configure kafbat to gather prometheus metrics from the broker nodes
  4. Navigate to a broker node and click on the metrics tab

Screenshots

No response

Logs

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions