PVO11Y-5445 Exclude mintmaker from kube_pod_* federation filters in staging

[kubasikus]

What

Exclude the mintmaker namespace from broad kube_pod_* federation match filters in staging.

Metrics excluded:

• kube_pod_container_status_waiting_reason
• kube_pod_container_status_restarts_total
• kube_pod_container_info
• kube_pod_status_unschedulable
• kube_pod_status_phase
• kube_pod_container_status_terminated_reason

Clusters affected: staging

PVO11Y-5445

Why

MintMaker runs high-volume, short-lived Renovate bot pods that generate significant series churn for kube_pod_* metrics forwarded to RHOBS. No alerts or dashboards consume these pod-level metrics for the mintmaker namespace — most pod alerts (PodNotReady, PodOOMKilled, ControllerPodOOMKilled) already explicitly exclude it.

Note: Two alerts — UnschedulablePods and CrashLoopBackOff — do not currently exclude mintmaker. Both are severity: warning routed to the mintmaker team via alert_routing_key. This change will prevent those alerts from firing for mintmaker pods. MintMaker has its own dedicated controller availability alerts, so the impact should be minimal, but the mintmaker team should be aware.

Validation

• kustomize build --enable-helm passes for staging federation overlay

[github-actions]

Kustomize Render Diff

Comparing 29b08a141 → 8eb770311

Component	Environment	Changes
components/monitoring/prometheus/staging/base	staging	+12 -6
components/monitoring/prometheus/staging/kflux-stg-es01	staging	+12 -6
components/monitoring/prometheus/staging/lightwell-dev	staging	+12 -6
components/monitoring/prometheus/staging/stone-stage-p01	staging	+12 -6
components/monitoring/prometheus/staging/stone-stg-rh01	staging	+12 -6

Total: 5 components, +60 -30 lines

📋 Full diff available in the workflow summary and as a downloadable artifact.

[qodo-for-redhat-appstudio]

PR Summary by Qodo

Exclude mintmaker from kube_pod_* federation filters in staging

⚙️ Configuration changes ✨ Enhancement 🕐 10-20 Minutes

AI Description

• Exclude the mintmaker namespace from broad kube_pod_* federation selectors in staging.
• Reduce RHOBS series churn caused by high-volume, short-lived Renovate bot pods.
• Keep existing allowlists intact while narrowing only the broad namespace regex matches.

Diagram

graph TD
  A["Staging cluster"] --> B["Prometheus federation"] --> C["endpoints-params.yaml"] --> D["kube_pod_* match filters"] --> E["Exclude mintmaker"] --> F{{"RHOBS"}}

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Upstream drop via scrape relabeling

• ➕ Reduces ingestion/processing earlier in the pipeline (not just federation output)
• ➕ Keeps federation rules simpler over time
• ➖ Requires changes in scrape configuration and tighter coordination with cluster monitoring owners
• ➖ Higher blast radius if relabeling affects non-federated consumers

2. Switch broad kube_pod_* selectors to explicit allowlists

• ➕ More predictable cardinality; avoids future churn from new namespaces
• ➕ Easier to reason about what is intentionally forwarded
• ➖ Ongoing maintenance burden as new namespaces legitimately need pod-level metrics
• ➖ Higher risk of accidentally omitting required metrics for new workloads

Recommendation: The current approach (exclude mintmaker via the existing broad namespace regex in staging federation) is the lowest-risk, fastest way to stop the known churn source without impacting other namespaces. Consider upstream relabeling or allowlists only if churn recurs from additional namespaces or if you want stronger long-term cardinality control.

Files changed (1) +6 / -6

Other (1) +6 / -6

endpoints-params.yamlExclude mintmaker from broad kube_pod_* federation namespace regexes +6/-6

Exclude mintmaker from broad kube_pod_* federation namespace regexes

• Updates staging federation match[] selectors for several kube_pod_* metrics to exclude the mintmaker namespace from broad namespace regex filters. Also removes mintmaker from the explicit allowlist regex for kube_pod_container_status_terminated_reason, aligning with the intent to drop pod-level mintmaker series from federation.

components/monitoring/prometheus/staging/base/federation/endpoints-params.yaml

[qodo-for-redhat-appstudio]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Tip of the day

💡 Did you know, you can route each action level your way: inline, summary, both, or drop

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous review results

Review updated until commit 77cfe89 ⚖️ Balanced

Results up to commit a6a815e ⚖️ Balanced

No changes from previous review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.38%. Comparing base (87c484f) to head (77cfe89).
⚠️ Report is 54 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13503      +/-   ##
==========================================
+ Coverage   58.78%   60.38%   +1.59%     
==========================================
  Files          24       24              
  Lines        1548     1628      +80     
==========================================
+ Hits          910      983      +73     
- Misses        559      563       +4     
- Partials       79       82       +3     

Flag	Coverage Δ
go	60.38% <ø> (+1.59%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[FernandesMF]

@kubasikus, thanks for bringing this up! I would like to propose a slight change to the current solution.

We do have two groups of pods in MintMaker, the ones relative to the core service, and those from the temporary workloads. While removing the second group from the metrics won't be a problem for us, we'd prefer to keep the metrics for the first group (which fortunately is much much smaller).

Since the temporary pods always have names starting with renovate, could we take a more targeted approach? The idea would be to keep the namespace-level exclusion you've added, but add a companion match[] selector for each metric that re-includes the mitnmaker namespace while filtering out the renovate pods by name:

Container Metrics:

• {__name__="kube_pod_container_status_waiting_reason", namespace="mintmaker", pod!~"renovate.*"}
• {__name__="kube_pod_container_status_restarts_total", namespace="mintmaker", pod!~"renovate.*"}
• {__name__="kube_pod_container_info", namespace="mintmaker", pod!~"renovate.*"}
• {__name__="kube_pod_container_status_terminated_reason", namespace="mintmaker", pod!~"renovate.*"}

Pod Metrics:

• {__name__="kube_pod_status_unschedulable", namespace="mintmaker", pod!~"renovate.*"}
• {__name__="kube_pod_status_phase", namespace="mintmaker", pod!~"renovate.*"}

Each of these would be placed right after the corresponding modified line. Since match[] entries are OR'd, this brings back only the non-renovate pods from our namespace, and the cardinatily would still be reduced greatly.

Do you think this would work, @kubasikus? I'll be happy to adjust the approach if needed!

Disclaimer 1: it's important to mention that part of the load that MintMaker causes is not directlty from it's own resources, but from the CIs that it triggers in the component repos after proposing updates.

Disclaimer 2: assisted by Claude Opus 4.6

[pacho-rh]

Including mintmaker but filtering out the renovate pods makes sense to me. Checking the metric in-cluster, doing so will be still be saving a ton on cardinality. renovate pods account for several thousands of series while non-renovate pod account for a few hundred series.

I tried updating this branch in include the additional match entries but unfortunately I do not have permission. We'll wait for @kubasikus to be back to update this PR.

[kubasikus]

Ran a query to see how much reduction will happen by filtering out the renovate pod series:

Looks like most of it, as @pacho-rh says. So let's do it this way :)

[qodo-for-redhat-appstudio]

Code review by qodo was updated up to the latest commit 77cfe89

[rh-rsaar]

Will this still monitor container_* metrics, e.g. about mem/cpu usage and container_cpu_cfs_throttled_periods_total and container_cpu_cfs_periods_total?

[kubasikus]

@rh-rsaar as mentioned in slack, yes, those metrics will still be available.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kubasikus, pumahaka

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• components/monitoring/OWNERS [kubasikus,pumahaka]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment