Skip to content

Custom Kube State Metric: Component Level Metric - #13554

Draft
martysp21 wants to merge 1 commit into
redhat-appstudio:mainfrom
martysp21:ksm-export-component-metrics-stage
Draft

Custom Kube State Metric: Component Level Metric#13554
martysp21 wants to merge 1 commit into
redhat-appstudio:mainfrom
martysp21:ksm-export-component-metrics-stage

Conversation

@martysp21

Copy link
Copy Markdown
Contributor

Add a new metric exposing component level metrics.
This is untracked POC only.

@openshift-ci
openshift-ci Bot requested review from ci-operator and raks-tt August 17, 2026 11:54
@openshift-ci

openshift-ci Bot commented Aug 17, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: martysp21

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

Copy link
Copy Markdown
Contributor

Kustomize Render Diff

Comparing 5e662778030eeb07d6

Component Environment Changes
components/monitoring/custom-kube-state-metrics/staging development +29 -5
components/monitoring/custom-kube-state-metrics/staging staging +29 -5

Total: 2 components, +58 -10 lines

📋 Full diff available in the workflow summary and as a downloadable artifact.

@qodo-for-redhat-appstudio

Copy link
Copy Markdown

PR Summary by Qodo

Expose component-level PipelineRun metrics via custom kube-state-metrics

✨ Enhancement ⚙️ Configuration changes 🕐 10-20 Minutes

Grey Divider

AI Description

• Add an info-style PipelineRun metric grouped by application/component and PAC event type.
• Keep hanging deletion metric high-signal by scoping high-cardinality labels to that metric only.
• Normalize PAC test/build event labels via ServiceMonitor metric relabeling.
Diagram

graph TD
  A["Tekton PipelineRun CRs"] --> B["custom-resource-state-config.yaml"] --> C["KSM: custom resource state"] --> D["konflux_pipelinerun_deletion_timestamp_seconds"]
  C --> E["konflux_pipelinerun_info"] --> F["ServiceMonitor relabeling"] --> G["Prometheus"]
  subgraph Legend
    direction LR
    _cr[(Kubernetes CR)] ~~~ _cfg["Config (YAML)"] ~~~ _svc(["Service"]) ~~~ _met["Metric"]
  end
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Prometheus recording rules for component-level rollups
  • ➕ Avoids adding/maintaining new kube-state-metrics mappings
  • ➕ Can enforce consistent label sets and aggregation semantics centrally
  • ➖ Requires querying/aggregation at rule-eval time (less discoverable than a raw info metric)
  • ➖ Harder to preserve “live object exists” semantics without careful rules
2. Expose app/component labels on existing deletion metric only
  • ➕ No new metric name to document/maintain
  • ➕ Keeps metric set smaller
  • ➖ Mixes “object identity / stuck deletion” and “inventory” concerns
  • ➖ Risk of higher cardinality if pipelinerun identity labels leak into inventory use-cases
3. Standardize labels at source (PipelineRun labels) instead of relabeling
  • ➕ Less scrape-time complexity; metrics reflect canonical labels
  • ➕ Avoids relabeling drift across environments
  • ➖ Requires changes in controllers/labeling policies; slower iteration for a POC
  • ➖ May not be feasible for third-party or legacy label keys

Recommendation: The PR’s approach is a good fit for a POC: introduce a dedicated low-cardinality info metric for app/component grouping while keeping high-cardinality identifiers (pipelinerun name, finalizers) constrained to the hanging-deletion metric. The ServiceMonitor relabeling is a pragmatic bridge to unify PAC test/build event-type labels without forcing upstream label migrations; just ensure the relabeling applies to the intended endpoint and doesn’t accidentally rewrite other metrics.

Files changed (3) +37 / -5

Enhancement (1) +21 / -5
custom-resource-state-config.yamlAdd PipelineRun info metric with app/component and PAC label dimensions +21/-5

Add PipelineRun info metric with app/component and PAC label dimensions

• Refactors PipelineRun metric config so pipelinerun/finalizers labels are attached only to the hanging-deletion metric. Adds a new info-style gauge metric (konflux_pipelinerun_info) labeled by application/component and pipeline/PAC event metadata to support component-level views with reduced cardinality.

components/monitoring/custom-kube-state-metrics/staging/custom-resource-state-config.yaml

Other (2) +16 / -0
kustomization.yamlPatch ServiceMonitor to apply scrape-time metric relabeling +4/-0

Patch ServiceMonitor to apply scrape-time metric relabeling

• Extends the staging kustomization to apply an additional JSON6902 patch targeting the custom-kube-state-metrics ServiceMonitor.

components/monitoring/custom-kube-state-metrics/staging/kustomization.yaml

servicemonitor-patch.yamlNormalize PAC event-type label and drop temporary label +12/-0

Normalize PAC event-type label and drop temporary label

• Adds metricRelabelings to copy pac_test_event_type into event_type when present, then drops pac_test_event_type. This makes test and build PipelineRuns share a single event_type label for querying.

components/monitoring/custom-kube-state-metrics/staging/servicemonitor-patch.yaml

@qodo-for-redhat-appstudio

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Duplicate series labelsets 🐞 Bug ≡ Correctness
Description
konflux_pipelinerun_info is emitted per PipelineRun but omits any unique per-run label, so multiple
PipelineRuns with the same application/component/type can produce identical label sets and collide
as the same Prometheus series. It also exports metadata.generation (not an info-style constant),
contradicting the help text and making sum()-based usage incorrect.
Code

components/monitoring/custom-kube-state-metrics/staging/custom-resource-state-config.yaml[R82-85]

+              labelsFromPath:
+                application: [metadata, labels, appstudio.openshift.io/application]
+                component: [metadata, labels, appstudio.openshift.io/component]
+                pipeline_type: [metadata, labels, pipelines.appstudio.openshift.io/type]
Relevance

●●● Strong

Accepted PipelineRun metric reviews favor correcting help, labels, and gauge semantics; this is a
direct correctness issue.

PR-#11933

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The repo’s custom-resource-state template shows metrics are defined under an each: stanza (per
object). The new konflux_pipelinerun_info metric is under the PipelineRun resource but its labels
exclude any per-PipelineRun identifier, and its value path is metadata.generation despite the help
claiming an info-style ~1 gauge.

components/monitoring/custom-kube-state-metrics/staging/custom-resource-state-config.yaml[13-28]
components/monitoring/custom-kube-state-metrics/staging/custom-resource-state-config.yaml[62-92]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`konflux_pipelinerun_info` is defined under a `PipelineRun` resource with an `each` stanza (per object), but its labels do not include a unique identifier (e.g., `metadata.name` or `metadata.uid`). That means multiple PipelineRuns can map to the exact same `{application,component,pipeline_type,event_type}` labelset.

This can cause collisions/duplicate-series in the scrape and makes the exported value misleading because it uses `metadata.generation` while the help text claims an info-style `~1` gauge.

## Issue Context
The config template in this repo explicitly models metrics as per-object via `each:`. For a “component-level” view, it’s safer to export a uniquely-keyed per-PipelineRun series and do component grouping in PromQL/recording rules, rather than trying to collapse uniqueness at exposition time.

## Fix Focus Areas
- components/monitoring/custom-kube-state-metrics/staging/custom-resource-state-config.yaml[80-92]

### Suggested change
- Add a unique label to `konflux_pipelinerun_info`, e.g.:
 - `pipelinerun: [metadata, name]` (or `pipelinerun_uid: [metadata, uid]`) to guarantee uniqueness.
- Make the value truly “info-style” by switching to `each.type: Info` if supported by your kube-state-metrics deployment, or adjust the help text to match the actual semantics (and avoid describing it as `~1` if you keep `generation`).
- Document/expect usage via `count by (application, component, pipeline_type, event_type) (konflux_pipelinerun_info)` (or a recording rule) for the component-level rollup.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. pipeline_type label dropped 🐞 Bug ◔ Observability
Description
The new konflux_pipelinerun_info adds a pipeline_type label, but the staging remote-write LabelKeep
allowlist regex does not include pipeline_type, so it will be stripped during
remote-write/federation. This breaks the metric’s intended grouping dimension and can collapse
distinct series downstream.
Code

components/monitoring/custom-kube-state-metrics/staging/custom-resource-state-config.yaml[R85-86]

+                pipeline_type: [metadata, labels, pipelines.appstudio.openshift.io/type]
+                event_type: [metadata, labels, pipelinesascode.tekton.dev/event-type]
Relevance

●●● Strong

Repository history accepts updating federation allowlists when new metric labels would otherwise be
dropped.

PR-#12920
PR-#12937

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The metric explicitly emits a label named pipeline_type. The staging remote-write LabelKeep
regex enumerates kept labels and does not include pipeline_type, so it will be removed.

components/monitoring/custom-kube-state-metrics/staging/custom-resource-state-config.yaml[80-87]
components/monitoring/prometheus/staging/base/federation/writeRelabelConfigs.yaml[31-51]
PR-#12920

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`konflux_pipelinerun_info` introduces a new Prometheus label named `pipeline_type`, but the staging federation/remote-write LabelKeep allowlist doesn’t include `pipeline_type`. As a result, remote-written data will lose this dimension.

## Issue Context
The allowlist is implemented as a `LabelKeep` regex; any label name not matched is dropped. The allowlist already keeps `event_type` but not `pipeline_type`.

## Fix Focus Areas
- components/monitoring/prometheus/staging/base/federation/writeRelabelConfigs.yaml[31-51]

### Suggested change
- Add `pipeline_type` to the allowlist regex (and do the same in production if this metric will be promoted), or rename the emitted label to an already-allowlisted label name if that’s the established convention.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

3. Endpoint index patch fragile 🐞 Bug ⚙ Maintainability
Description
The ServiceMonitor patch targets /spec/endpoints/0, so a future reorder/insert of endpoints in the
base ServiceMonitor can silently apply these relabelings to the wrong endpoint. This is a
maintainability reliability risk because the patch encodes array position rather than
asserting/matching the intended endpoint (port: metrics).
Code

components/monitoring/custom-kube-state-metrics/staging/servicemonitor-patch.yaml[R2-4]

+- op: add
+  path: /spec/endpoints/0/metricRelabelings
+  value:
Relevance

●●● Strong

A closely matching accepted precedent recommends guarding JSON patches against fragile array-index
targeting.

PR-#11344

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The base ServiceMonitor defines endpoints as an array with metrics at index 0 and telemetry at
index 1. The patch modifies endpoints[0] without any assertion, which makes it vulnerable to future
base changes.

components/monitoring/custom-kube-state-metrics/staging/servicemonitor-patch.yaml[1-4]
components/monitoring/custom-kube-state-metrics/base/servicemonitor.yaml[14-22]
PR-#11344

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The JSON6902 patch hard-codes `/spec/endpoints/0`, which assumes the `metrics` endpoint remains at index 0 forever. If the base ServiceMonitor changes endpoint ordering, the patch may apply to the wrong endpoint.

## Issue Context
The base ServiceMonitor currently defines two endpoints (`metrics` then `telemetry`). The patch should at least fail fast if that ordering changes, or be rewritten to target the endpoint more robustly.

## Fix Focus Areas
- components/monitoring/custom-kube-state-metrics/staging/servicemonitor-patch.yaml[1-12]
- components/monitoring/custom-kube-state-metrics/base/servicemonitor.yaml[10-22]

### Suggested change
- Add a `test` op before the `add` op to assert you are patching the intended endpoint, e.g.:
 - `op: test`, `path: /spec/endpoints/0/port`, `value: metrics`
- (Optional) Consider a patch approach that matches on the endpoint identity instead of array index if feasible in your kustomize setup.

Note: a `test` op won’t make the patch resilient to reordering, but it will prevent silent misapplication by failing the build/apply when the base changes.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context
✅ Compliance rules (platform): 3 rules

Grey Divider

Tip of the day
💡 Did you know, you can route each action level your way: inline, summary, both, or drop

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment on lines +82 to +85
labelsFromPath:
application: [metadata, labels, appstudio.openshift.io/application]
component: [metadata, labels, appstudio.openshift.io/component]
pipeline_type: [metadata, labels, pipelines.appstudio.openshift.io/type]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Duplicate series labelsets 🐞 Bug ≡ Correctness

konflux_pipelinerun_info is emitted per PipelineRun but omits any unique per-run label, so multiple
PipelineRuns with the same application/component/type can produce identical label sets and collide
as the same Prometheus series. It also exports metadata.generation (not an info-style constant),
contradicting the help text and making sum()-based usage incorrect.
Agent Prompt
## Issue description
`konflux_pipelinerun_info` is defined under a `PipelineRun` resource with an `each` stanza (per object), but its labels do not include a unique identifier (e.g., `metadata.name` or `metadata.uid`). That means multiple PipelineRuns can map to the exact same `{application,component,pipeline_type,event_type}` labelset.

This can cause collisions/duplicate-series in the scrape and makes the exported value misleading because it uses `metadata.generation` while the help text claims an info-style `~1` gauge.

## Issue Context
The config template in this repo explicitly models metrics as per-object via `each:`. For a “component-level” view, it’s safer to export a uniquely-keyed per-PipelineRun series and do component grouping in PromQL/recording rules, rather than trying to collapse uniqueness at exposition time.

## Fix Focus Areas
- components/monitoring/custom-kube-state-metrics/staging/custom-resource-state-config.yaml[80-92]

### Suggested change
- Add a unique label to `konflux_pipelinerun_info`, e.g.:
  - `pipelinerun: [metadata, name]` (or `pipelinerun_uid: [metadata, uid]`) to guarantee uniqueness.
- Make the value truly “info-style” by switching to `each.type: Info` if supported by your kube-state-metrics deployment, or adjust the help text to match the actual semantics (and avoid describing it as `~1` if you keep `generation`).
- Document/expect usage via `count by (application, component, pipeline_type, event_type) (konflux_pipelinerun_info)` (or a recording rule) for the component-level rollup.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment on lines +85 to +86
pipeline_type: [metadata, labels, pipelines.appstudio.openshift.io/type]
event_type: [metadata, labels, pipelinesascode.tekton.dev/event-type]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. Pipeline_type label dropped 🐞 Bug ◔ Observability

The new konflux_pipelinerun_info adds a pipeline_type label, but the staging remote-write LabelKeep
allowlist regex does not include pipeline_type, so it will be stripped during
remote-write/federation. This breaks the metric’s intended grouping dimension and can collapse
distinct series downstream.
Agent Prompt
## Issue description
`konflux_pipelinerun_info` introduces a new Prometheus label named `pipeline_type`, but the staging federation/remote-write LabelKeep allowlist doesn’t include `pipeline_type`. As a result, remote-written data will lose this dimension.

## Issue Context
The allowlist is implemented as a `LabelKeep` regex; any label name not matched is dropped. The allowlist already keeps `event_type` but not `pipeline_type`.

## Fix Focus Areas
- components/monitoring/prometheus/staging/base/federation/writeRelabelConfigs.yaml[31-51]

### Suggested change
- Add `pipeline_type` to the allowlist regex (and do the same in production if this metric will be promoted), or rename the emitted label to an already-allowlisted label name if that’s the established convention.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment on lines +2 to +4
- op: add
path: /spec/endpoints/0/metricRelabelings
value:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Informational

3. Endpoint index patch fragile 🐞 Bug ⚙ Maintainability

The ServiceMonitor patch targets /spec/endpoints/0, so a future reorder/insert of endpoints in the
base ServiceMonitor can silently apply these relabelings to the wrong endpoint. This is a
maintainability reliability risk because the patch encodes array position rather than
asserting/matching the intended endpoint (port: metrics).
Agent Prompt
## Issue description
The JSON6902 patch hard-codes `/spec/endpoints/0`, which assumes the `metrics` endpoint remains at index 0 forever. If the base ServiceMonitor changes endpoint ordering, the patch may apply to the wrong endpoint.

## Issue Context
The base ServiceMonitor currently defines two endpoints (`metrics` then `telemetry`). The patch should at least fail fast if that ordering changes, or be rewritten to target the endpoint more robustly.

## Fix Focus Areas
- components/monitoring/custom-kube-state-metrics/staging/servicemonitor-patch.yaml[1-12]
- components/monitoring/custom-kube-state-metrics/base/servicemonitor.yaml[10-22]

### Suggested change
- Add a `test` op before the `add` op to assert you are patching the intended endpoint, e.g.:
  - `op: test`, `path: /spec/endpoints/0/port`, `value: metrics`
- (Optional) Consider a patch approach that matches on the endpoint identity instead of array index if feasible in your kustomize setup.

Note: a `test` op won’t make the patch resilient to reordering, but it will prevent silent misapplication by failing the build/apply when the base changes.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.38%. Comparing base (50fdd86) to head (5d5fe7a).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main   #13554   +/-   ##
=======================================
  Coverage   60.38%   60.38%           
=======================================
  Files          24       24           
  Lines        1628     1628           
=======================================
  Hits          983      983           
  Misses        563      563           
  Partials       82       82           
Flag Coverage Δ
go 60.38% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@martysp21
martysp21 marked this pull request as draft August 17, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant