NO-JIRA: Ensure Platform Prometheus targets are protected #30014
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
openshift-ci
bot
added
the
do-not-merge/work-in-progress
juzhao
reviewed
Jul 24, 2025
juzhao
reviewed
Jul 24, 2025
juzhao
reviewed
Jul 24, 2025
Member
|
/cc |
hongkailiu
force-pushed
the
servicemonitor
branch
4 times, most recently
from
307a1cc to
1d33862
Compare
|
Job Failure Risk Analysis for sha: 1d33862
|
hongkailiu
force-pushed
the
servicemonitor
branch
from
1d33862 to
ef1d7b2
Compare
Member
Author
|
Some testing result: The failure is expected as https://issues.redhat.com/browse/OCPBUGS-57585 is not fixed yet.
$ git --no-pager log --pretty=oneline -1
ef1d7b2fb67c5db7aa441e270b5b6792f56112c0 (HEAD -> servicemonitor, hongkailiu/servicemonitor) Ensure ServiceMonitor's endpoints are protected
$ make WHAT=cmd/openshift-tests
$ cat /tmp/osServicePrincipal.json
{}
$ COMPONENT_NAMESPACE=openshift-cluster-version KUBECONFIG=/Users/hongkliu/.kube/config AZURE_AUTH_LOCATION=/tmp/osServicePrincipal.json ./openshift-tests run-test "[sig-instrumentation][Late] OpenShift service monitors [apigroup:image.openshift.io] should not be accessible without authorization"
...
Running Suite: - /Users/hongkliu/repo/openshift/origin
=======================================================
Random Seed: 1753732342 - will randomize all specs
Will run 1 of 1 specs
------------------------------
[sig-instrumentation][Late] OpenShift service monitors [apigroup:image.openshift.io] should not be accessible without authorization
github.com/openshift/origin/test/extended/prometheus/prometheus.go:72
STEP: Creating a kubernetes client @ 07/28/25 15:52:23.353
I0728 15:52:23.353887 21800 discovery.go:214] Invalidating discovery information
STEP: verifying all service monitors are configured with authorization @ 07/28/25 15:52:23.426
I0728 15:52:23.468686 21800 prometheus.go:92] service monitor openshift-cluster-version/cluster-version-operator has authorization
STEP: verifying all targets returns 401 or 403 without authorization @ 07/28/25 15:52:23.468
I0728 15:52:24.250271 21800 builder.go:121] Running '/Users/hongkliu/bin/kubectl --server=https://api.ci-ln-j1glv1b-1d09d.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/Users/hongkliu/.kube/config --namespace=openshift-monitoring exec prometheus-k8s-0 -- /bin/sh -x -c curl -k -s -o /dev/null -w '%{http_code}' "https://10.0.0.5:9099/metrics"'
I0728 15:52:25.138655 21800 builder.go:146] stderr: "+ curl -k -s -o /dev/null -w '%{http_code}' https://10.0.0.5:9099/metrics\n"
I0728 15:52:25.138757 21800 builder.go:147] stdout: "200"
[FAILED] in [It] - github.com/openshift/origin/test/extended/prometheus/prometheus.go:119 @ 07/28/25 15:52:25.139
• [FAILED] [1.799 seconds]
[sig-instrumentation][Late] OpenShift service monitors [apigroup:image.openshift.io] [It] should not be accessible without authorization
github.com/openshift/origin/test/extended/prometheus/prometheus.go:72
[FAILED] Expected
<[]error | len:1, cap:1>: [
<*fmt.wrapError | 0x1400789e0c0>{
msg: "the scaple url https://10.0.0.5:9099/metrics for namespace openshift-cluster-version is accessible without authorization: last response from server was not in [401 403]: 200",
err: <*errors.errorString | 0x140079f81b0>{
s: "last response from server was not in [401 403]: 200",
},
},
]
to be empty
In [It] at: github.com/openshift/origin/test/extended/prometheus/prometheus.go:119 @ 07/28/25 15:52:25.139
------------------------------
Summarizing 1 Failure:
[FAIL] [sig-instrumentation][Late] OpenShift service monitors [apigroup:image.openshift.io] [It] should not be accessible without authorization
github.com/openshift/origin/test/extended/prometheus/prometheus.go:119
Ran 1 of 1 Specs in 1.799 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
[
{
"name": "[sig-instrumentation][Late] OpenShift service monitors [apigroup:image.openshift.io] should not be accessible without authorization",
"lifecycle": "blocking",
"duration": 1798,
"startTime": "2025-07-28 19:52:23.342847 UTC",
"endTime": "2025-07-28 19:52:25.141598 UTC",
"result": "failed",
"output": " STEP: Creating a kubernetes client @ 07/28/25 15:52:23.353\n STEP: verifying all service monitors are configured with authorization @ 07/28/25 15:52:23.426\nI0728 15:52:23.468686 21800 prometheus.go:92] service monitor openshift-cluster-version/cluster-version-operator has authorization\n STEP: verifying all targets returns 401 or 403 without authorization @ 07/28/25 15:52:23.468\nI0728 15:52:24.250271 21800 builder.go:121] Running '/Users/hongkliu/bin/kubectl --server=https://api.ci-ln-j1glv1b-1d09d.ci.azure.devcluster.openshift.com:6443 --kubeconfig=/Users/hongkliu/.kube/config --namespace=openshift-monitoring exec prometheus-k8s-0 -- /bin/sh -x -c curl -k -s -o /dev/null -w '%{http_code}' \"https://10.0.0.5:9099/metrics\"'\nI0728 15:52:25.138655 21800 builder.go:146] stderr: \"+ curl -k -s -o /dev/null -w '%{http_code}' https://10.0.0.5:9099/metrics\\n\"\nI0728 15:52:25.138757 21800 builder.go:147] stdout: \"200\"\n [FAILED] in [It] - github.com/openshift/origin/test/extended/prometheus/prometheus.go:119 @ 07/28/25 15:52:25.139\n",
"error": "fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:119]: Expected\n \u003c[]error | len:1, cap:1\u003e: [\n \u003c*fmt.wrapError | 0x1400789e0c0\u003e{\n msg: \"the scaple url https://10.0.0.5:9099/metrics for namespace openshift-cluster-version is accessible without authorization: last response from server was not in [401 403]: 200\",\n err: \u003c*errors.errorString | 0x140079f81b0\u003e{\n s: \"last response from server was not in [401 403]: 200\",\n },\n },\n ]\nto be empty"
}
]Error: 1 tests failed
error: 1 tests failed |
hongkailiu
changed the title
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
Contributor
|
FYI: see bugs for endpoints without authorization(except CVO) in https://issues.redhat.com/browse/MON-4304 |
Contributor
|
/retest-required |
|
Job Failure Risk Analysis for sha: ef1d7b2
|
Contributor
|
/test verify |
|
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: b105e70
New tests seen in this PR at sha: b105e70
|
hongkailiu
force-pushed
the
servicemonitor
branch
2 times, most recently
from
c8ea5a6 to
ba8f912
Compare
|
Job Failure Risk Analysis for sha: ba8f912
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: ba8f912
New tests seen in this PR at sha: ba8f912
|
Member
Author
|
Interesting, I need to figure out how these are protected: They (checked the first two) returned 403. My current guess:
They can be protected without any configuration on the service monitor? Edit:
|
hongkailiu
force-pushed
the
servicemonitor
branch
3 times, most recently
from
f68c4ec to
686823e
Compare
Contributor
|
/lgtm |
Member
Author
|
Thanks for helping me make the pull in a better shape. We have to find a team lead to approve this. 🤔 |
Member
Author
|
/retest-required |
|
Job Failure Risk Analysis for sha: de9d258
Risk analysis has seen new tests most likely introduced by this PR. New tests seen in this PR at sha: de9d258
|
Member
Author
|
/test e2e-aws-ovn-fips |
|
Risk analysis has seen new tests most likely introduced by this PR. New tests seen in this PR at sha: de9d258
|
1 similar comment
|
Risk analysis has seen new tests most likely introduced by this PR. New tests seen in this PR at sha: de9d258
|
hongkailiu
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@hongkailiu: This pull request explicitly references no jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Co-authored-by: Simon Pasquier <[email protected]>
Co-authored-by: Simon Pasquier <[email protected]>
Co-authored-by: Simon Pasquier <[email protected]>
Contributor
|
/lgtm |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hongkailiu, machine424, simonpasquier The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
1 similar comment
Contributor
|
@hongkailiu: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
openshift-merge-bot
bot
merged commit 69329e3
into
openshift:main
23 of 43 checks passed
Contributor
|
[ART PR BUILD NOTIFIER] Distgit: openshift-enterprise-tests |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The new test checks (only for OpenShift components)
if each service monitor has authorization configuration, andEither of the above is not satisfied leads to failure of the test.
Moreover, The env. var.
MONITORING_AUTH_TEST_NAMESPACEcan be used to focus onvalidating the resources from a single namespace.