Skip to content

Conversation

@openshift-bot
Copy link
Contributor

@openshift-bot openshift-bot commented Dec 5, 2025

disruptions Information

There were (3680) added jobs and (686) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 297 jobs increased and 1010 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
host-to-host-new-connections 4.21 amd64 ovn metal compact 113 1h49m56.8s 34.93% 3h20m38.36s 59.17%
pod-to-host-reused-connections 4.21 amd64 ovn metal compact 113 1h5m57.8s 31.31% 1h59m42.04s 107.82%
openshift-api-http2-localhost-new-connections 4.21 amd64 ovn metal compact 113 53m53.6s 34.80% 1h24m42.48s 58.70%
kube-api-http1-localhost-new-connections 4.21 amd64 ovn metal compact 113 54m14.8s 32.42% 1h27m6.4s 55.88%
openshift-api-http2-service-network-new-connections 4.21 amd64 ovn metal compact 113 26m37.6s 51.24% 35m13.04s 54.74%
kube-api-http2-service-network-new-connections 4.21 amd64 ovn metal compact 113 25m32.6s 49.51% 35m31.4s 53.83%
kube-api-http1-service-network-new-connections 4.21 amd64 ovn metal compact 113 25m51.2s 45.28% 36m13.12s 53.50%
kube-api-http1-internal-lb-new-connections 4.21 amd64 ovn metal compact 113 24m41.2s 46.15% 37m34.16s 41.70%
openshift-api-http2-internal-lb-new-connections 4.21 amd64 ovn metal compact 113 23m45s 43.94% 36m27.28s 45.13%
openshift-api-http2-service-network-reused-connections 4.21 amd64 ovn metal compact 113 14m1s 89.63% 20m38.12s 46.14%
kube-api-http2-internal-lb-new-connections 4.21 amd64 ovn metal compact 113 22m56.4s 38.67% 36m14.12s 43.90%
kube-api-http1-service-network-reused-connections 4.21 amd64 ovn metal compact 113 14m33s 65.36% 20m43.52s 66.70%
kube-api-http2-internal-lb-reused-connections 4.21 amd64 ovn metal compact 113 13m27.4s 65.64% 18m36.64s 37.71%
kube-api-http2-service-network-reused-connections 4.21 amd64 ovn metal compact 113 13m13.2s 58.77% 20m34.36s 59.69%
kube-api-http1-internal-lb-reused-connections 4.21 amd64 ovn metal compact 113 13m6.4s 57.88% 18m43.68s 53.84%
openshift-api-http2-internal-lb-reused-connections 4.21 amd64 ovn metal compact 113 14m3s 48.93% 18m59.8s 44.26%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 411 3m37.5s 528.61% 8m17s 38.59%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 411 2m33.5s 529.10% 5m42.7s 47.31%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 509 3m9s 200.00% 4m26.68s 251.91%
kube-api-http1-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 509 3m7s 196.83% 4m25.2s 249.96%
azure-network-liveness-new-connections 4.21 arm64 ovn aws ha 196 2m4.5s 2493.75% 5m5s 48.17%
gcp-network-liveness-new-connections 4.21 arm64 ovn aws ha 196 1m38.5s 2636.11% 3m48s 77.10%
azure-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 116 1m49.75s 499.73% 4m31.45s 318.13%
gcp-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 116 1m4s 2185.71% 2m42.95s 196.17%
host-to-host-new-connections 4.21 4.20 amd64 ovn azure ha 1007 56s 5500.00% 2m25.94s 57.81%
gcp-network-liveness-new-connections 4.21 amd64 ovn rosa ha 108 54.25s 17983.33% 2m37.59s 200.74%
host-to-host-new-connections 4.21 4.21 amd64 ovn azure ha 474 43.05s 1335.00% 1m58.37s 57.70%
pod-to-host-new-connections 4.21 4.21 amd64 ovn azure ha 474 35s 1742.11% 1m36.67s 148.51%
azure-network-liveness-new-connections 4.21 amd64 ovn kubevirt external 117 27.4s 2092.00% 2m31.08s 1998.33%
gcp-network-liveness-new-connections 4.21 amd64 ovn kubevirt external 117 22.6s 8940.00% 1m33.72s 1049.94%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1017 1m4s 39.59% 2m0.84s 162.87%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1017 1m3s 40.47% 1m56s 157.95%
ingress-to-oauth-server-reused-connections 4.21 amd64 ovn metal compact 113 8s 75.82% 14.4s 131.51%
ingress-to-console-reused-connections 4.21 amd64 ovn metal compact 113 7.4s 62.64% 9.88s 58.84%
metrics-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1017 3s 200.00% 52.92s 194.00%
ingress-to-oauth-server-new-connections 4.21 amd64 ovn metal compact 113 3.4s 119.35% 7.88s 136.64%
host-to-host-new-connections 4.21 4.21 amd64 ovn aws ha 509 2s 135.29% 15.84s 1532.99%
cache-kube-api-new-connections 4.21 ppc64le ovn none ha 168 3s 39.53% 59.6s 46.29%
ingress-to-oauth-server-new-connections 4.21 ppc64le ovn none ha 168 2s 73.91% 1m2.28s 60.43%
kube-api-http1-service-network-reused-connections 4.21 4.20 amd64 ovn vsphere ha 106 1s 185.71% 2s 100.00%
metrics-api-new-connections 4.21 amd64 ovn metal compact 113 1.4s 40.00% 2s 38.89%
kube-api-reused-connections 4.21 amd64 ovn metal compact 113 1.4s 40.00% 2s 100.00%
cache-oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 904 0s 0.00% 32.97s 153.62%
openshift-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 509 1m52.2s 0.00% 4m10.2s 5016.56%
oauth-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1133 0s 0.00% 7.68s 53.60%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1017 2s 0.00% 23.84s 2284.00%
openshift-api-reused-connections 4.21 amd64 ovn metal compact 113 1s 0.00% 2s 100.00%
oauth-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1017 1s 0.00% 20.36s 1998.97%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1017 1m14s 0.00% 2m47.36s 130.52%
openshift-api-new-connections 4.21 4.21 amd64 ovn azure ha 474 0s 0.00% 43.08s 527.99%
oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 904 0s 0.00% 28s 64.71%
host-to-pod-new-connections 4.21 amd64 ovn metal compact 113 1m3.2s 0.00% 7m11.76s 73.36%
openshift-api-http2-localhost-reused-connections 4.21 amd64 ovn metal compact 113 26m25.8s 0.00% 53m16.12s 77.85%
oauth-api-new-connections 4.21 4.21 amd64 ovn azure ha 474 0s 0.00% 42.62s 2031.00%
openshift-api-http2-internal-lb-new-connections 4.21 4.20 amd64 ovn gcp ha 904 0s 0.00% 55.94s 143.22%
openshift-api-new-connections 4.21 4.20 amd64 ovn aws ha 1133 1s 0.00% 8s 38.65%
pod-to-host-reused-connections 4.21 4.20 amd64 ovn azure ha 1007 3s 0.00% 29s 222.22%
ingress-to-oauth-server-new-connections 4.21 amd64 ovn none single 156 1m25s 0.00% 1m25s 1197.71%
kube-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1133 0s 0.00% 5.68s 50.66%
host-to-host-reused-connections 4.21 amd64 ovn metal compact 113 58m33.8s 0.00% 2h10m7.36s 58.47%
cache-oauth-api-new-connections 4.21 ppc64le ovn none ha 168 2s 0.00% 1m0.94s 43.86%
ingress-to-oauth-server-new-connections 4.21 amd64 ovn rosa ha 108 0s 0.00% 1.86s 257.69%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1017 1m14s 0.00% 2m39.68s 151.03%
image-registry-new-connections 4.21 amd64 ovn rosa ha 108 0s 0.00% 930ms 257.69%
cache-openshift-api-new-connections 4.21 4.21 amd64 ovn azure ha 474 0s 0.00% 45.27s 1054.85%
ingress-to-console-new-connections 4.21 amd64 ovn kubevirt external 117 13.2s 0.00% 18s 31.87%
oauth-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1017 3.8s 0.00% 52s 44.77%
cache-kube-api-new-connections 4.21 4.21 amd64 ovn azure ha 474 0s 0.00% 48.27s 720.92%
kube-api-new-connections 4.21 4.20 amd64 ovn aws ha 1133 400ms 0.00% 6s 50.00%
metrics-api-new-connections 4.21 4.20 amd64 ovn gcp ha 904 850ms 0.00% 36s 140.00%
openshift-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1133 2s 0.00% 8.68s 44.67%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1007 33.7s 0.00% 2m21.94s 209.85%
metrics-api-new-connections 4.21 4.21 amd64 ovn aws ha 509 13s 0.00% 31.92s 3486.52%
kube-api-new-connections 4.21 ppc64le ovn none ha 168 2s 0.00% 1m0.94s 49.29%
kube-api-http2-localhost-reused-connections 4.21 amd64 ovn metal compact 113 26m21.6s 0.00% 54m23.84s 79.79%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1133 2m6.4s 0.00% 4m29s 422.63%
pod-to-pod-reused-connections 4.21 amd64 ovn metal compact 113 1m53.8s 0.00% 13m29.24s 252.83%
kube-api-http2-service-network-reused-connections 4.21 4.20 amd64 ovn vsphere ha 106 1s 0.00% 1s 85.19%
cache-kube-api-reused-connections 4.21 amd64 ovn metal compact 113 1s 0.00% 2s 80.18%
azure-network-liveness-new-connections 4.21 amd64 ovn rosa ha 108 1m23.35s 0.00% 3m5.62s 366.38%
kube-api-http2-localhost-new-connections 4.21 amd64 ovn metal compact 113 53m0.6s 0.00% 1h25m23.84s 57.77%
ingress-to-console-new-connections 4.21 ppc64le ovn none ha 168 1.65s 0.00% 1m0.94s 42.82%
metrics-api-reused-connections 4.21 amd64 ovn aws ha 270 0s 0.00% 1s 257.14%
host-to-host-new-connections 4.21 amd64 ovn kubevirt external 117 0s 0.00% 1s 53.85%
pod-to-service-new-connections 4.21 multi ovn aws ha 235 0s 0.00% 1s 150.00%
kube-api-http1-localhost-reused-connections 4.21 amd64 ovn metal compact 113 29m24.4s 0.00% 59m13.08s 78.37%
openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 904 1s 0.00% 28.97s 105.17%
cache-openshift-api-new-connections 4.21 ppc64le ovn none ha 168 2s 0.00% 58.93s 40.64%
gcp-network-liveness-new-connections 4.21 amd64 ovn none single 156 0s 0.00% 4s 30.29%
metrics-api-new-connections 4.21 4.21 amd64 ovn azure ha 474 0s 0.00% 36.35s 3535.00%
host-to-pod-reused-connections 4.21 amd64 ovn metal compact 113 47.6s 0.00% 7m28.12s 89.85%
ingress-to-oauth-server-reused-connections 4.21 4.21 amd64 ovn aws ha 509 5.6s 0.00% 54.92s 5392.00%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 509 25s 0.00% 40.68s 1042.70%
openshift-api-http2-service-network-reused-connections 4.21 4.20 amd64 ovn vsphere ha 106 1s 0.00% 2s 100.00%
openshift-api-http2-internal-lb-reused-connections 4.21 4.21 amd64 ovn gcp ha 1017 2.2s 0.00% 1m22s 839.29%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1133 1m22s 0.00% 2m52.84s 479.03%
openshift-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1017 2s 0.00% 1m36s 4700.00%
openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1017 1s 0.00% 28.88s 882.31%
azure-network-liveness-new-connections 4.21 amd64 ovn none single 156 0s 0.00% 6.9s 30.68%
pod-to-service-new-connections 4.21 amd64 ovn metal compact 113 0s 0.00% 3.52s 3100.00%
pod-to-service-new-connections 4.21 4.21 amd64 ovn aws ha 509 9.6s 0.00% 34.92s 3392.00%
cache-openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 904 0s 0.00% 29.94s 120.80%
host-to-host-reused-connections 4.21 4.20 amd64 ovn azure ha 1007 2s 0.00% 43s 196.96%
ingress-to-oauth-server-reused-connections 4.21 4.20 amd64 ovn aws ha 1133 0s 0.00% 1.68s 68.00%
oauth-api-new-connections 4.21 amd64 ovn metal compact 113 1s 0.00% 1.88s 54.10%
image-registry-new-connections 4.21 amd64 ovn aws ha 270 0s 0.00% 2s 100.00%
openshift-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1017 4s 0.00% 59.52s 69.86%
metrics-api-new-connections 4.21 ppc64le ovn none ha 168 2s 0.00% 1m0.94s 39.55%
pod-to-host-new-connections 4.21 s390x ovn none ha 149 53.2s 0.00% 11m42.68s 73866.32%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 1017 1m13.4s 0.00% 5m16.52s 941.18%
oauth-api-reused-connections 4.21 amd64 ovn metal compact 113 1s 0.00% 2s 100.00%
openshift-api-new-connections 4.21 ppc64le ovn none ha 168 2.65s 0.00% 58.93s 44.37%
cache-openshift-api-reused-connections 4.21 amd64 ovn metal compact 113 1s 0.00% 2s 100.00%
pod-to-pod-new-connections 4.21 amd64 ovn metal compact 113 2m7s 0.00% 13m12.04s 191.75%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1007 55.7s 0.00% 2m34.64s 101.93%
cache-oauth-api-reused-connections 4.21 amd64 ovn metal compact 113 1s 0.00% 2s 80.18%
oauth-api-new-connections 4.21 4.20 amd64 ovn aws ha 1133 0s 0.00% 7.68s 53.60%
metrics-api-reused-connections 4.21 4.21 amd64 ovn aws ha 509 1s 0.00% 2s 124.72%
openshift-api-http2-service-network-new-connections 4.21 4.20 amd64 ovn gcp ha 904 0s 0.00% 1m31.88s 251.76%
pod-to-host-new-connections 4.21 amd64 ovn metal compact 113 1h51m7.2s 0.00% 2h50m21.16s 67.71%
oauth-api-new-connections 4.21 ppc64le ovn none ha 168 3s 0.00% 58.26s 35.80%
openshift-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1017 20s 0.00% 1m51.24s 42.91%
kube-api-http1-internal-lb-reused-connections 4.21 4.21 amd64 ovn aws ha 509 1s 0.00% 3s 237.08%
ingress-to-oauth-server-new-connections 4.21 4.20 amd64 ovn vsphere ha 106 1s 0.00% 1s 270.37%
cache-oauth-api-reused-connections 4.21 4.21 amd64 ovn azure ha 474 1s 0.00% 2.27s 131.63%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 149 32.4s 0.00% 3m48.4s 7999.29%
pod-to-host-new-connections 4.21 4.20 amd64 ovn azure ha 1007 52.7s 0.00% 2m23.64s 93.53%
host-to-service-new-connections 4.21 amd64 ovn metal compact 113 0s 0.00% 880ms 700.00%
ingress-to-console-reused-connections 4.21 amd64 ovn kubevirt external 117 0s 0.00% 1s 53.85%

alerts Information

There were (20981) added jobs and (6384) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 31 jobs increased and 204 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 128 11m54.5s 377.93% 13m39.8s 164.96%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn azure ha 193 9m58s 304.05% 19m31.52s 208.62%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.20 amd64 ovn gcp ha 185 7m46s 294.92% 18m20.8s 30.43%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 4.21 amd64 ovn aws ha 200 12m44.5s 50.20% 15m36.62s 60.22%
CannotRetrieveUpdates openshift-cluster-version Warning 4.21 amd64 ovn vsphere ha 235 5m58s 72.12% 10m28s 73.10%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn metal ha 295 0s 0.00% 7m26.04s 116.00%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 378 0s 0.00% 2m3.36s 135.60%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn gcp ha 171 3m28s 0.00% 7m54.6s 114.75%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn gcp ha 224 58s 0.00% 3m27.54s 641.21%
CsvAbnormalFailedOver2Min openshift-operator-lifecycle-manager Warning 4.21 4.20 amd64 ovn azure ha 187 0s 0.00% 2m58s 75.44%
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn vsphere ha 235 3h9m20.4s 0.00% 5h58m33s 53.82%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 128 0s 0.00% 3m10.34s 12760.81%
CannotRetrieveUpdates openshift-cluster-version Warning 4.21 amd64 ovn azure ha 145 18m47.6s 0.00% 38m1.28s 37.31%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn aws ha 378 2m2.5s 0.00% 10m18.7s 51.49%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn vsphere ha 235 0s 0.00% 3m15.28s 4258.93%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn azure ha 145 6m42.8s 0.00% 16m20.88s 793.82%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn aws ha 200 28s 0.00% 3m56.9s 395.61%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 378 1m58s 0.00% 11m2.9s 1042.93%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn gcp ha 171 9m28s 0.00% 26m28s 63.54%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.20 amd64 ovn gcp ha 185 57.6s 0.00% 3m2.48s 30.34%
/cc @xueqzhan, @neisw, @smg247

@openshift-ci-robot
Copy link

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 5, 2025
@openshift-ci-robot
Copy link

@openshift-bot: This pull request explicitly references no jira issue.

Details

In response to this:

disruptions Information

There were (2748) added jobs and (0) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 136 jobs increased and 891 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 483 4m31.5s 684.68% 7m46.36s 30.05%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 483 3m3.7s 652.87% 5m3s 30.24%
gcp-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 104 46.7s 1567.86% 4m59.56s 444.46%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn azure ha 551 1m17s 104.24% 1m26.5s 127.99%
azure-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 104 48.5s 165.03% 7m31.56s 595.56%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 577 1m18s 56.94% 3m31.04s 322.59%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 577 1m6s 49.66% 2m10s 190.05%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1m0s 33.78% 1m11.96s 60.02%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1202 1m0s 30.86% 1m12s 56.62%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 551 5s 400.00% 9s 800.00%
kube-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 551 5s 85.19% 7s 138.10%
host-to-host-new-connections 4.21 4.20 amd64 ovn gcp ha 618 0s 0.00% 1s 78.57%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn aws ha 577 0s 0.00% 54.12s 1772.66%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 4m11.99s 728.91%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 0s 0.00% 12.89s 1189.00%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1202 0s 0.00% 14s 1300.00%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn azure ha 551 2s 0.00% 14s 104.08%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1130 0s 0.00% 4m13.42s 230.92%
cache-openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 2s 100.00%
pod-to-host-new-connections 4.21 4.20 amd64 ovn azure ha 1130 2s 0.00% 2m11.95s 77.78%
host-to-service-new-connections 4.21 4.20 amd64 ovn azure ha 1130 0s 0.00% 1.71s 71.00%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1188 1s 0.00% 5m21.37s 524.38%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 136 250ms 0.00% 1m5.15s 2210.28%
openshift-api-http2-internal-lb-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 45s 415.46%
kube-api-http1-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 577 1m6.2s 0.00% 2m9.24s 70.55%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1202 1m8s 0.00% 1m34.9s 49.19%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1130 0s 0.00% 2m32.42s 232.72%
openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 10.97s 273.13%
openshift-api-http2-internal-lb-reused-connections 4.21 4.20 amd64 ovn azure ha 1130 1s 0.00% 2s 100.00%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 577 1m16.2s 0.00% 3m38.68s 155.59%
image-registry-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 2s 106.19%
openshift-api-http2-internal-lb-reused-connections 4.21 4.21 amd64 ovn aws ha 577 7.2s 0.00% 2m1.44s 189.63%
openshift-api-http2-internal-lb-new-connections 4.21 4.20 amd64 ovn azure ha 1130 0s 0.00% 1s 170.27%
kube-api-http2-service-network-reused-connections 4.21 amd64 ovn aws ha 202 0s 0.00% 990ms 253.57%
oauth-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 3s 209.28%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1188 1s 0.00% 3m55.04s 687.40%
aws-network-liveness-new-connections 4.21 multi ovn aws ha 180 0s 0.00% 1s 150.00%
openshift-api-reused-connections 4.21 4.21 amd64 ovn aws ha 577 2s 0.00% 56.44s 1781.33%

alerts Information

There were (10014) added jobs and (1300) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 31 jobs increased and 195 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
ClusterOperatorDegraded openshift-cluster-version Warning 4.21 4.21 amd64 ovn gcp ha 279 2h19m53.4s 347.15% 2h25m27.08s 294.69%
TargetDown openshift-machine-config-operator Warning 4.21 arm64 ovn aws ha 139 1h57m36.6s 55.54% 2h13m1.8s 49.95%
KubeAggregatedAPIDown default Warning 4.21 4.21 amd64 ovn aws single 166 22m12s 155.17% 35m55.5s 36.16%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 139 3m31s 41.14% 11m57.24s 131.82%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 139 0s 0.00% 2m28s 9900.00%
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn vsphere ha 225 3h11m7.2s 0.00% 6h22m36.96s 64.15%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn aws ha 196 28s 0.00% 2m1s 153.14%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn gcp ha 279 2.8s 0.00% 2m28s 428.57%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 354 0s 0.00% 3m39.24s 278.00%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn vsphere ha 225 0s 0.00% 4m21.52s 5737.50%
OVNKubernetesResourceRetryFailure openshift-ovn-kubernetes Warning 4.21 amd64 ovn metal ha 308 0s 0.00% 9m28s 176.53%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn gcp ha 158 2m16s 0.00% 27m7.3s 67.59%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn metal ha 308 0s 0.00% 8m13.4s 138.93%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn azure ha 139 0s 0.00% 10m30.24s 474.30%
/cc @xueqzhan, @neisw, @smg247

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from deads2k and sjenning December 5, 2025 01:07
@openshift-ci-robot
Copy link

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@neisw
Copy link
Contributor

neisw commented Dec 5, 2025

/lgtm
/verified bypass
/retest-required

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Dec 5, 2025
@openshift-ci-robot
Copy link

@neisw: The verified label has been added.

Details

In response to this:

/lgtm
/verified bypass
/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 5, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 5, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neisw, openshift-bot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 5, 2025
@openshift-bot openshift-bot force-pushed the no-issue--automated---update-synthetic-test-data branch from 27d6341 to 19745d6 Compare December 12, 2025 01:03
@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Dec 12, 2025
@openshift-ci-robot
Copy link

@openshift-bot: This pull request explicitly references no jira issue.

Details

In response to this:

disruptions Information

There were (3616) added jobs and (344) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 285 jobs increased and 800 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 732 4m55.8s 495.17% 7m24.97s 791.01%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 732 5m3.25s 327.11% 7m33.66s 430.22%
ingress-to-oauth-server-new-connections 4.21 s390x ovn none ha 156 4m58.25s 175.90% 13m22.3s 62.67%
ingress-to-console-new-connections 4.21 s390x ovn none ha 156 4m58.25s 175.90% 13m22.85s 62.78%
cache-kube-api-new-connections 4.21 s390x ovn none ha 156 4m48s 183.19% 13m22.95s 62.84%
kube-api-new-connections 4.21 s390x ovn none ha 156 4m47.25s 183.28% 13m23.5s 62.96%
cache-openshift-api-new-connections 4.21 s390x ovn none ha 156 4m47.25s 182.45% 13m23.5s 62.96%
openshift-api-new-connections 4.21 s390x ovn none ha 156 4m47.25s 182.45% 13m22.95s 62.84%
oauth-api-new-connections 4.21 s390x ovn none ha 156 4m46.5s 181.71% 13m23.5s 62.96%
cache-oauth-api-new-connections 4.21 s390x ovn none ha 156 4m46.5s 181.71% 13m23.5s 62.96%
metrics-api-new-connections 4.21 s390x ovn none ha 156 4m47s 140.57% 13m22.3s 62.77%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 732 3m19s 215.87% 4m37s 265.53%
kube-api-http1-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 732 3m11.45s 203.89% 4m27.76s 253.34%
metrics-api-reused-connections 4.21 s390x ovn none ha 156 3m1s 229.69% 13m14.8s 62.28%
ingress-to-console-reused-connections 4.21 s390x ovn none ha 156 3m15.75s 135.28% 13m16.35s 62.06%
ingress-to-oauth-server-reused-connections 4.21 s390x ovn none ha 156 3m15.5s 134.13% 13m15.8s 61.95%
cache-openshift-api-reused-connections 4.21 s390x ovn none ha 156 2m4.5s 128.86% 13m15.25s 62.03%
cache-oauth-api-reused-connections 4.21 s390x ovn none ha 156 2m14.25s 107.82% 13m15.25s 62.01%
azure-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 112 1m27.75s 379.51% 7m27.72s 589.65%
gcp-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 112 1m9s 2364.29% 4m55.72s 437.48%
cache-kube-api-reused-connections 4.21 s390x ovn none ha 156 2m9s 105.09% 13m15.25s 62.03%
openshift-api-reused-connections 4.21 s390x ovn none ha 156 2m8.75s 103.72% 13m15.8s 62.14%
oauth-api-reused-connections 4.21 s390x ovn none ha 156 2m7.75s 102.14% 13m15.8s 62.14%
kube-api-reused-connections 4.21 s390x ovn none ha 156 2m6.75s 100.55% 13m15.25s 62.03%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn vsphere ha 100 54.15s 2607.50% 2m49.38s 82.72%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn azure ha 684 1m25s 86.00% 1m33.17s 102.81%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn vsphere ha 100 41.05s 1125.37% 1m37.67s 34.51%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m14.8s 66.78% 2m8.76s 186.32%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m14.8s 63.14% 2m9.96s 182.71%
kube-api-http1-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m14.8s 63.14% 2m9.96s 182.71%
kube-api-http1-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m12.8s 62.32% 2m6.84s 182.05%
openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 11s 307.41% 48.92s 1563.95%
kube-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 8s 700.00% 23s 248.48%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 684 5s 400.00% 7s 600.00%
service-load-balancer-with-pdb-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 3s 50.00% 4s 100.00%
kube-api-http1-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 2s 106.19%
openshift-api-http2-internal-lb-reused-connections 4.21 4.20 amd64 ovn azure ha 1112 1s 0.00% 2s 100.00%
kube-api-http1-internal-lb-reused-connections 4.21 amd64 ovn metal compact 107 8m21.6s 0.00% 17m25.92s 43.20%
kube-api-http1-internal-lb-reused-connections 4.21 4.21 amd64 ovn aws ha 732 1s 0.00% 3.69s 314.61%
metrics-api-reused-connections 4.21 4.21 amd64 ovn aws ha 732 1s 0.00% 16.69s 1775.28%
cache-openshift-api-new-connections 4.21 amd64 ovn azure external 786 0s 0.00% 1.45s 45.00%
cache-kube-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 74.77%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 732 25s 0.00% 40.38s 1034.27%
openshift-api-http2-service-network-reused-connections 4.21 multi ovn aws ha 218 0s 0.00% 830ms 107.50%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 49.8s 0.00% 1m38s 233.33%
openshift-api-new-connections 4.21 amd64 ovn azure external 786 0s 0.00% 1.45s 45.00%
cache-oauth-api-new-connections 4.21 amd64 ovn azure external 786 0s 0.00% 1.45s 45.00%
image-registry-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 2s 106.19%
image-registry-reused-connections 4.21 4.20 amd64 ovn gcp ha 763 1s 0.00% 2s 100.00%
metrics-api-new-connections 4.21 4.21 amd64 ovn aws ha 732 14s 0.00% 28.38s 3088.76%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1112 0s 0.00% 2m32.78s 233.51%
host-to-service-new-connections 4.21 4.20 amd64 ovn azure ha 1112 0s 0.00% 2s 100.00%
kube-api-http1-service-network-reused-connections 4.21 multi ovn aws ha 218 0s 0.00% 830ms 107.50%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1193 2s 0.00% 3m54.64s 686.06%
pod-to-host-new-connections 4.21 4.20 amd64 ovn azure ha 1112 4s 0.00% 1m37.56s 31.45%
kube-api-http1-service-network-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 12s 0.00% 35s 3400.00%
pod-to-host-new-connections 4.21 4.21 amd64 ovn aws ha 732 0s 0.00% 4s 43.88%
openshift-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1193 2s 0.00% 8.08s 34.67%
metrics-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 2.96s 196.00%
cache-oauth-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 74.77%
openshift-api-http2-internal-lb-new-connections 4.21 amd64 ovn metal compact 107 20m11.1s 0.00% 34m4.3s 35.65%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1112 1s 0.00% 4m13.78s 231.39%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn aws ha 732 47.45s 0.00% 1m56.9s 3944.98%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 732 25s 0.00% 37s 1180.28%
kube-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 94.00%
openshift-api-http2-internal-lb-new-connections 4.21 4.20 amd64 ovn azure ha 1112 0s 0.00% 1s 170.27%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 156 3s 0.00% 2m55.15s 6110.99%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1193 3s 0.00% 5m18.92s 519.62%
pod-to-service-new-connections 4.21 4.21 amd64 ovn vsphere ha 100 50ms 0.00% 1.01s 380.95%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 4m11.96s 728.82%
kube-api-http1-service-network-reused-connections 4.21 amd64 ovn metal compact 107 10m4.7s 0.00% 18m46.3s 50.99%
openshift-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 732 2m2.35s 0.00% 3m36.76s 4332.72%
aws-network-liveness-new-connections 4.21 multi ovn aws ha 218 0s 0.00% 830ms 107.50%
oauth-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1193 0s 0.00% 7s 40.00%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 16s 0.00% 25s 2400.00%
cache-oauth-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 21.96s 2163.92%
openshift-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 45s 0.00% 1m40.88s 4944.00%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 16s 0.00% 25.96s 2496.00%
pod-to-service-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 18.88s 1846.39%
openshift-api-new-connections 4.21 4.20 amd64 ovn aws ha 1193 0s 0.00% 8s 38.65%
host-to-host-new-connections 4.21 4.21 amd64 ovn aws ha 732 0s 0.00% 5s 415.46%
kube-api-http1-service-network-new-connections 4.21 amd64 ovn metal compact 107 17m48.5s 0.00% 30m54.86s 31.02%
openshift-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 20.8s 0.00% 53.72s 53.31%

alerts Information

There were (15045) added jobs and (2006) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 40 jobs increased and 179 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
ClusterOperatorDegraded openshift-cluster-version Warning 4.21 4.21 amd64 ovn gcp ha 272 1h40m31.9s 221.34% 2h10m39.9s 254.57%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn vsphere ha 281 1h26m24s 126.53% 1h51m24.4s 75.48%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 4.21 amd64 ovn aws ha 311 17m24s 105.11% 20m40.4s 112.18%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn aws ha 483 14m19.2s 47.93% 22m44.16s 59.02%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 174 6m37.8s 166.09% 13m43.66s 166.21%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn gcp ha 209 12m1.2s 46.73% 19m57.68s 71.46%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn vsphere ha 281 27m26s 0.00% 34m28s 46060.71%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn aws ha 311 1m58s 0.00% 3m51.4s 384.10%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 483 0s 0.00% 4m33.04s 370.76%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 arm64 ovn aws ha 174 13m32.1s 0.00% 22m8.72s 33.61%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn aws single 129 29m24.4s 0.00% 50m10.4s 33.83%
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn aws single 129 1h4m36.8s 0.00% 2h3m41.28s 36.87%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn gcp ha 272 28s 0.00% 2m28s 428.57%
CsvAbnormalFailedOver2Min openshift-operator-lifecycle-manager Warning 4.21 4.21 amd64 ovn aws ha 311 2m58s 0.00% 6m53s 132.02%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn aws single 129 7m46s 0.00% 18m40.4s 90.87%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 174 0s 0.00% 1m26.54s 5747.30%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 483 0s 0.00% 3m3.04s 249.58%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn metal ha 393 35.2s 0.00% 9m56s 188.62%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn gcp ha 209 15m52s 0.00% 30m34.4s 88.92%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn vsphere ha 281 38m28s 0.00% 1h36m46.8s 375.90%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn azure ha 256 2m4s 0.00% 12m1.6s 90.09%
/cc @xueqzhan, @neisw, @smg247

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Dec 12, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 12, 2025

New changes are detected. LGTM label has been removed.

@openshift-ci-robot
Copy link

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-bot openshift-bot force-pushed the no-issue--automated---update-synthetic-test-data branch from 19745d6 to bea3310 Compare December 19, 2025 01:03
@openshift-ci-robot
Copy link

@openshift-bot: This pull request explicitly references no jira issue.

Details

In response to this:

disruptions Information

There were (3630) added jobs and (466) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 288 jobs increased and 877 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
kube-api-http2-internal-lb-new-connections 4.21 amd64 ovn metal compact 107 23m54.3s 44.50% 34m41.06s 37.74%
kube-api-http1-internal-lb-new-connections 4.21 amd64 ovn metal compact 107 24m5.7s 42.65% 36m33.14s 37.87%
openshift-api-http2-internal-lb-new-connections 4.21 amd64 ovn metal compact 107 23m16.3s 41.04% 35m14.02s 40.27%
kube-api-http2-service-network-new-connections 4.21 amd64 ovn metal compact 107 23m35.9s 38.12% 30m38.78s 32.71%
openshift-api-http2-service-network-reused-connections 4.21 amd64 ovn metal compact 107 12m58.7s 75.58% 19m20.22s 36.94%
openshift-api-http2-service-network-new-connections 4.21 amd64 ovn metal compact 107 23m8.5s 31.45% 30m18.8s 33.19%
kube-api-http1-service-network-new-connections 4.21 amd64 ovn metal compact 107 23m8.5s 30.04% 31m27.7s 33.34%
kube-api-http1-service-network-reused-connections 4.21 amd64 ovn metal compact 107 14m4.5s 59.96% 20m25.78s 64.32%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 676 5m42.25s 588.63% 7m47.75s 836.62%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 676 5m37.5s 375.35% 8m7.75s 470.07%
kube-api-http2-service-network-reused-connections 4.21 amd64 ovn metal compact 107 12m17.2s 47.56% 17m15.1s 33.92%
ingress-to-oauth-server-new-connections 4.21 s390x ovn none ha 157 5m44.8s 218.96% 13m19.96s 62.20%
ingress-to-console-new-connections 4.21 s390x ovn none ha 157 5m44.8s 218.96% 13m20.52s 62.31%
openshift-api-new-connections 4.21 s390x ovn none ha 157 5m31.4s 225.86% 13m20.64s 62.38%
oauth-api-new-connections 4.21 s390x ovn none ha 157 5m31.4s 225.86% 13m21.2s 62.49%
kube-api-new-connections 4.21 s390x ovn none ha 157 5m30.6s 226.04% 13m21.2s 62.49%
cache-openshift-api-new-connections 4.21 s390x ovn none ha 157 5m30.6s 225.07% 13m21.2s 62.49%
cache-oauth-api-new-connections 4.21 s390x ovn none ha 157 5m30.6s 225.07% 13m21.2s 62.49%
cache-kube-api-new-connections 4.21 s390x ovn none ha 157 5m30.6s 225.07% 13m20.64s 62.38%
cache-openshift-api-reused-connections 4.21 s390x ovn none ha 157 4m28s 392.65% 13m12.8s 61.53%
metrics-api-reused-connections 4.21 s390x ovn none ha 157 4m27.6s 387.43% 13m12.36s 61.78%
metrics-api-new-connections 4.21 s390x ovn none ha 157 5m28s 174.94% 13m19.96s 62.30%
cache-kube-api-reused-connections 4.21 s390x ovn none ha 157 4m27.6s 325.44% 13m12.8s 61.53%
openshift-api-reused-connections 4.21 s390x ovn none ha 157 4m27.8s 323.73% 13m13.36s 61.65%
oauth-api-reused-connections 4.21 s390x ovn none ha 157 4m27.8s 323.73% 13m13.36s 61.65%
kube-api-reused-connections 4.21 s390x ovn none ha 157 4m27.4s 323.10% 13m12.8s 61.53%
cache-oauth-api-reused-connections 4.21 s390x ovn none ha 157 4m27.8s 314.55% 13m12.8s 61.51%
ingress-to-console-reused-connections 4.21 s390x ovn none ha 157 4m37.2s 233.17% 13m13.92s 61.56%
ingress-to-oauth-server-reused-connections 4.21 s390x ovn none ha 157 4m35.6s 230.06% 13m13.36s 61.45%
kube-api-http1-internal-lb-reused-connections 4.21 amd64 ovn metal compact 107 11m26.6s 37.84% 17m49.62s 46.44%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 676 3m21.25s 356.35% 4m34.5s 512.45%
kube-api-http1-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 676 3m22.5s 221.43% 4m32.5s 259.59%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1120 2m17s 120.79% 3m37.24s 241.52%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1120 2m16.05s 106.14% 3m37.43s 199.49%
host-to-pod-new-connections 4.21 amd64 ovn metal compact 107 2m34.6s 80.29% 5m58.36s 43.89%
host-to-pod-reused-connections 4.21 amd64 ovn metal compact 107 2m24s 82.51% 5m13.04s 32.62%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1120 1m37s 111.56% 2m16.86s 197.72%
kube-api-http1-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1120 1m37s 111.56% 2m15s 193.67%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1120 1m34.05s 109.70% 2m11.81s 193.11%
gcp-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 113 49.8s 1678.57% 4m2.32s 340.42%
azure-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 113 1m1.8s 237.70% 6m29.64s 500.18%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn azure ha 638 1m24.15s 84.14% 1m30.63s 97.28%
openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1120 16s 492.59% 53.62s 1723.81%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 638 5s 400.00% 7s 600.00%
ingress-to-oauth-server-reused-connections 4.21 amd64 ovn metal compact 107 7.7s 69.23% 10.94s 75.88%
ingress-to-console-reused-connections 4.21 amd64 ovn metal compact 107 7.7s 69.23% 9.94s 59.81%
service-load-balancer-with-pdb-reused-connections 4.21 4.21 amd64 ovn azure ha 638 3s 233.33% 3s 206.12%
ingress-to-console-new-connections 4.21 amd64 ovn metal compact 107 3s 200.00% 4.94s 48.35%
ingress-to-oauth-server-new-connections 4.21 amd64 ovn metal compact 107 3s 93.55% 7.88s 136.64%
service-load-balancer-with-pdb-reused-connections 4.21 4.21 amd64 ovn gcp ha 1120 3s 50.00% 4s 100.00%
metrics-api-new-connections 4.21 4.20 amd64 ovn gcp ha 803 0s 0.00% 20.94s 39.60%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1136 34.5s 0.00% 5m8.35s 499.09%
openshift-api-new-connections 4.21 4.20 amd64 ovn aws ha 1136 1s 0.00% 8s 38.65%
kube-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 2s 100.00%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1056 9s 0.00% 1m50.9s 44.82%
pod-to-host-reused-connections 4.21 amd64 ovn metal compact 107 1h0m11.1s 0.00% 1h22m4.58s 42.50%
oauth-api-new-connections 4.21 4.20 amd64 ovn aws ha 1136 1s 0.00% 7s 40.00%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 157 17.2s 0.00% 1m58.24s 4092.91%
cache-oauth-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 2s 80.18%
aws-network-liveness-new-connections 4.21 multi ovn aws ha 225 0s 0.00% 760ms 90.00%
cache-openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1120 12s 0.00% 28s 2700.00%
oauth-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1136 0s 0.00% 6.65s 33.00%
cache-openshift-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 94.00%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1120 19s 0.00% 1m4.53s 6353.00%
kube-api-http1-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1120 1s 0.00% 2s 106.19%
metrics-api-reused-connections 4.21 amd64 ovn aws ha 246 0s 0.00% 1s 257.14%
metrics-api-reused-connections 4.21 4.21 amd64 ovn aws ha 676 1s 0.00% 22.5s 2428.09%
openshift-api-http2-internal-lb-reused-connections 4.21 amd64 ovn metal compact 107 11m52.9s 0.00% 17m22.88s 31.99%
cache-kube-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 74.77%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1120 18s 0.00% 27s 2600.00%
cache-oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 803 0s 0.00% 16.92s 30.15%
cache-oauth-api-reused-connections 4.21 4.21 amd64 ovn azure ha 638 1s 0.00% 2s 104.08%
openshift-api-http2-internal-lb-reused-connections 4.21 4.20 amd64 ovn azure ha 1056 1s 0.00% 2s 100.00%
aws-network-liveness-new-connections 4.21 4.20 amd64 ovn gcp ha 803 0s 0.00% 1s 78.57%
openshift-api-http2-service-network-new-connections 4.21 4.20 amd64 ovn gcp ha 803 0s 0.00% 37.82s 44.79%
openshift-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1136 2s 0.00% 9s 50.00%
openshift-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 676 2m22.75s 0.00% 4m19s 5196.52%
oauth-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1120 21s 0.00% 1m3.05s 75.53%
openshift-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1120 25s 0.00% 1m12.29s 106.31%
kube-api-new-connections 4.21 4.20 amd64 ovn aws ha 1136 0s 0.00% 5.65s 41.25%
openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 803 0s 0.00% 20.96s 48.44%
pod-to-host-new-connections 4.21 4.21 amd64 ovn aws ha 676 0s 0.00% 5s 79.86%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn gcp ha 1120 1m2s 0.00% 2m0.24s 308.98%
pod-to-service-new-connections 4.21 4.21 amd64 ovn gcp ha 1120 1s 0.00% 19s 1858.76%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1056 5s 0.00% 2m7.9s 179.20%
oauth-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 94.00%
pod-to-host-new-connections 4.21 s390x ovn none ha 156 1s 0.00% 6m25s 40426.32%
openshift-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 2s 100.00%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1136 28.5s 0.00% 3m47.7s 662.81%
image-registry-reused-connections 4.21 4.20 amd64 ovn gcp ha 803 1s 0.00% 2s 100.00%
cache-openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 803 0s 0.00% 18.9s 39.38%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn azure ha 638 1s 0.00% 15s 118.66%

alerts Information

There were (17149) added jobs and (4516) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 28 jobs increased and 207 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 138 10m37s 326.09% 13m16.16s 157.32%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 4.21 amd64 ovn aws ha 216 12m22s 45.78% 16m47.7s 72.37%
CannotRetrieveUpdates openshift-cluster-version Warning 4.21 amd64 ovn vsphere ha 242 5m25s 56.25% 8m28s 40.02%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn azure ha 218 3m58s 60.81% 15m12.7s 140.44%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn gcp ha 182 9m57.9s 0.00% 24m9.4s 49.27%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 398 0s 0.00% 2m54.12s 232.54%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn aws single 103 3m28s 0.00% 13m7.92s 34.23%
CsvAbnormalFailedOver2Min openshift-operator-lifecycle-manager Warning 4.21 4.21 amd64 ovn aws ha 216 0s 0.00% 5m29.3s 85.00%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn aws ha 216 58s 0.00% 3m53.5s 388.49%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn gcp ha 182 1m27.9s 0.00% 7m18.42s 98.38%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn azure ha 157 0s 0.00% 12m13.12s 568.05%
CsvAbnormalFailedOver2Min openshift-operator-lifecycle-manager Warning 4.21 4.20 amd64 ovn azure ha 197 0s 0.00% 2m58s 75.44%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 398 32.5s 0.00% 7m28.84s 673.86%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 138 0s 0.00% 3m46.9s 15231.08%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn gcp ha 222 57.9s 0.00% 2m57.58s 534.21%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn vsphere ha 242 0s 0.00% 11m31.9s 15344.20%
/cc @xueqzhan, @neisw, @smg247

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-bot openshift-bot force-pushed the no-issue--automated---update-synthetic-test-data branch from bea3310 to 4ad3ea2 Compare December 26, 2025 01:03
@openshift-ci-robot
Copy link

@openshift-bot: This pull request explicitly references no jira issue.

Details

In response to this:

disruptions Information

There were (3754) added jobs and (466) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 355 jobs increased and 873 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
host-to-host-new-connections 4.21 amd64 ovn metal compact 106 1h54m29s 40.50% 3h22m31.9s 60.67%
pod-to-host-reused-connections 4.21 amd64 ovn metal compact 106 1h5m34.5s 30.53% 2h1m10.75s 110.38%
openshift-api-http2-localhost-new-connections 4.21 amd64 ovn metal compact 106 53m24.25s 33.57% 1h25m1.45s 59.30%
kube-api-http1-localhost-new-connections 4.21 amd64 ovn metal compact 106 54m13.75s 32.38% 1h27m34.75s 56.73%
kube-api-http2-localhost-new-connections 4.21 amd64 ovn metal compact 106 53m27.75s 30.36% 1h25m40.85s 58.29%
openshift-api-http2-service-network-new-connections 4.21 amd64 ovn metal compact 106 25m16.75s 43.59% 30m19s 33.21%
kube-api-http2-service-network-new-connections 4.21 amd64 ovn metal compact 106 24m37.25s 44.11% 30m39.15s 32.74%
kube-api-http2-internal-lb-new-connections 4.21 amd64 ovn metal compact 106 24m2.75s 45.35% 34m42.05s 37.80%
kube-api-http1-internal-lb-new-connections 4.21 amd64 ovn metal compact 106 24m14.75s 43.54% 36m34.95s 37.98%
openshift-api-http2-internal-lb-new-connections 4.21 amd64 ovn metal compact 106 23m37.5s 43.18% 35m14.35s 40.29%
kube-api-http1-service-network-new-connections 4.21 amd64 ovn metal compact 106 24m49s 39.45% 31m27.75s 33.34%
openshift-api-http2-service-network-reused-connections 4.21 amd64 ovn metal compact 106 13m29.75s 82.58% 19m31.65s 38.29%
kube-api-http1-service-network-reused-connections 4.21 amd64 ovn metal compact 106 14m8.25s 60.67% 20m35.2s 65.59%
kube-api-http1-internal-lb-reused-connections 4.21 amd64 ovn metal compact 106 12m35.75s 51.73% 17m49.85s 46.47%
kube-api-http2-service-network-reused-connections 4.21 amd64 ovn metal compact 106 12m29.5s 50.02% 17m17.45s 34.22%
ingress-to-oauth-server-new-connections 4.21 s390x ovn none ha 149 5m46.4s 220.44% 13m38.68s 65.99%
ingress-to-console-new-connections 4.21 s390x ovn none ha 149 5m46.4s 220.44% 13m39.16s 66.09%
openshift-api-http2-internal-lb-reused-connections 4.21 amd64 ovn metal compact 106 13m23.5s 41.95% 17m22.9s 32.00%
openshift-api-new-connections 4.21 s390x ovn none ha 149 5m38.2s 232.55% 13m39.12s 66.12%
oauth-api-new-connections 4.21 s390x ovn none ha 149 5m38.2s 232.55% 13m39.6s 66.22%
kube-api-new-connections 4.21 s390x ovn none ha 149 5m37.8s 233.14% 13m39.6s 66.22%
cache-kube-api-new-connections 4.21 s390x ovn none ha 149 5m37.8s 232.15% 13m39.12s 66.12%
cache-oauth-api-new-connections 4.21 s390x ovn none ha 149 5m37.8s 232.15% 13m39.6s 66.22%
cache-openshift-api-new-connections 4.21 s390x ovn none ha 149 5m37.8s 232.15% 13m39.6s 66.22%
cache-openshift-api-reused-connections 4.21 s390x ovn none ha 149 4m32s 400.00% 13m32.4s 65.53%
metrics-api-new-connections 4.21 s390x ovn none ha 149 5m36s 181.64% 13m38.68s 66.09%
ingress-to-console-reused-connections 4.21 s390x ovn none ha 149 4m59.6s 260.10% 13m33.36s 65.52%
metrics-api-reused-connections 4.21 s390x ovn none ha 149 4m30.8s 393.26% 13m31.88s 65.76%
ingress-to-oauth-server-reused-connections 4.21 s390x ovn none ha 149 4m54.8s 253.05% 13m32.88s 65.42%
oauth-api-reused-connections 4.21 s390x ovn none ha 149 4m31.4s 329.43% 13m32.88s 65.62%
openshift-api-reused-connections 4.21 s390x ovn none ha 149 4m31.4s 329.43% 13m32.88s 65.62%
cache-kube-api-reused-connections 4.21 s390x ovn none ha 149 4m30.8s 330.52% 13m32.4s 65.53%
kube-api-reused-connections 4.21 s390x ovn none ha 149 4m30.2s 327.53% 13m32.4s 65.53%
cache-oauth-api-reused-connections 4.21 s390x ovn none ha 149 4m31.4s 320.12% 13m32.4s 65.51%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 614 3m46.05s 553.32% 8m9.87s 36.61%
openshift-api-http2-internal-lb-reused-connections 4.21 4.21 amd64 ovn aws ha 730 2m32.1s 2435.00% 4m28.26s 539.78%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 730 3m29s 231.75% 4m37.71s 266.47%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 614 2m47.05s 584.63% 5m43.48s 47.64%
azure-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 117 2m31.2s 726.23% 7m36.24s 602.77%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn gcp ha 830 2m31.2s 704.26% 7m37.55s 33.47%
gcp-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 117 1m42s 3542.86% 5m4.24s 452.96%
openshift-api-reused-connections 4.21 4.21 amd64 ovn aws ha 730 1m34.55s 4627.50% 2m33.26s 5008.67%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn gcp ha 830 1m42.55s 476.12% 5m12.39s 30.12%
pod-to-pod-new-connections 4.21 amd64 ovn metal compact 106 3m14.75s 76.32% 13m30.1s 198.40%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1067 2m18.7s 110.15% 3m38s 200.28%
host-to-pod-reused-connections 4.21 amd64 ovn metal compact 106 2m18.75s 75.86% 5m13.7s 32.90%
azure-network-liveness-new-connections 4.21 arm64 ovn aws ha 176 58.25s 1113.54% 5m21s 55.95%
kube-api-http1-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1067 1m38s 113.74% 2m16.36s 196.63%
pod-to-pod-reused-connections 4.21 amd64 ovn metal compact 106 2m48.25s 43.19% 13m51.85s 262.68%
kube-api-http2-internal-lb-reused-connections 4.21 4.21 amd64 ovn aws ha 730 56s 833.33% 1m52s 1319.52%
gcp-network-liveness-new-connections 4.21 arm64 ovn aws ha 176 44.75s 1143.06% 3m7.5s 45.64%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn azure ha 705 1m15.8s 101.06% 1m22s 116.13%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn azure ha 705 1m23s 81.62% 1m29s 93.73%
host-to-pod-new-connections 4.21 amd64 ovn metal compact 106 1m57.25s 36.73% 5m59.3s 44.27%
host-to-host-new-connections 4.21 4.20 amd64 ovn azure ha 1010 30.55s 2955.00% 2m41.38s 74.50%
metrics-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1067 12s 1100.00% 37.02s 105.67%
ingress-to-oauth-server-reused-connections 4.21 amd64 ovn metal compact 106 8.75s 92.31% 14.8s 137.94%
ingress-to-console-reused-connections 4.21 amd64 ovn metal compact 106 8.75s 92.31% 10.95s 76.05%
kube-api-http1-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 705 5s 455.56% 7s 614.29%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 705 5s 400.00% 8s 700.00%
host-to-host-new-connections 4.21 4.21 amd64 ovn azure ha 705 6s 100.00% 1m45.6s 40.69%
ingress-to-oauth-server-new-connections 4.21 amd64 ovn metal compact 106 3s 93.55% 7.95s 138.74%
service-load-balancer-with-pdb-reused-connections 4.21 4.21 amd64 ovn gcp ha 1067 3s 50.00% 4s 100.00%
metrics-api-new-connections 4.21 amd64 ovn metal compact 106 1s 0.00% 1.95s 35.42%
pod-to-host-new-connections 4.21 4.20 amd64 ovn azure ha 1010 17s 0.00% 2m7.83s 72.23%
metrics-api-reused-connections 4.21 amd64 ovn aws ha 256 0s 0.00% 1s 257.14%
service-load-balancer-with-pdb-new-connections 4.21 4.21 amd64 ovn gcp ha 1067 1s 0.00% 4.34s 49.14%
openshift-api-http2-localhost-reused-connections 4.21 amd64 ovn metal compact 106 26m24.5s 0.00% 53m26.55s 78.43%
openshift-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn vsphere ha 102 13.95s 0.00% 34.84s 97.62%
host-to-host-reused-connections 4.21 4.20 amd64 ovn azure ha 1010 0s 0.00% 26.73s 84.60%
host-to-service-new-connections 4.21 amd64 ovn metal compact 106 0s 0.00% 950ms 763.64%
host-to-host-reused-connections 4.21 amd64 ovn metal compact 106 59m29.25s 0.00% 2h10m31.65s 58.97%
cache-oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 830 0s 0.00% 22.13s 70.23%
pod-to-host-new-connections 4.21 amd64 ovn metal compact 106 1h52m30.75s 0.00% 2h50m49.65s 68.18%
openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 830 0s 0.00% 22.42s 58.78%
oauth-api-reused-connections 4.21 4.21 amd64 ovn azure ha 705 1s 0.00% 1.96s 100.00%
pod-to-service-new-connections 4.21 4.21 amd64 ovn vsphere ha 102 0s 0.00% 1s 376.19%
openshift-api-http2-internal-lb-new-connections 4.21 4.20 amd64 ovn gcp ha 830 0s 0.00% 37.97s 65.09%
cache-openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 830 0s 0.00% 23.26s 71.53%
cache-openshift-api-reused-connections 4.21 amd64 ovn metal compact 106 1s 0.00% 2s 100.00%
openshift-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1067 26s 0.00% 1m14.34s 112.16%
openshift-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1067 1m5s 0.00% 1m45s 34.89%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1010 27.55s 0.00% 2m44.01s 258.02%
oauth-api-reused-connections 4.21 amd64 ovn metal compact 106 1s 0.00% 2s 100.00%
cache-oauth-api-reused-connections 4.21 amd64 ovn metal compact 106 1s 0.00% 2s 80.18%
aws-network-liveness-new-connections 4.21 multi ovn aws ha 205 0s 0.00% 960ms 140.00%
kube-api-http2-localhost-reused-connections 4.21 amd64 ovn metal compact 106 25m54.5s 0.00% 54m53.1s 81.41%
openshift-api-reused-connections 4.21 amd64 ovn metal compact 106 1s 0.00% 2s 100.00%
ingress-to-console-reused-connections 4.21 amd64 ovn aws external 839 0s 0.00% 13.24s 32.40%
kube-api-http1-internal-lb-reused-connections 4.21 4.21 amd64 ovn gcp ha 1067 1s 0.00% 2.34s 134.00%
openshift-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1121 2s 0.00% 8.8s 46.67%
openshift-api-http2-internal-lb-reused-connections 4.21 4.20 amd64 ovn azure ha 1010 1s 0.00% 2s 100.00%
kube-api-new-connections 4.21 4.20 amd64 ovn aws ha 1121 0s 0.00% 5.8s 45.00%
pod-to-service-new-connections 4.21 4.21 amd64 ovn gcp ha 1067 1s 0.00% 19.34s 1893.81%
openshift-api-new-connections 4.21 4.20 amd64 ovn aws ha 1121 1s 0.00% 8s 38.65%
pod-to-service-new-connections 4.21 amd64 ovn metal compact 106 0s 0.00% 3.8s 3354.55%
pod-to-host-reused-connections 4.21 4.20 amd64 ovn azure ha 1010 0s 0.00% 15.91s 76.78%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn aws ha 730 55s 0.00% 2m9.71s 4388.24%
metrics-api-new-connections 4.21 4.20 amd64 ovn gcp ha 830 0s 0.00% 32s 113.33%
image-registry-reused-connections 4.21 4.20 amd64 ovn gcp ha 830 1s 0.00% 2s 100.00%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 149 47.4s 0.00% 3m48.4s 7999.29%
image-registry-reused-connections 4.21 4.21 amd64 ovn gcp ha 1067 1s 0.00% 2s 106.19%
pod-to-host-new-connections 4.21 s390x ovn none ha 148 1.65s 0.00% 8m12.8s 51773.68%
openshift-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 730 2m19.65s 0.00% 4m14.55s 5105.52%
kube-api-reused-connections 4.21 amd64 ovn metal compact 106 1s 0.00% 2s 100.00%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1121 2m27s 0.00% 6m15.2s 628.97%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1067 18.7s 0.00% 27.34s 2634.00%
ingress-to-console-new-connections 4.21 amd64 ovn none single 100 1m25s 0.00% 1m25s 1431.53%
ingress-to-console-new-connections 4.21 amd64 ovn aws external 839 0s 0.00% 13.62s 36.20%
kube-api-http1-service-network-new-connections 4.21 4.21 amd64 ovn gcp ha 1067 17.7s 0.00% 38.34s 3734.00%
host-to-host-new-connections 4.21 4.21 amd64 ovn aws ha 730 0s 0.00% 11.97s 1134.02%
openshift-api-http2-internal-lb-new-connections 4.21 4.20 amd64 ovn azure ha 1010 0s 0.00% 1s 170.27%
cache-openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1067 14s 0.00% 28.34s 2734.00%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 1067 2m6.7s 0.00% 5m48.38s 1045.99%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 730 27s 0.00% 45.55s 1476.12%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1121 1m51s 0.00% 4m11s 740.87%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1010 34s 0.00% 4m12.59s 229.84%
openshift-api-http2-service-network-new-connections 4.21 4.20 amd64 ovn gcp ha 830 0s 0.00% 39.42s 50.92%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1067 23.1s 0.00% 1m7.34s 6634.00%
kube-api-http1-localhost-reused-connections 4.21 amd64 ovn metal compact 106 29m46.5s 0.00% 59m40.45s 79.74%
cache-kube-api-reused-connections 4.21 amd64 ovn metal compact 106 1s 0.00% 2s 80.18%

alerts Information

There were (17164) added jobs and (4088) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 32 jobs increased and 205 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn azure ha 214 12m28s 405.41% 18m46.3s 196.71%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn gcp ha 268 11m7s 465.25% 19m7s 47.83%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn aws ha 225 9m46s 170.05% 19m52.72s 41.03%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 4.21 amd64 ovn aws ha 225 13m0s 53.24% 15m32.16s 59.45%
CannotRetrieveUpdates openshift-cluster-version Warning 4.21 amd64 ovn vsphere ha 247 5m28s 57.69% 11m16.6s 86.49%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 128 3m34.2s 43.28% 15m7.94s 193.45%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn vsphere ha 247 0s 0.00% 2m37.6s 3417.86%
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn vsphere ha 247 3h10m55.6s 0.00% 6h18m55.84s 62.57%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn gcp ha 268 1m17.5s 0.00% 2m58s 535.71%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 414 0s 0.00% 4m50.2s 400.34%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 128 0s 0.00% 1m33.7s 6231.08%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn gcp ha 188 4m16s 0.00% 25m1.9s 54.68%
CsvAbnormalFailedOver2Min openshift-operator-lifecycle-manager Warning 4.21 4.20 amd64 ovn azure ha 249 0s 0.00% 2m58s 75.44%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn aws ha 225 58s 0.00% 2m19.28s 191.38%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn gcp ha 188 37.7s 0.00% 4m58.68s 35.15%
OVNKubernetesResourceRetryFailure openshift-ovn-kubernetes Warning 4.21 amd64 ovn metal ha 332 0s 0.00% 8m5.54s 136.39%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn metal ha 332 0s 0.00% 4m56s 43.34%
ClusterOperatorDegraded openshift-cluster-version Warning 4.21 4.21 amd64 ovn gcp ha 268 0s 0.00% 1h11m51.9s 95.01%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 414 0s 0.00% 1m44.56s 99.69%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn azure ha 158 4m2.5s 0.00% 12m19.6s 573.96%
/cc @xueqzhan, @neisw, @smg247

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-bot openshift-bot force-pushed the no-issue--automated---update-synthetic-test-data branch from 4ad3ea2 to d4723f7 Compare January 2, 2026 01:04
@neisw
Copy link
Contributor

neisw commented Jan 5, 2026

/test unit

@openshift-bot openshift-bot force-pushed the no-issue--automated---update-synthetic-test-data branch from d4723f7 to 1da0ca1 Compare January 9, 2026 01:03
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 9, 2026

@openshift-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-serial-1of2 bea3310 link true /test e2e-aws-ovn-serial-1of2
ci/prow/e2e-metal-ipi-ovn-ipv6 bea3310 link true /test e2e-metal-ipi-ovn-ipv6
ci/prow/unit 1da0ca1 link true /test unit

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants