Skip to content

fix(KFLUXVNGD-1267): Move disaster-recovery empty-base under staging/ to match ApplicationSet path - #13541

Merged
openshift-merge-bot[bot] merged 1 commit into
redhat-appstudio:mainfrom
avi-biton:disable-disaster-recovery
Aug 18, 2026
Merged

fix(KFLUXVNGD-1267): Move disaster-recovery empty-base under staging/ to match ApplicationSet path#13541
openshift-merge-bot[bot] merged 1 commit into
redhat-appstudio:mainfrom
avi-biton:disable-disaster-recovery

Conversation

@avi-biton

Copy link
Copy Markdown
Contributor

The disaster-recovery ApplicationSet resolves paths as components/disaster-recovery/{{environment}}/{{clusterDir}}, defaulting to environment=staging and clusterDir=empty-base for clusters not explicitly listed. The empty-base directory was at the wrong level (components/disaster-recovery/empty-base/) so the fallback path components/disaster-recovery/staging/empty-base/ did not exist.

This caused ArgoCD to report errors for clusters where disaster-recovery should not be deployed (e.g. lightwell-dev). Moving empty-base into the staging/ directory aligns the file layout with the ApplicationSet path template, giving non-targeted clusters a valid empty kustomization that deploys nothing.

Checks:

  • empty-base directory lives at components/disaster-recovery/staging/empty-base/
  • No per-cluster overlay exists at components/disaster-recovery/staging/lightwell-dev/
  • kustomize build components/disaster-recovery/staging/empty-base/ succeeds with zero resources

Assisted-by: Cursor (claude-4.6-opus)

@github-actions

github-actions Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Kustomize Render Diff

Comparing a7db15802384efdd95

Component Environment Changes
components/disaster-recovery/staging/empty-base staging build error

Total: 1 components, +0 -0 lines

📋 Full diff available in the workflow summary and as a downloadable artifact.

@qodo-for-redhat-appstudio

Copy link
Copy Markdown

PR Summary by Qodo

Fix ArgoCD DR ApplicationSet fallback by moving empty-base under staging/

🐞 Bug fix ⚙️ Configuration changes 🕐 10-20 Minutes

Grey Divider

AI Description

• Move disaster-recovery empty-base overlay under staging/ to match ApplicationSet path
 template.
• Provide a valid “deploy nothing” kustomization for clusters not explicitly targeted.
• Prevent ArgoCD sync errors caused by missing fallback path directories.
Diagram

graph TD
  A["ArgoCD"] --> B["ApplicationSet: disaster-recovery"] --> C["Path template: {{env}}/{{clusterDir}}"] --> D["staging/empty-base"] --> E["kustomize build"] --> F["Zero resources"]
  C --> G["staging/<cluster overlay>"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Change ApplicationSet defaults to point to previous empty-base location
  • ➕ Avoids moving directories in the repo
  • ➖ Keeps an inconsistent directory layout versus {{environment}}/{{clusterDir}} structure
  • ➖ Higher long-term confusion/maintenance risk
2. Enumerate all excluded clusters with explicit no-op overlays
  • ➕ Makes exclusions explicit per cluster
  • ➖ Doesn’t scale as clusters change
  • ➖ More boilerplate than a single correct fallback path

Recommendation: The chosen approach is the most maintainable: keep the ApplicationSet template consistent and place empty-base where the default path expects it. This resolves ArgoCD errors for non-targeted clusters while guaranteeing the fallback deploys nothing.

Files changed (1) +0 / -0

Bug fix (1) +0 / -0
kustomization.yamlCreate empty-base kustomization at staging fallback path +0/-0

Create empty-base kustomization at staging fallback path

• Adds an explicit empty kustomization ('resources: []') under 'components/disaster-recovery/staging/empty-base/'. This provides a valid fallback target for the ApplicationSet defaults so ArgoCD can sync excluded clusters without errors or deployed resources.

components/disaster-recovery/staging/empty-base/kustomization.yaml

@qodo-for-redhat-appstudio

qodo-for-redhat-appstudio Bot commented Aug 16, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Tip of the day
💡 Did you know, you can keep summaries lean with Finding overflow, which tucks the rest behind 'View more'

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Previous review results

Review updated until commit b2dbf8e ⚖️ Balanced

Results up to commit bbe479e ⚖️ Balanced


No changes from previous review

Qodo Logo

@konflux-ci-qe-bot

konflux-ci-qe-bot commented Aug 16, 2026

Copy link
Copy Markdown

🤖 Pipeline Failure Analysis

Category: Configuration

The Pipelines-as-Code controller failed to validate GitHub webhook payload signatures, preventing the triggering of new PipelineRuns after a disaster recovery restore.

📋 Technical Details

Immediate Cause

The Pipelines-as-Code controller failed to validate incoming GitHub webhook payload signatures, as evidenced by "payload signature check failed" errors in the logs. This directly blocked the controller from processing webhook events and minting GitHub App tokens.

Contributing Factors

This issue likely stemmed from an invalid or out-of-sync webhook secret following the disaster recovery restore. The restoration process potentially failed to correctly re-establish the necessary shared secret between GitHub and the Pipelines-as-Code controller.

Impact

The inability to process webhook events meant that no new PipelineRuns were triggered in the dr-test-kokohazamar-same-version-dr-managed namespace. Consequently, the test timed out after 3600 seconds, as it expected 6-12 successful release PipelineRuns but found zero.

🔍 Evidence

appstudio-konflux-disaster-recovery/redhat-appstudio-konflux-disaster-recovery

Category: configuration
Root Cause: The Pipelines-as-Code controller failed to validate GitHub webhook payload signatures after the disaster recovery restore, which prevented new PipelineRuns from being triggered. This was likely due to an invalid or out-of-sync webhook secret.

Logs:

artifacts/appstudio-konflux-disaster-recovery/redhat-appstudio-konflux-disaster-recovery/build-log.txt line 976
[FAILED] Timed out after 3600.000s.
artifacts/appstudio-konflux-disaster-recovery/redhat-appstudio-konflux-disaster-recovery/build-log.txt line 977
expected 6–12 successful release PipelineRuns in namespace dr-test-kokohazamar-same-version-dr-managed (got overshoot beyond 2x tolerance)
artifacts/appstudio-konflux-disaster-recovery/redhat-appstudio-konflux-disaster-recovery/build-log.txt line 978
Expected <int>: 0 to be >= <int>: 6
artifacts/appstudio-konflux-disaster-recovery/redhat-appstudio-konflux-disaster-recovery/build-log.txt line 979
In [It] at: /tmp/infra-deployments/tests/disaster-recovery/dr_same_version.go:147 @ 08/17/26 09:41:39.382
artifacts/appstudio-konflux-disaster-recovery/redhat-appstudio-konflux-disaster-recovery/build-log.txt line 630
{"level":"error","ts":"2026-08-17T09:24:49.398Z","logger":"pipelinesascode","caller":"adapter/sinker.go:39","msg":"failed to parse event: payload signature check failed","commit":"[REDACTED: Hex High Entropy String]","provider":"github","event-id":"7e1ad578-9a1d-11f1-8e02-27b1609fc499","stacktrace":"github.com/openshift-pipelines/pipelines-as-code/pkg/adapter.(*sinker).processEventPayload\n\t/go/src/github.com/openshift-pipelines/pipelines-as-code/pkg/adapter/sinker.go:39\ngithub.com/openshift-pipelines/pipelines-as-code/pkg/adapter.(*sinker).processEvent\n\t/go/src/github.com/openshift-pipelines/pipelines-as-code/pkg/adapter/sinker.go:75\ngithub.com/openshift-pipelines/pipelines-as-code/pkg/adapter.(*listener).Start.listener.handleEvent.func3.1\n\t/go/src/github.com/openshift-pipelines/pipelines-as-code/pkg/adapter/adapter.go:256"}
artifacts/appstudio-konflux-disaster-recovery/redhat-appstudio-konflux-disaster-recovery/build-log.txt line 633
{"level":"warn","ts":"2026-08-17T09:25:01.645Z","logger":"pipelinesascode","caller":"github/parse_payload.go:240","msg":"[SECURITY] Blocked GitHub App token minting before webhook signature validation completed","commit":"[REDACTED: Hex High Entropy String]","provider":"github","event-id":"8568e1f8-9a1d-11f1-9bc5-3b22c832165b","severity":"warning","security-impact":"github-app-token-mint-blocked","reason":"webhook-signature-validation-failed","error":"payload signature check failed","event-type":"push","installation-id":72200095,"github-enterprise-host-present":false,"remote-addr":"10.128.2.24:42570"}

Analysis powered by prow-failure-analysis | Build: 2089256476047577088

@gbenhaim

Copy link
Copy Markdown
Member

/test appstudio-konflux-disaster-recovery

@qodo-for-redhat-appstudio

qodo-for-redhat-appstudio Bot commented Aug 17, 2026

Copy link
Copy Markdown

Unsupported PR languages

@avi-biton

Copy link
Copy Markdown
Contributor Author

/test appstudio-konflux-disaster-recovery

@qodo-for-redhat-appstudio

qodo-for-redhat-appstudio Bot commented Aug 17, 2026

Copy link
Copy Markdown

Unsupported PR languages

@redhat-appstudio-qe-bot

Copy link
Copy Markdown

❗ Detected an outage of the following critical component(s)❗

  • github: API Requests, Pull Requests

Due to this issue E2E tests will probably fail. Please keep an eye on the following status pages:

and add a comment /retest-required once the reported issues are solved

… to match ApplicationSet path

The disaster-recovery ApplicationSet resolves paths as
components/disaster-recovery/{{environment}}/{{clusterDir}}, defaulting
to environment=staging and clusterDir=empty-base for clusters not
explicitly listed. The empty-base directory was at the wrong level
(components/disaster-recovery/empty-base/) so the fallback path
components/disaster-recovery/staging/empty-base/ did not exist.

This caused ArgoCD to report errors for clusters where
disaster-recovery should not be deployed (e.g. lightwell-dev).
Moving empty-base into the staging/ directory aligns the file layout
with the ApplicationSet path template, giving non-targeted clusters
a valid empty kustomization that deploys nothing.

Assisted-by: Cursor (claude-4.6-opus)
Signed-off-by: Avi Biton <abiton@redhat.com>
@avi-biton
avi-biton force-pushed the disable-disaster-recovery branch from bbe479e to b2dbf8e Compare August 18, 2026 06:24
@qodo-for-redhat-appstudio

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit b2dbf8e

@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.38%. Comparing base (a7db158) to head (b2dbf8e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main   #13541   +/-   ##
=======================================
  Coverage   60.38%   60.38%           
=======================================
  Files          24       24           
  Lines        1628     1628           
=======================================
  Hits          983      983           
  Misses        563      563           
  Partials       82       82           
Flag Coverage Δ
go 60.38% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@manish-jangra manish-jangra left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve
/lgtm

@openshift-ci

openshift-ci Bot commented Aug 18, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: avi-biton, manish-jangra

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot
openshift-merge-bot Bot merged commit fd23bf2 into redhat-appstudio:main Aug 18, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants