CMP-3882: Create an e2e suite for CEL/CustomRule tests

[Anna-Koudelkova]

Third attempt to start with the refactor based on feature groups. Not complete as we have not finished porting all the test cases upstream and also I need some guidance on importance of the test cases.

This is heavily coming from the refactor doc, trying to cover the CEL scanner and CustomRule tests.

1. Test case execution order
   One thing I have found out is that golang would go through all the test cases and when it is parallel, it puts it on hold and run the serial test cases one by one first and the parallel test cases gets to run last.

make e2e-cel-critical   E2E_CONTENT_IMAGE_PATH="quay.io/rh-ee-akoudelk/content_repo:testcontent"  E2E_BROKEN_CONTENT_IMAGE_PATH="ghcr.io/complianceascode/test-broken-content-ocp"  E2E_GO_TEST_FLAGS="-v -timeout 60m"

=== RUN   TestCustomRuleTailoredProfile
=== PAUSE TestCustomRuleTailoredProfile
=== RUN   TestCustomRuleWithMultipleInputs
=== PAUSE TestCustomRuleWithMultipleInputs
=== RUN   TestCustomRuleValidation
=== PAUSE TestCustomRuleValidation
=== RUN   TestCustomRuleCheckTypeAndScannerTypeValidation
    main_test.go:557: Skipping non-critical test
--- SKIP: TestCustomRuleCheckTypeAndScannerTypeValidation (0.00s)
=== RUN   TestCustomRuleFailureReasonInCheckResult
=== PAUSE TestCustomRuleFailureReasonInCheckResult
=== RUN   TestCustomRuleCascadingStatusUpdate
    main_test.go:999: Skipping non-critical test
--- SKIP: TestCustomRuleCascadingStatusUpdate (0.00s)
=== CONT  TestCustomRuleTailoredProfile
=== CONT  TestCustomRuleValidation
=== CONT  TestCustomRuleFailureReasonInCheckResult
=== CONT  TestCustomRuleWithMultipleInputs
2026/03/13 10:50:50 CustomRule ready (Ready)
=== NAME  TestCustomRuleFailureReasonInCheckResult
    main_test.go:795: CustomRule test-custom-rule-failure-reason-in-check-result-replica-check is ready
2026/03/13 10:50:50 CustomRule ready (Error)
=== NAME  TestCustomRuleValidation
    main_test.go:502: CustomRule validation correctly rejected invalid expression
2026/03/13 10:50:50 CustomRule ready (Ready)
2026/03/13 10:50:50 CustomRule ready (Ready)
2026/03/13 10:50:56 TailoredProfile ready (READY)
=== NAME  TestCustomRuleFailureReasonInCheckResult
    main_test.go:830: TailoredProfile test-custom-rule-failure-reason-in-check-result-tp is ready
2026/03/13 10:50:56 CustomRule ready (Error)
=== NAME  TestCustomRuleValidation
    main_test.go:550: CustomRule validation correctly detected undeclared variable
2026/03/13 10:50:56 ScanSettingBinding status READY
=== NAME  TestCustomRuleWithMultipleInputs
    main_test.go:417: ScanSettingBinding test-custom-rule-with-multiple-inputs-ssb is now ready
--- PASS: TestCustomRuleValidation (10.86s)
2026/03/13 10:50:56 waiting until suite test-custom-rule-tailored-profile-ssb reaches target status 'DONE'. Current status: PENDING
2026/03/13 10:51:01 waiting until suite test-custom-rule-failure-reason-in-check-result-ssb reaches target status 'DONE'. Current status: PENDING
2026/03/13 10:51:01 waiting until suite test-custom-rule-tailored-profile-ssb reaches target status 'DONE'. Current status: PENDING
2026/03/13 10:51:01 waiting until suite test-custom-rule-with-multiple-inputs-ssb reaches target status 'DONE'. Current status: RUNNING
2026/03/13 10:51:06 Waiting for scan test-custom-rule-tailored-profile-tp to appear in suite: test-custom-rule-tailored-profile-ssb
2026/03/13 10:51:06 waiting until suite test-custom-rule-failure-reason-in-check-result-ssb reaches target status 'DONE'. Current status: PENDING
2026/03/13 10:51:11 Waiting for scan test-custom-rule-failure-reason-in-check-result-tp to appear in suite: test-custom-rule-failure-reason-in-check-result-ssb
2026/03/13 10:51:11 waiting until suite test-custom-rule-tailored-profile-ssb reaches target status 'DONE'. Current status: LAUNCHING
2026/03/13 10:51:11 ComplianceScan ready (DONE)
2026/03/13 10:51:11 All scans in ComplianceSuite have finished (test-custom-rule-with-multiple-inputs-ssb)
=== NAME  TestCustomRuleWithMultipleInputs
    main_test.go:445: CustomRule with multiple inputs test completed successfully.
    main_test.go:446:   - Validated that rule test-custom-rule-with-multiple-inputs-network-policy has FAIL status
--- PASS: TestCustomRuleWithMultipleInputs (27.06s)
2026/03/13 10:51:16 waiting until suite test-custom-rule-failure-reason-in-check-result-ssb reaches target status 'DONE'. Current status: PENDING
2026/03/13 10:51:16 waiting until suite test-custom-rule-tailored-profile-ssb reaches target status 'DONE'. Current status: RUNNING
2026/03/13 10:51:21 waiting until suite test-custom-rule-tailored-profile-ssb reaches target status 'DONE'. Current status: RUNNING
2026/03/13 10:51:21 waiting until suite test-custom-rule-failure-reason-in-check-result-ssb reaches target status 'DONE'. Current status: PENDING
2026/03/13 10:51:26 waiting until suite test-custom-rule-failure-reason-in-check-result-ssb reaches target status 'DONE'. Current status: PENDING
2026/03/13 10:51:31 waiting until suite test-custom-rule-failure-reason-in-check-result-ssb reaches target status 'DONE'. Current status: RUNNING
2026/03/13 10:51:31 ComplianceScan ready (DONE)
2026/03/13 10:51:31 All scans in ComplianceSuite have finished (test-custom-rule-tailored-profile-ssb)
2026/03/13 10:51:32 Triggered rescan for scan test-custom-rule-tailored-profile-tp
2026/03/13 10:51:36 waiting until suite test-custom-rule-failure-reason-in-check-result-ssb reaches target status 'DONE'. Current status: RUNNING
2026/03/13 10:51:37 waiting until suite test-custom-rule-tailored-profile-ssb reaches target status 'DONE'. Current status: RUNNING
2026/03/13 10:51:46 ComplianceScan ready (DONE)
2026/03/13 10:51:46 All scans in ComplianceSuite have finished (test-custom-rule-failure-reason-in-check-result-ssb)
=== NAME  TestCustomRuleFailureReasonInCheckResult
    main_test.go:871: Scan completed as non-compliant (some deployments have <3 replicas)
    main_test.go:901: FailureReason correctly appears in ComplianceCheckResult warnings: One or more deployments have less than 3 replicas, which violates the high availability requirement
2026/03/13 10:51:47 Triggered rescan for scan test-custom-rule-failure-reason-in-check-result-tp
2026/03/13 10:51:47 ComplianceScan ready (DONE)
2026/03/13 10:51:47 All scans in ComplianceSuite have finished (test-custom-rule-tailored-profile-ssb)
=== NAME  TestCustomRuleTailoredProfile
    main_test.go:268: Created pod without label that should be ignored: test-custom-rule-tailored-profile-ignored-pod
    main_test.go:269: Test completed successfully. CustomRule correctly:
    main_test.go:270:   - Identified non-compliant pod with the test label
    main_test.go:271:   - Ignored pods without the test label
    main_test.go:272:   - Validated that rule test-custom-rule-tailored-profile-security-context has FAIL status
--- PASS: TestCustomRuleTailoredProfile (63.16s)
2026/03/13 10:51:53 waiting until suite test-custom-rule-failure-reason-in-check-result-ssb reaches target status 'DONE'. Current status: RUNNING
2026/03/13 10:52:03 ComplianceScan ready (DONE)
2026/03/13 10:52:03 All scans in ComplianceSuite have finished (test-custom-rule-failure-reason-in-check-result-ssb)
=== NAME  TestCustomRuleFailureReasonInCheckResult
    main_test.go:992: FailureReason correctly appears in ComplianceCheckResult warnings: One or more deployments have less than 3 replicas, which violates the high availability requirement
    main_test.go:993: TestCustomRuleFailureReasonInCheckResult completed successfully
--- PASS: TestCustomRuleFailureReasonInCheckResult (78.46s)
PASS

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Anna-Koudelkova

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [Anna-Koudelkova]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@Anna-Koudelkova: This pull request references CMP-3882 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Third attempt to start with the refactor based on feature groups. Not complete as we have not finished porting all the test cases upstream and also I need some guidance on importance of the test cases.

This is heavily coming from the refactor doc, trying to cover the CEL scanner and CustomRule tests.

1. Test case execution order
   One thing I have found out is that golang would go through all the test cases and when it is parallel, it puts it on hold and run the serial test cases one by one first and the parallel test cases gets to run last.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1109-0407ec424efb566dd26dd9f640821b2abb3d33c0