SECURESIGN-3167 CTLog recovery

[petrpinkas]

User description

https://issues.redhat.com/browse/SECURESIGN-3167

Summary by Sourcery

Enable CTLog operator to self-heal missing or invalid server configuration secrets by validating and regenerating them, supported by a new validation utility and comprehensive e2e recovery tests.

New Features:

• Introduce automated self-healing for CTLog configuration secrets by detecting and recreating missing or invalid secrets
• Add IsSecretDataValid utility to validate CTLog config secrets against the expected Trillian backend address

Enhancements:

• Always revalidate custom server config secrets in the serverConfig handler and trigger recreation on mismatch

Tests:

• Add end-to-end integration test for CTLog recovery flow, including secret deletion, automatic reconciliation, and service readiness verification with helper functions for secret management

---

PR Type

Enhancement, Tests

---

Description

• Add CTLog config secret validation and self-healing recovery mechanism

• Implement automatic detection and recreation of missing or invalid config secrets

• Add integration tests for CTLog recovery scenarios with Trillian address validation

• Introduce helper functions for config secret management and Trillian address extraction

---

Diagram Walkthrough

flowchart LR
  A["CTLog CR"] -->|"spec.Trillian"| B["serverConfig.Handle"]
  B -->|"validate secret"| C["IsSecretDataValid"]
  C -->|"missing/invalid"| D["Recreate Secret"]
  D -->|"with correct config"| E["Secret with Trillian Address"]
  E -->|"verify"| F["CTLog Ready"]
  B -->|"test scenarios"| G["Integration Tests"]
  G -->|"validate recovery"| H["Config Restored"]

Loading

File Walkthrough

	Relevant files
Enhancement	server_config.go Add config secret validation and recovery logic                    internal/controller/ctlog/actions/server_config.go Modified CanHandle() to always validate secret existence and validity instead of checking generation Added validation for custom server config secrets with error handling for missing or invalid configurations Implemented secret validation logic that checks for missing secrets and validates Trillian address configuration Added recovery mechanism to detect and recreate invalid or missing config secrets with proper logging and events +57/-1    ctlog_config.go Add secret validation helper function                                        internal/controller/ctlog/utils/ctlog_config.go Added IsSecretDataValid() function to validate CTLog config secrets contain correct Trillian backend address Validates secret data exists, contains required config key, and has non-empty configuration Performs string parsing to extract and verify Trillian address from config data Used by operator for self-healing to detect missing or invalid configurations +35/-0
Tests	ctlog_recovery_test.go Add CTLog recovery integration tests                                          test/e2e/ctlog_recovery_test.go Comprehensive integration test suite for CTLog recovery and validation scenarios Tests config secret creation with correct Trillian address configuration Simulates disaster scenario by deleting config secret and verifies automatic recreation Validates operator detects missing/invalid secrets and recreates them with correct configuration Verifies TreeID preservation and CTLog deployment readiness after recovery +247/-0  ctlog.go Add config secret test helper functions                                    test/e2e/support/tas/ctlog/ctlog.go Added GetConfigSecret() helper to retrieve config secrets by name Added DeleteConfigSecret() helper to delete config secrets for testing Added GetTrillianAddressFromSecret() helper to extract Trillian address from config secret data Supports test scenarios for config secret management and validation +37/-0

---

[sourcery-ai]

Reviewer's Guide

This PR enhances the CTLog operator to self-heal its configuration secret by validating and recreating it when missing or invalid, adds utility and parsing helpers for managing CTLog config secrets in tests and controller logic, and introduces end-to-end integration tests that simulate secret loss and verify operator recovery behavior and TreeID preservation.

Sequence diagram for CTLog config secret validation and recovery

sequenceDiagram
participant Operator
participant Kubernetes
participant Secret
Operator->>Kubernetes: Get CTLog config secret
alt Secret not found
    Operator->>Operator: Log info (missing secret)
    Operator->>Operator: Record event (CTLogConfigMissing)
    Operator->>Kubernetes: Recreate config secret
else Error accessing secret
    Operator->>Operator: Log error
    Operator->>Operator: Record event (CTLogConfigError)
    Operator->>Kubernetes: Recreate config secret
else Secret exists
    Operator->>Secret: Validate secret data
    alt Secret valid
        Operator->>Operator: Continue (no action)
    else Secret invalid
        Operator->>Operator: Log info (invalid secret)
        Operator->>Operator: Record event (CTLogConfigInvalid)
        Operator->>Kubernetes: Recreate config secret
    end
end

Loading

Class diagram for CTLog config secret validation helper

classDiagram
class ctlogUtils {
  +IsSecretDataValid(secretData map[string][]byte, expectedTrillianAddr string) bool
}

ctlogUtils : IsSecretDataValid checks secret data for valid Trillian address

Loading

File-Level Changes

Change	Details	Files
Enhance server_config handler for CTLog to always validate and self-heal its config secret	CanHandle() default now always returns true to trigger validation Added import of apierrors for NotFound check Validate custom secret existence and data key before status update Check existing secret in Status.ServerConfigRef, detect missing or invalid data and emit events Use IsSecretDataValid to decide if secret matches expected Trillian address or must be recreated	internal/controller/ctlog/actions/server_config.go
Add utility functions in test support for config secret management and parsing	Implement GetConfigSecret() to fetch a config secret Implement DeleteConfigSecret() to delete a config secret Implement GetTrillianAddressFromSecret() to extract raw config string for address matching	test/e2e/support/tas/ctlog/ctlog.go
Introduce IsSecretDataValid and string-parsing helpers in CTLog utils	Import strings package Implement IsSecretDataValid() to scan config lines for backend_spec containing expected address	internal/controller/ctlog/utils/ctlog_config.go
Introduce comprehensive e2e integration tests for CTLog recovery	Simulate missing or invalid config by deleting secret Trigger reconciliation via CR annotation Assert operator recreates secret with correct Trillian address Verify CTLog deployment, pod readiness, Ready status, and TreeID preservation	test/e2e/ctlog_recovery_test.go

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[qodo-merge-pro]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Weak config validation Description: Secret validation parses the config using simple substring checks on "backend_spec" and expected address, which could be bypassed by crafted content or formatting differences, potentially allowing acceptance of an invalid configuration. ctlog_config.go [215-236] Referred Code func IsSecretDataValid(secretData map[string][]byte, expectedTrillianAddr string) bool { if secretData == nil { return false } configData, ok := secretData[ConfigKey] if !ok { return false } configString := string(configData) if len(configString) == 0 { return false } for _, line := range strings.Split(configString, "\n") { if strings.Contains(line, "backend_spec") && strings.Contains(line, expectedTrillianAddr) { return true } } ... (clipped 1 lines)
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Passed
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Passed
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Limited auditing: New controller logic performs secret validation and recreation but only emits basic events and logs without structured, comprehensive audit details of actions taken and outcomes. Referred Code // Validate existing secret before attempting recreation if instance.Status.ServerConfigRef != nil && instance.Status.ServerConfigRef.Name != "" { secret, err := kubernetes.GetSecret(i.Client, instance.Namespace, instance.Status.ServerConfigRef.Name) if err != nil { if apierrors.IsNotFound(err) { i.Logger.Info("Server config secret is missing, will recreate", "secret", instance.Status.ServerConfigRef.Name) i.Recorder.Event(instance, corev1.EventTypeWarning, "CTLogConfigMissing", fmt.Sprintf("Config secret is missing, will recreate: %s", instance.Status.ServerConfigRef.Name)) } else { i.Logger.Error(err, "Error accessing server config secret, will attempt to recreate", "secret", instance.Status.ServerConfigRef.Name) i.Recorder.Event(instance, corev1.EventTypeWarning, "CTLogConfigError", fmt.Sprintf("Error accessing config secret, will recreate: %s", instance.Status.ServerConfigRef.Name)) } } else { // Secret exists and is accessible - validate it expectedTrillianAddr := fmt.Sprintf("%s:%d", instance.Spec.Trillian.Address, *instance.Spec.Trillian.Port) if ctlogUtils.IsSecretDataValid(secret.Data, expectedTrillianAddr) { return i.Continue() // nothing to do ... (clipped 9 lines)
	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Fragile parsing: Secret validation relies on simple substring matching of 'backend_spec' which may miss edge cases or alternative formatting and could yield false negatives/positives. Referred Code func IsSecretDataValid(secretData map[string][]byte, expectedTrillianAddr string) bool { if secretData == nil { return false } configData, ok := secretData[ConfigKey] if !ok { return false } configString := string(configData) if len(configString) == 0 { return false } for _, line := range strings.Split(configString, "\n") { if strings.Contains(line, "backend_spec") && strings.Contains(line, expectedTrillianAddr) { return true } } ... (clipped 1 lines)
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Unstructured logs: The new log and event messages are unstructured strings and include secret names, which may be sensitive identifiers and harder to audit programmatically. Referred Code if err != nil { if apierrors.IsNotFound(err) { i.Logger.Info("Server config secret is missing, will recreate", "secret", instance.Status.ServerConfigRef.Name) i.Recorder.Event(instance, corev1.EventTypeWarning, "CTLogConfigMissing", fmt.Sprintf("Config secret is missing, will recreate: %s", instance.Status.ServerConfigRef.Name)) } else { i.Logger.Error(err, "Error accessing server config secret, will attempt to recreate", "secret", instance.Status.ServerConfigRef.Name) i.Recorder.Event(instance, corev1.EventTypeWarning, "CTLogConfigError", fmt.Sprintf("Error accessing config secret, will recreate: %s", instance.Status.ServerConfigRef.Name)) } } else { // Secret exists and is accessible - validate it expectedTrillianAddr := fmt.Sprintf("%s:%d", instance.Spec.Trillian.Address, *instance.Spec.Trillian.Port) if ctlogUtils.IsSecretDataValid(secret.Data, expectedTrillianAddr) { return i.Continue() // nothing to do } // Secret is invalid, will recreate i.Logger.Info("Server config secret is invalid, will recreate", "secret", secret.Name, ... (clipped 5 lines)
Update

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[sourcery-ai]

Hey there - I've reviewed your changes - here's some feedback:

• The IsSecretDataValid helper relies on naive string matching of the protobuf text output, which can be brittle—consider using a TextUnmarshaler or protobuf parser to robustly extract and validate the backend_spec field.
• CanHandle always returning true forces reconciliation even when nothing has changed; refining its logic (e.g., comparing observed generation or secret checksums) could reduce unnecessary reconcile loops.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The IsSecretDataValid helper relies on naive string matching of the protobuf text output, which can be brittle—consider using a TextUnmarshaler or protobuf parser to robustly extract and validate the backend_spec field.
- CanHandle always returning true forces reconciliation even when nothing has changed; refining its logic (e.g., comparing observed generation or secret checksums) could reduce unnecessary reconcile loops.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[qodo-merge-pro]

PR Code Suggestions ✨

Latest suggestions up to 018dfb5

Category	Suggestion	Impact
Possible issue	Guard against nil Trillian port Add a nil check for instance.Spec.Trillian.Port before dereferencing it to prevent a potential panic, returning an error and setting a failure condition if it is nil. internal/controller/ctlog/actions/server_config.go [120] +if instance.Spec.Trillian.Port == nil { + return i.Error(ctx, fmt.Errorf("trillian port not specified"), instance, + metav1.Condition{ + Type: ConfigCondition, + Status: metav1.ConditionFalse, + Reason: constants.Failure, + Message: "Trillian port must be specified", + ObservedGeneration: instance.Generation, + }) +} expectedTrillianAddr := fmt.Sprintf("%s:%d", instance.Spec.Trillian.Address, *instance.Spec.Trillian.Port) Apply / Chat Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a potential nil pointer dereference on instance.Spec.Trillian.Port that would crash the controller, and the proposed fix is robust and idiomatic for a Kubernetes controller.	High
Incremental [*]	Strengthen config secret validation Strengthen the IsSecretDataValid function by adding checks for LogConfigs, LogId, RootsPemFile, and BackendOverrides to prevent accepting incomplete or inconsistent configurations. internal/controller/ctlog/utils/ctlog_config.go [219-249] func IsSecretDataValid(secretData map[string][]byte, expectedTrillianAddr string) bool { if secretData == nil { return false } - configData, ok := secretData[ConfigKey] if !ok || len(configData) == 0 { return false } - // Parse the protobuf text format configuration var multiConfig configpb.LogMultiConfig if err := prototext.Unmarshal(configData, &multiConfig); err != nil { - // Failed to parse - invalid configuration return false } - // Validate that at least one backend exists - if multiConfig.Backends == nil || multiConfig.Backends.Backend == nil || len(multiConfig.Backends.Backend) == 0 { + // Validate backends exist and contain expected address + if multiConfig.Backends == nil || len(multiConfig.Backends.Backend) == 0 { + return false + } + foundExpected := false + for _, backend := range multiConfig.Backends.Backend { + if backend.GetBackendSpec() == expectedTrillianAddr { + foundExpected = true + break + } + } + if !foundExpected { return false } - // Check if any backend matches the expected Trillian address - for _, backend := range multiConfig.Backends.Backend { - if backend.BackendSpec == expectedTrillianAddr { - return true + // Validate at least one log config exists + if multiConfig.LogConfigs == nil || len(multiConfig.LogConfigs.Config) == 0 { + return false + } + for _, cfg := range multiConfig.LogConfigs.Config { + // Basic sanity checks on per-log config + if cfg.GetLogId() == 0 { + return false + } + // Ensure roots file path is present (ctfe requires roots) + if cfg.GetRootsPemFile() == "" && cfg.GetRootsPemFilePath() == "" { + return false } } - return false + // Ensure overrides (if any) do not contradict expected backend + if o := multiConfig.BackendOverrides; o != nil { + for _, ov := range o.Override { + if ov.GetBackendSpec() != "" && ov.GetBackendSpec() != expectedTrillianAddr { + return false + } + } + } + return true } Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion proposes valid and useful enhancements to the IsSecretDataValid function, making the configuration validation more robust and capable of catching more edge cases that could lead to runtime failures.	Medium
	Include secret name in events Restore the secret name to the CTLogConfigCreated event message to improve debugging and observability, as was done in the previous version of the code. internal/controller/ctlog/actions/server_config.go [210] -i.Recorder.Event(instance, corev1.EventTypeNormal, "CTLogConfigCreated", "Config secret created successfully") +i.Recorder.Eventf(instance, corev1.EventTypeNormal, "CTLogConfigCreated", "Config secret created successfully: %s", newConfig.Name) Apply / Chat Suggestion importance[1-10]: 5 __ Why: The suggestion correctly points out that removing the secret name from the event message is a regression in observability. Including the resource name is a best practice for creating useful Kubernetes events.	Low
General	Add robust config sanity checks Before unmarshalling the config data, trim whitespace and add checks to ensure the data is not empty or excessively large. internal/controller/ctlog/utils/ctlog_config.go [231-234] -if err := prototext.Unmarshal(configData, &multiConfig); err != nil { - // Failed to parse - invalid configuration - return false - } +// Basic sanity checks to avoid false negatives and resource overuse +trimmed := bytes.TrimSpace(configData) +if len(trimmed) == 0 || len(trimmed) > 1_000_000 { // 1MB safety cap + return false +} +if err := prototext.Unmarshal(trimmed, &multiConfig); err != nil { + // Failed to parse - invalid configuration + return false +} Apply / Chat Suggestion importance[1-10]: 4 __ Why: The suggestion adds useful hardening by checking for empty or excessively large config data, but its impact is minor as the existing code already handles most invalid cases and Kubernetes secrets have a size limit.	Low
Update

---

Previous suggestions

✅ Suggestions up to commit 3e1cde2

Category	Suggestion	Impact
Possible issue	Validate custom server config content Add validation for the Trillian address within the custom server config secret using the ctlogUtils.IsSecretDataValid function to prevent misconfigurations. internal/controller/ctlog/actions/server_config.go [66-100] if instance.Spec.ServerConfigRef != nil { // Validate that the custom secret is accessible secret, err := kubernetes.GetSecret(i.Client, instance.Namespace, instance.Spec.ServerConfigRef.Name) if err != nil { return i.Error(ctx, fmt.Errorf("error accessing custom server config secret: %w", err), instance, metav1.Condition{ Type: ConfigCondition, Status: metav1.ConditionFalse, Reason: constants.Failure, Message: fmt.Sprintf("Error accessing custom server config secret: %s", instance.Spec.ServerConfigRef.Name), ObservedGeneration: instance.Generation, }) } - if secret.Data == nil || secret.Data[ctlogUtils.ConfigKey] == nil { + + expectedTrillianAddr := fmt.Sprintf("%s:%d", instance.Spec.Trillian.Address, *instance.Spec.Trillian.Port) + if !ctlogUtils.IsSecretDataValid(secret.Data, expectedTrillianAddr) { return i.Error(ctx, fmt.Errorf("custom server config secret is invalid"), instance, metav1.Condition{ Type: ConfigCondition, Status: metav1.ConditionFalse, Reason: constants.Failure, - Message: fmt.Sprintf("Custom server config secret is missing '%s' key: %s", ctlogUtils.ConfigKey, instance.Spec.ServerConfigRef.Name), + Message: fmt.Sprintf("Custom server config secret has invalid Trillian address. Expected: %s", expectedTrillianAddr), ObservedGeneration: instance.Generation, }) } instance.Status.ServerConfigRef = instance.Spec.ServerConfigRef i.Recorder.Event(instance, corev1.EventTypeNormal, "CTLogConfigUpdated", "CTLog config updated") meta.SetStatusCondition(&instance.Status.Conditions, metav1.Condition{ Type: ConfigCondition, Status: metav1.ConditionTrue, Reason: constants.Ready, Message: "Using custom server config", ObservedGeneration: instance.Generation, }) return i.StatusUpdate(ctx, instance) } Suggestion importance[1-10]: 8 __ Why: This suggestion addresses a potential bug where a custom config with an invalid Trillian address would be accepted, leading to runtime failures that are not reflected in the operator's status.	Medium
	✅ Use protobuf unmarshaling for validation Suggestion Impact: The commit replaced string-based parsing with prototext.Unmarshal into a protobuf config (LogMultiConfig), validated presence of backends, and compared BackendSpec to the expected address, aligning with the suggestion’s intent. code diff: @@ -218,17 +222,25 @@ } configData, ok := secretData[ConfigKey] - if !ok { - return false - } - - configString := string(configData) - if len(configString) == 0 { - return false - } - - for _, line := range strings.Split(configString, "\n") { - if strings.Contains(line, "backend_spec") && strings.Contains(line, expectedTrillianAddr) { + if !ok || len(configData) == 0 { + return false + } + + // Parse the protobuf text format configuration + var multiConfig configpb.LogMultiConfig + if err := prototext.Unmarshal(configData, &multiConfig); err != nil { + // Failed to parse - invalid configuration + return false + } + + // Validate that at least one backend exists + if multiConfig.Backends == nil || multiConfig.Backends.Backend == nil || len(multiConfig.Backends.Backend) == 0 { + return false + } + + // Check if any backend matches the expected Trillian address + for _, backend := range multiConfig.Backends.Backend { + if backend.BackendSpec == expectedTrillianAddr { return true } Refactor IsSecretDataValid to unmarshal the prototext configuration into a struct for validation, instead of using fragile string matching. internal/controller/ctlog/utils/ctlog_config.go [215-237] func IsSecretDataValid(secretData map[string][]byte, expectedTrillianAddr string) bool { if secretData == nil { return false } configData, ok := secretData[ConfigKey] - if !ok { + if !ok || len(configData) == 0 { return false } - configString := string(configData) - if len(configString) == 0 { + var cfg configpb.LogConfig + if err := prototext.Unmarshal(configData, &cfg); err != nil { return false } - for _, line := range strings.Split(configString, "\n") { - if strings.Contains(line, "backend_spec") && strings.Contains(line, expectedTrillianAddr) { - return true - } + if cfg.GetLogBackend() == nil { + return false } - return false + return cfg.GetLogBackend().GetBackendSpec() == expectedTrillianAddr } [Suggestion processed] Suggestion importance[1-10]: 6 __ Why: The suggestion correctly identifies that string-based parsing of prototext is brittle and proposes a more robust solution using prototext.Unmarshal, which improves code quality and reliability.	Low

[qodo-merge-pro]

CI Feedback 🧐

A test triggered by this PR failed. Here is an AI-generated analysis of the failure:

Action: Build-operator

Failed stage: Build operator container [❌]

Failed test name: CTlog controller test: should successfully reconcile a custom resource for CTlog

Failure summary: The action failed due to multiple test failures in the CTlog controller suite: - Controller e2e tests timed out waiting for the ctlog Deployment to appear: - Error: deployments.apps "ctlog" not found (404) - Locations: - /internal/controller/ctlog/ctlog_controller_test.go:158 ([It] should successfully reconcile a custom resource for CTlog) - /internal/controller/ctlog/ctlog_hot_update_test.go:135 ([It] should successfully reconcile a custom resource for CTlog) - Earlier in reconciliation, an error occurred while creating a Secret in the handle-keys action: - could not generate Secret: resourceVersion should not be set on objects to be created (logged at base_action.go:92), which likely prevented successful deployment creation. - Unit tests for server config logic also failed: - TestServerConfig_CanHandle: expected false, got true (at server_config_test.go:152) - TestServerConfig_Handle: expected non-nil status.serverConfigRef, got nil (at server_config_test.go:189) - TestServerConfig_Handle/use_spec.serverConfigRef: unexpected error secrets "config" not found (at server_config_test.go:479) Overall, the reconciliation never resulted in a ctlog Deployment, causing timeouts and test failures.

Relevant error logs: 1: ##[group]Runner Image Provisioner 2: Hosted Compute Agent ... 290: ok github.com/securesign/operator/internal/action/pvc 0.286s coverage: 93.6% of statements 291: ok github.com/securesign/operator/internal/action/rbac 0.242s coverage: 81.2% of statements 292: github.com/securesign/operator/internal/action/transitions coverage: 0.0% of statements 293: ok github.com/securesign/operator/internal/action/tree 0.634s coverage: 75.9% of statements 294: ? github.com/securesign/operator/internal/annotations [no test files] 295: ? github.com/securesign/operator/internal/apis [no test files] 296: github.com/securesign/operator/internal/clidownload coverage: 0.0% of statements 297: ? github.com/securesign/operator/internal/config [no test files] 298: ? github.com/securesign/operator/internal/constants [no test files] 299: ? github.com/securesign/operator/internal/controller [no test files] 300: Running Suite: Controller Suite - /home/runner/work/secure-sign-operator/secure-sign-operator/internal/controller/ctlog 301: ======================================================================================================================= 302: Random Seed: �[1m1762787241�[0m 303: Will run �[1m2�[0m of �[1m2�[0m specs 304: �[38;5;243m------------------------------�[0m 305: �[38;5;9m• [FAILED] [122.357 seconds]�[0m 306: �[0mCTlog controller �[38;5;243mCTlog controller test �[38;5;9m�[1m[It] should successfully reconcile a custom resource for CTlog�[0m ... 358: I1110 15:07:29.532878 4546 ctlog_controller.go:127] "Executing handle-fulcio-cert" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="89f69070-f409-46b9-bf23-37a25c24e987" 359: I1110 15:07:29.532887 4546 ctlog_controller.go:133] "Executing handle-fulcio-cert" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="89f69070-f409-46b9-bf23-37a25c24e987" 360: I1110 15:07:29.533179 4546 suite.go:118] "Normal FulcioCertDiscovered Fulcio certificate detected" logger="Event" 361: �[1mSTEP:�[0m Key Secret is created �[38;5;243m@ 11/10/25 15:07:29.536�[0m 362: I1110 15:07:29.547520 4546 controller.go:489] "Reconcile successful" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="89f69070-f409-46b9-bf23-37a25c24e987" 363: I1110 15:07:29.547559 4546 controller.go:460] "Reconciling" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="67470bd6-066f-42dc-a19f-b863083fb1ef" 364: I1110 15:07:29.547596 4546 ctlog_controller.go:127] "Executing move to pending phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="67470bd6-066f-42dc-a19f-b863083fb1ef" 365: I1110 15:07:29.547607 4546 ctlog_controller.go:127] "Executing resolve server TLS" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="67470bd6-066f-42dc-a19f-b863083fb1ef" 366: I1110 15:07:29.547620 4546 ctlog_controller.go:127] "Executing move to create phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="67470bd6-066f-42dc-a19f-b863083fb1ef" 367: I1110 15:07:29.547627 4546 ctlog_controller.go:127] "Executing handle-fulcio-cert" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="67470bd6-066f-42dc-a19f-b863083fb1ef" 368: I1110 15:07:29.547662 4546 ctlog_controller.go:127] "Executing handle-keys" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="67470bd6-066f-42dc-a19f-b863083fb1ef" 369: I1110 15:07:29.547673 4546 ctlog_controller.go:133] "Executing handle-keys" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="67470bd6-066f-42dc-a19f-b863083fb1ef" 370: I1110 15:07:29.547866 4546 reflector.go:358] "Starting reflector" logger="controller-runtime.cache" type="*v1.Secret" resyncPeriod="10h5m29.114473056s" reflector="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:290" 371: I1110 15:07:29.547878 4546 reflector.go:404] "Listing and watching" logger="controller-runtime.cache" type="*v1.Secret" reflector="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:290" 372: I1110 15:07:29.550647 4546 reflector.go:436] "Caches populated" logger="controller-runtime.cache" type="*v1.Secret" reflector="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:290" 373: E1110 15:07:29.656190 4546 base_action.go:92] "error during action execution" err="could not generate Secret: resourceVersion should not be set on objects to be created" logger="handle-keys" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="67470bd6-066f-42dc-a19f-b863083fb1ef" 374: E1110 15:07:29.656226 4546 controller.go:474] "Reconciler error" err="could not generate Secret: resourceVersion should not be set on objects to be created" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="67470bd6-066f-42dc-a19f-b863083fb1ef" 375: I1110 15:07:29.661827 4546 controller.go:460] "Reconciling" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="bde79bcb-821c-4604-8d31-7f1057db749d" ... 4688: I1110 15:07:33.106847 4546 ctlog_controller.go:127] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4be1580a-a51b-4bd7-a254-aafc7517d714" 4689: I1110 15:07:33.106854 4546 ctlog_controller.go:133] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4be1580a-a51b-4bd7-a254-aafc7517d714" 4690: I1110 15:07:33.106902 4546 server_config.go:125] "Server config secret is invalid, will recreate" logger="server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4be1580a-a51b-4bd7-a254-aafc7517d714" secret="ctlog-config-testxzh7h" reason="Trillian configuration mismatch" 4691: I1110 15:07:33.107009 4546 suite.go:118] "Warning CTLogConfigInvalid Config secret has invalid configuration, will recreate" logger="Event" 4692: I1110 15:07:33.108619 4546 suite.go:118] "Normal CTLogConfigCreated Config secret created successfully" logger="Event" 4693: I1110 15:07:33.114616 4546 server_config.go:248] "Remove invalid Secret with ctlog configuration" logger="server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4be1580a-a51b-4bd7-a254-aafc7517d714" Name="ctlog-config-testxzh7h" 4694: I1110 15:07:33.114634 4546 controller.go:489] "Reconcile successful" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4be1580a-a51b-4bd7-a254-aafc7517d714" 4695: I1110 15:07:33.114666 4546 controller.go:460] "Reconciling" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="fe7696a4-77f4-4895-8c12-10fee1361e48" 4696: I1110 15:07:33.114722 4546 ctlog_controller.go:127] "Executing move to pending phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="fe7696a4-77f4-4895-8c12-10fee1361e48" 4697: I1110 15:07:33.114733 4546 ctlog_controller.go:127] "Executing resolve server TLS" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="fe7696a4-77f4-4895-8c12-10fee1361e48" 4698: I1110 15:07:33.114746 4546 ctlog_controller.go:127] "Executing move to create phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="fe7696a4-77f4-4895-8c12-10fee1361e48" 4699: I1110 15:07:33.114753 4546 ctlog_controller.go:127] "Executing handle-fulcio-cert" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="fe7696a4-77f4-4895-8c12-10fee1361e48" 4700: I1110 15:07:33.114767 4546 suite.go:118] "Normal CTLogConfigCleanedUp Old config secret deleted successfully" logger="Event" 4701: I1110 15:07:33.114798 4546 ctlog_controller.go:127] "Executing handle-keys" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="fe7696a4-77f4-4895-8c12-10fee1361e48" 4702: I1110 15:07:33.114806 4546 ctlog_controller.go:127] "Executing resolve tree" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="fe7696a4-77f4-4895-8c12-10fee1361e48" 4703: make: *** [Makefile:129: test] Error 1 4704: I1110 15:07:33.114813 4546 ctlog_controller.go:127] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="fe7696a4-77f4-4895-8c12-10fee1361e48" ... 242894: I1110 15:09:29.686470 4546 server_config.go:248] "Remove invalid Secret with ctlog configuration" logger="server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="0b55e428-6810-4478-862e-7808abe129a5" Name="ctlog-config-testpp8vv" 242895: I1110 15:09:29.686490 4546 controller.go:489] "Reconcile successful" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="0b55e428-6810-4478-862e-7808abe129a5" 242896: I1110 15:09:29.686506 4546 controller.go:460] "Reconciling" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" 242897: I1110 15:09:29.686533 4546 ctlog_controller.go:127] "Executing move to pending phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" 242898: I1110 15:09:29.686535 4546 suite.go:118] "Normal CTLogConfigCleanedUp Old config secret deleted successfully" logger="Event" 242899: I1110 15:09:29.686541 4546 ctlog_controller.go:127] "Executing resolve server TLS" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" 242900: I1110 15:09:29.686549 4546 ctlog_controller.go:127] "Executing move to create phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" 242901: I1110 15:09:29.686553 4546 ctlog_controller.go:127] "Executing handle-fulcio-cert" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" 242902: I1110 15:09:29.686572 4546 ctlog_controller.go:127] "Executing handle-keys" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" 242903: I1110 15:09:29.686578 4546 ctlog_controller.go:127] "Executing resolve tree" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" 242904: I1110 15:09:29.686583 4546 ctlog_controller.go:127] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" 242905: I1110 15:09:29.686588 4546 ctlog_controller.go:133] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" 242906: I1110 15:09:29.686643 4546 server_config.go:125] "Server config secret is invalid, will recreate" logger="server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="6f244554-a3c2-498a-b1b8-87f4434eee82" secret="ctlog-config-testnqvh8" reason="Trillian configuration mismatch" 242907: I1110 15:09:29.686741 4546 suite.go:118] "Warning CTLogConfigInvalid Config secret has invalid configuration, will recreate" logger="Event" 242908: I1110 15:09:29.688137 4546 suite.go:118] "Normal CTLogConfigCreated Config secret created successfully" logger="Event" 242909: �[38;5;9m[FAILED]�[0m in [It] - /home/runner/work/secure-sign-operator/secure-sign-operator/internal/controller/ctlog/ctlog_controller_test.go:158 �[38;5;243m@ 11/10/25 15:09:29.69�[0m 242910: �[1mSTEP:�[0m removing the custom resource for the Kind CTlog �[38;5;243m@ 11/10/25 15:09:29.69�[0m ... 242913: I1110 15:09:29.693407 4546 controller.go:460] "Reconciling" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" 242914: I1110 15:09:29.693426 4546 ctlog_controller.go:127] "Executing move to pending phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" 242915: I1110 15:09:29.693432 4546 ctlog_controller.go:127] "Executing resolve server TLS" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" 242916: I1110 15:09:29.693437 4546 ctlog_controller.go:127] "Executing move to create phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" 242917: I1110 15:09:29.693459 4546 suite.go:118] "Normal CTLogConfigCleanedUp Old config secret deleted successfully" logger="Event" 242918: I1110 15:09:29.693463 4546 ctlog_controller.go:127] "Executing handle-fulcio-cert" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" 242919: I1110 15:09:29.693490 4546 ctlog_controller.go:127] "Executing handle-keys" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" 242920: I1110 15:09:29.693503 4546 ctlog_controller.go:127] "Executing resolve tree" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" 242921: I1110 15:09:29.693511 4546 ctlog_controller.go:127] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" 242922: I1110 15:09:29.693517 4546 ctlog_controller.go:133] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" 242923: I1110 15:09:29.693575 4546 server_config.go:125] "Server config secret is invalid, will recreate" logger="server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="default/test" namespace="default" name="test" reconcileID="4f566d43-74e5-4e2c-ac21-fa7a27ab104d" secret="ctlog-config-test4wqvl" reason="Trillian configuration mismatch" 242924: I1110 15:09:29.693687 4546 suite.go:118] "Warning CTLogConfigInvalid Config secret has invalid configuration, will recreate" logger="Event" 242925: �[1mSTEP:�[0m Deleting the Namespace to perform the tests �[38;5;243m@ 11/10/25 15:09:29.694�[0m 242926: I1110 15:09:29.695491 4546 suite.go:118] "Normal CTLogConfigCreated Config secret created successfully" logger="Event" 242927: �[38;5;243m<< Timeline�[0m 242928: �[38;5;9m[FAILED] Timed out after 120.001s. 242929: Expected success, but got an error: 242930: <*errors.StatusError | 0xc000fc5720>: 242931: deployments.apps "ctlog" not found ... 242935: ListMeta: { 242936: SelfLink: "", 242937: ResourceVersion: "", 242938: Continue: "", 242939: RemainingItemCount: nil, 242940: }, 242941: Status: "Failure", 242942: Message: "deployments.apps \"ctlog\" not found", 242943: Reason: "NotFound", 242944: Details: {Name: "ctlog", Group: "apps", Kind: "deployments", UID: "", Causes: nil, RetryAfterSeconds: 0}, 242945: Code: 404, 242946: }, 242947: }�[0m 242948: �[38;5;9mIn �[1m[It]�[0m�[38;5;9m at: �[1m/home/runner/work/secure-sign-operator/secure-sign-operator/internal/controller/ctlog/ctlog_controller_test.go:158�[0m �[38;5;243m@ 11/10/25 15:09:29.69�[0m 242949: �[38;5;243m------------------------------�[0m 242950: �[38;5;9m• [FAILED] [120.020 seconds]�[0m 242951: �[0mCTlog update test �[38;5;243mCTlog update test �[38;5;9m�[1m[It] should successfully reconcile a custom resource for CTlog�[0m ... 510989: I1110 15:11:29.851598 4546 suite.go:118] "Normal CTLogConfigCreated Config secret created successfully" logger="Event" 510990: I1110 15:11:29.856707 4546 server_config.go:248] "Remove invalid Secret with ctlog configuration" logger="server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="5c9c94c3-a3da-4fc2-a0b6-2430967ab1fd" Name="ctlog-config-test4z77q" 510991: I1110 15:11:29.856722 4546 controller.go:489] "Reconcile successful" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="5c9c94c3-a3da-4fc2-a0b6-2430967ab1fd" 510992: I1110 15:11:29.856743 4546 controller.go:460] "Reconciling" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 510993: I1110 15:11:29.856770 4546 ctlog_controller.go:127] "Executing move to pending phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 510994: I1110 15:11:29.856778 4546 ctlog_controller.go:127] "Executing resolve server TLS" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 510995: I1110 15:11:29.856787 4546 ctlog_controller.go:127] "Executing move to create phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 510996: I1110 15:11:29.856795 4546 ctlog_controller.go:127] "Executing handle-fulcio-cert" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 510997: I1110 15:11:29.856820 4546 ctlog_controller.go:127] "Executing handle-keys" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 510998: I1110 15:11:29.856819 4546 suite.go:118] "Normal CTLogConfigCleanedUp Old config secret deleted successfully" logger="Event" 510999: I1110 15:11:29.856828 4546 ctlog_controller.go:127] "Executing resolve tree" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 511000: I1110 15:11:29.856836 4546 ctlog_controller.go:127] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 511001: I1110 15:11:29.856845 4546 ctlog_controller.go:133] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 511002: I1110 15:11:29.856910 4546 server_config.go:125] "Server config secret is invalid, will recreate" logger="server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" secret="ctlog-config-testq75v9" reason="Trillian configuration mismatch" 511003: I1110 15:11:29.857005 4546 suite.go:118] "Warning CTLogConfigInvalid Config secret has invalid configuration, will recreate" logger="Event" 511004: �[38;5;9m[FAILED]�[0m in [It] - /home/runner/work/secure-sign-operator/secure-sign-operator/internal/controller/ctlog/ctlog_hot_update_test.go:135 �[38;5;243m@ 11/10/25 15:11:29.858�[0m 511005: �[1mSTEP:�[0m removing the custom resource for the Kind CTlog �[38;5;243m@ 11/10/25 15:11:29.858�[0m ... 511008: I1110 15:11:29.864466 4546 server_config.go:248] "Remove invalid Secret with ctlog configuration" logger="server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" Name="ctlog-config-testq75v9" 511009: I1110 15:11:29.864485 4546 controller.go:489] "Reconcile successful" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="f2b817f6-958b-4c9b-9a75-5e493d0c490b" 511010: I1110 15:11:29.864511 4546 controller.go:460] "Reconciling" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" 511011: I1110 15:11:29.864542 4546 ctlog_controller.go:127] "Executing move to pending phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" 511012: I1110 15:11:29.864552 4546 ctlog_controller.go:127] "Executing resolve server TLS" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" 511013: I1110 15:11:29.864562 4546 ctlog_controller.go:127] "Executing move to create phase" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" 511014: I1110 15:11:29.864569 4546 ctlog_controller.go:127] "Executing handle-fulcio-cert" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" 511015: I1110 15:11:29.864598 4546 ctlog_controller.go:127] "Executing handle-keys" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" 511016: I1110 15:11:29.864606 4546 ctlog_controller.go:127] "Executing resolve tree" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" 511017: I1110 15:11:29.864615 4546 ctlog_controller.go:127] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" 511018: I1110 15:11:29.864625 4546 ctlog_controller.go:133] "Executing server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" 511019: I1110 15:11:29.864637 4546 suite.go:118] "Normal CTLogConfigCleanedUp Old config secret deleted successfully" logger="Event" 511020: I1110 15:11:29.864696 4546 server_config.go:125] "Server config secret is invalid, will recreate" logger="server config" controller="ctlog" controllerGroup="rhtas.redhat.com" controllerKind="CTlog" CTlog="update/test" namespace="update" name="test" reconcileID="a7106efe-3adb-4ce0-acbd-12fd938cfd03" secret="ctlog-config-testw5j5w" reason="Trillian configuration mismatch" 511021: I1110 15:11:29.864836 4546 suite.go:118] "Warning CTLogConfigInvalid Config secret has invalid configuration, will recreate" logger="Event" 511022: �[38;5;243m<< Timeline�[0m 511023: �[38;5;9m[FAILED] Timed out after 120.000s. 511024: Expected success, but got an error: 511025: <*errors.StatusError | 0xc0005a0be0>: 511026: deployments.apps "ctlog" not found ... 511030: ListMeta: { 511031: SelfLink: "", 511032: ResourceVersion: "", 511033: Continue: "", 511034: RemainingItemCount: nil, 511035: }, 511036: Status: "Failure", 511037: Message: "deployments.apps \"ctlog\" not found", 511038: Reason: "NotFound", 511039: Details: {Name: "ctlog", Group: "apps", Kind: "deployments", UID: "", Causes: nil, RetryAfterSeconds: 0}, 511040: Code: 404, 511041: }, 511042: }�[0m 511043: �[38;5;9mIn �[1m[It]�[0m�[38;5;9m at: �[1m/home/runner/work/secure-sign-operator/secure-sign-operator/internal/controller/ctlog/ctlog_hot_update_test.go:135�[0m �[38;5;243m@ 11/10/25 15:11:29.858�[0m 511044: �[38;5;243m------------------------------�[0m 511045: �[38;5;9m�[1mSummarizing 2 Failures:�[0m 511046: �[38;5;9m[FAIL]�[0m �[0mCTlog controller �[38;5;243mCTlog controller test �[38;5;9m�[1m[It] should successfully reconcile a custom resource for CTlog�[0m 511047: �[38;5;243m/home/runner/work/secure-sign-operator/secure-sign-operator/internal/controller/ctlog/ctlog_controller_test.go:158�[0m 511048: �[38;5;9m[FAIL]�[0m �[0mCTlog update test �[38;5;243mCTlog update test �[38;5;9m�[1m[It] should successfully reconcile a custom resource for CTlog�[0m 511049: �[38;5;243m/home/runner/work/secure-sign-operator/secure-sign-operator/internal/controller/ctlog/ctlog_hot_update_test.go:135�[0m 511050: �[38;5;9m�[1mRan 2 of 2 Specs in 249.267 seconds�[0m 511051: �[38;5;9m�[1mFAIL!�[0m -- �[38;5;10m�[1m0 Passed�[0m | �[38;5;9m�[1m2 Failed�[0m | �[38;5;11m�[1m0 Pending�[0m | �[38;5;14m�[1m0 Skipped�[0m 511052: --- FAIL: TestController (249.27s) 511053: FAIL 511054: coverage: 82.2% of statements 511055: FAIL github.com/securesign/operator/internal/controller/ctlog 249.294s 511056: --- FAIL: TestServerConfig_CanHandle (0.09s) 511057: --- FAIL: TestServerConfig_CanHandle/ConditionTrue:_spec.serverConfigRef_is_nil_and_status.serverConfigRef_is_not_nil (0.01s) 511058: server_config_test.go:152: CanHandle() = true, want false 511059: --- FAIL: TestServerConfig_CanHandle/ConditionTrue:_observedGeneration_==_generation (0.01s) 511060: server_config_test.go:152: CanHandle() = true, want false 511061: --- FAIL: TestServerConfig_Handle (0.02s) 511062: server_config_test.go:189: 511063: Expected 511064: <*v1alpha1.LocalObjectReference | 0x0>: nil 511065: not to be nil 511066: --- FAIL: TestServerConfig_Handle/use_spec.serverConfigRef (0.02s) 511067: server_config_test.go:479: CanHandle() = &{{false 0s} error accessing custom server config secret: secrets "config" not found}, want &{{false 0s} <nil>} 511068: testing.go:1679: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test 511069: FAIL ... 511091: ok github.com/securesign/operator/internal/controller/trillian/actions/db 0.324s coverage: 22.4% of statements 511092: ok github.com/securesign/operator/internal/controller/trillian/actions/logserver 0.147s coverage: 10.8% of statements 511093: ok github.com/securesign/operator/internal/controller/trillian/actions/logsigner 0.160s coverage: 10.8% of statements 511094: github.com/securesign/operator/internal/controller/trillian/utils coverage: 0.0% of statements 511095: ok github.com/securesign/operator/internal/controller/tsa 17.988s coverage: 86.7% of statements 511096: ok github.com/securesign/operator/internal/controller/tsa/actions 0.559s coverage: 41.9% of statements 511097: github.com/securesign/operator/internal/controller/tsa/utils coverage: 0.0% of statements 511098: ok github.com/securesign/operator/internal/controller/tuf 13.695s coverage: 87.9% of statements 511099: ok github.com/securesign/operator/internal/controller/tuf/actions 0.176s coverage: 15.5% of statements 511100: ? github.com/securesign/operator/internal/controller/tuf/constants [no test files] 511101: github.com/securesign/operator/internal/controller/tuf/utils coverage: 0.0% of statements 511102: github.com/securesign/operator/internal/images coverage: 0.0% of statements 511103: github.com/securesign/operator/internal/labels coverage: 0.0% of statements 511104: github.com/securesign/operator/internal/testing/action coverage: 0.0% of statements 511105: github.com/securesign/operator/internal/testing/common/tsa coverage: 0.0% of statements 511106: github.com/securesign/operator/internal/testing/errors coverage: 0.0% of statements 511107: github.com/securesign/operator/internal/testing/http coverage: 0.0% of statements 511108: github.com/securesign/operator/internal/testing/kubernetes coverage: 0.0% of statements 511109: github.com/securesign/operator/internal/utils coverage: 0.0% of statements 511110: ok github.com/securesign/operator/internal/utils/kubernetes 0.432s coverage: 28.3% of statements 511111: ok github.com/securesign/operator/internal/utils/kubernetes/ensure 0.177s coverage: 28.8% of statements 511112: ok github.com/securesign/operator/internal/utils/kubernetes/ensure/deployment 0.145s coverage: 66.7% of statements 511113: github.com/securesign/operator/internal/utils/kubernetes/job coverage: 0.0% of statements 511114: ok github.com/securesign/operator/internal/utils/tls 0.138s coverage: 75.0% of statements 511115: github.com/securesign/operator/internal/utils/tls/ensure coverage: 0.0% of statements 511116: FAIL 511117: ##[error]Process completed with exit code 2. 511118: Post job cleanup.