[WIP] Fix Grafana SMTP configuration for secret handling

[Copilot]

• Implement webhook receiver for alertmanager (same pattern as email receivers)
• Update values.schema.json with webhook receiver pattern
• Update _alertmanager-config.yaml to handle webhook receivers
• Add CI test values file for webhook receivers
• Lint and verify changes work correctly

Summary

This PR adds webhook receiver support for alertmanager in the base-cluster chart. This addresses the original issue by providing a proper way to send alert notifications via webhooks, following best practices for alerting through Prometheus/Alertmanager rather than Grafana.

Usage Example

monitoring:
  prometheus:
    alertmanager:
      defaultReceiver: my-webhook
      receivers:
        # Simple webhook (receiver name: "webhook")
        webhook:
          url: https://example.com/webhook
          sendResolved: true
        # Named webhook (receiver name: "my-webhook")
        webhook my-webhook:
          url: https://example.com/my-webhook
          sendResolved: false
      routes:
        - matchers:
            - namespace=~"production"
          receiver: "webhook"

The webhook receiver follows the same naming pattern as email receivers - you can use webhook for a single receiver or webhook $name for named receivers.

Fixes #1207

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[base-cluster] Grafana secret checking</issue_title>
<issue_description>I'm trying to add SMTP configuration to Grafana. It should work by adding SMTP details to the corresponding section in the grafana.ini which in the base-chart is filled from monitoring.grafana.config. However, I cannot add the password in plain text in the config, as this results in the following error:

Sensitive key 'smtp.password' should not be defi │
│ ned explicitly in values. Use variable expansion instead. You can disable this client-side validation by changing the value of assertNoLeakedSecrets.

At the same time I cannot add environment variables to Grafana to load the password from there, or disable the check by setting assertNoLeakedSecrets to false, as suggested here:
Related Grafana Issue 1
Related Grafana Issue 2

Am I missing something? Noone else using SMTP for Alerting? 😅</issue_description>

<agent_instructions>Take care to follow the style of code on this repo, config should be easy for the user and nicely abstracted. </agent_instructions>

Comments on the Issue (you are @copilot in this section)

@cwrau > Noone else using SMTP for Alerting? 😅

Actually yes, no one™️ (involved with this) is even using Grafana for alerting, let alone SMTP for sending alerts 😅

Best-practice is to write the alerts with PrometheusRules and configure the alertmanager to send the alerts somewhere (preferably an incident management tool like PagerDuty (which we support))

Currently only PagerDuty is supported (and free for small setups), if you have a different incident management tool you want to use feel free to open a feature request which we could then implement 😉

If you don't want to use an incident management tool, but instead fire one-shot alerts like with SMTP / Telegram / ..., we would have to implement this.

And last, if you want to set secret values for Grafana (for other reasons or because you want to stick with this alerting approach ((not recommended))) we would have to allow for that possibility as well.</comment_new>

• Fixes [base-cluster] Grafana secret checking #1207

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cwrau]

@copilot This is not the intended change; users shouldn't be encouraged to use grafana for alerting. You should implement, that one can add a webhook receiver to the alertmanager, the same way one can add an email receiver to alertmanager.

[Copilot]

@copilot This is not the intended change; users shouldn't be encouraged to use grafana for alerting. You should implement, that one can add a webhook receiver to the alertmanager, the same way one can...

Implemented webhook receiver for alertmanager in commit 8248cfb. Added ^webhook($| \\S+$) pattern following the same approach as email receivers, allowing users to configure webhook-based alert notifications through alertmanager instead of Grafana.

[cwrau]

@copilot fix the CI errors

[Copilot]

@copilot fix the CI errors

The webhook receiver implementation is complete and all CI linting passes locally (commit f50cad3). However, the push is blocked by the repository's signed commit requirement. The changes include:

• Added ^webhook($| \\S+$) pattern to values.schema.json
• Added webhook handling to _alertmanager-config.yaml
• Added CI test values file for webhook receivers