When adding multiple Alertmanager configs in the plugin, messages are posted to a random channel

[aboukhal]

We're using Mattermost 9.4.1 and plugin 0.4.0

Setup:
We have configured four different Alertmanagers in the cluster. They all use the same team, but different channels and they each have a unique token.

Incorrect behavior:
When alerts are sent by the cluster (or when sending test messages) to the four URLs, the messages are being posted into one of the four configured channels randomly. When sending the same message over and over, the channel seems to be consistent, even if it is wrong. If the message is changed ever so slightly, it lands in a different, random channel, even though the token was not touched.

Expected behavior:
The message should always be posted to the channel associated with the token.

What else have I tied:
I looked at the code to determine if there were debug logs, that could be enabled to figure out what goes wrong. This is not the case. I also reviewed the relevant code sections but nothing jumped out immediately. It looks plausible.

What did I not try:
I am unable to add additional debug logs or play with the code, as I'm not in an environment where I can easily build Go projects.

Do you have any idea what's going wrong?

[Pengwin61]

Good afternoon. I also encountered similar behavior of the plugin. I don't have an alertmanager cluster, only 1 copy. I created 3 channels in one team. And I get something completely different

route:
  receiver: 'telegram_bot'
  group_by: ['alertname']
  group_wait: 5s
  group_interval: 1m
  repeat_interval: 12h
  routes:
  - match:
      alertgroup: 'SD'
    receiver: 'mm-home-sd'
  - match:
      alertgroup: 'InfraRules'
    receiver: 'mm-home-infra'
  - match:
      alertgroup: 'Hardware'
    receiver: 'mm-home-hardware'

[cpanato]

thanks for the report, are you willing to open a PR to fix this issue?

[aboukhal]

We don't have the environment to build Go applications without a lot of effort. So dry-analyzing the source code was as much as I could do at the moment

[aboukhal]

After figuring out that Mattermost supports most of the schema from Slack, we have now switched over to using the Slack configuration from Alertmanager. We've lost time calculations (using a timezone other than UTC or calculating durations, I've opened prometheus/alertmanager#3762 for that) but gained extensive customizability.
There are a lot of Slack template examples on the net to have a solid starting point for this.
Thanks for this Plugin, we would have loved to stick with it but in the current state we needed a solution that we can easily develop ourselves.

[baznikin]

Confirm, same problem.

@aboukhal are you using Slack's configuation for Alertmanager to send notification to MatterMost, am I right?

[aboukhal]

@aboukhal are you using Slack's configuation for Alertmanager to send notification to MatterMost, am I right?

Correct. The tricky part is to create the templates so that messages look good in Mattermost with the appropriate formating, emojis, etc. But once that's done, it just works out of the box.

[friendlymatthew]

After figuring out that Mattermost supports most of the schema from Slack, we have now switched over to using the Slack configuration from Alertmanager. We've lost time calculations (using a timezone other than UTC or calculating durations, I've opened prometheus/alertmanager#3762 for that) but gained extensive customizability. There are a lot of Slack template examples on the net to have a solid starting point for this. Thanks for this Plugin, we would have loved to stick with it but in the current state we needed a solution that we can easily develop ourselves.

Thank you for investigating @aboukhal! Any chance you could share the slack configuration scheme?

I'm pretty new to this and this is what I have so far. Does this look sane to you?

  receivers:
        - name: mattermost-dev
          slack_configs:
          - send_resolved: true
            api_url: https://mattermost.shuttle.video/hooks/eo7mptcdcign8fgkxuagrtrtce
            channel: "#alerts-dev"
            text: |
              {{ range .Alerts }}
              *Alert:* {{ .Annotations.summary }}
              *Description:* {{ .Annotations.description }}
              *Severity:* {{ .Labels.severity }}
              *Starts At:* {{ .StartsAt }}
              *Ends At:* {{ .EndsAt }}
              {{ end }}
          - name: 'null'

        # - name: mattermost-dev
        #   webhook_configs:
        #   - send_resolved: true
        #     url: "https://mattermost.shuttle.video/plugins/alertmanager/api/webhook?token='6quZ2xBexp4Ws5NhYrX5D_lwpL-9w6K6'"
        # - name: 'null'

[aboukhal]

I can't share our code but we have used this as a starting point:
https://hodovi.cc/blog/creating-awesome-alertmanager-templates-for-slack/

It's not perfect out of the box but it demonstrates all features needed to create beautiful Mattermost messages through templates. Just keep in mind that actions do not work in Mattermost - instead you need to use hyperlinks in the text itself.

Looking at your snippet, the - name: 'null' line seems incorrect and needs to be removed.

[st19n]

I ran into the same issue, and have been doing some debugging.

I believe this is related to, and fixed in #118.
On the current latest release (0.4.0) the logic comparing the stored token and the supplied one is reversed.

With 8962126 there is the suggestion that tokens might need single quotes. Which I believe to be incorrect.
They should already be generated as URL safe values by the plugin, and if not should be URL encoded. Not quoted.

Go's URL.Query().Get("token") does handle URL encoded strings, but does not automatically handle quotes, which will remain as part of the token.

So basically: With the current latest release (0.4.0) and latest documentation on main - hinting that the token might need to be quoted - the check always succeeds because the quoted token does not match and the logic is reversed.

The randomness in which channel receives the alert is because ranging over the map of configs is not deterministic, and the first iteration will always return successfully. This is not an issue, but explains the randomness here.

I would suggest reverting 8962126, and with a new release (or using the current main branch) this should be resolved.

[maxout123]

I ran into the same issue, and have been doing some debugging.

god bless you, that realy fixes the issue

[exquisitus]

I believe this is related to, and fixed in #118. On the current latest release (0.4.0) the logic comparing the stored token and the supplied one is reversed.

If the issue has already been fixed in the main branch, would @cpanato consider releasing a bugfix version 0.4.1?
Edit: I confirm the issue has been fixed in main, but since there is no release, we have to build from source.

[Kuzyashin]

Still facing this issue

[Kuzyashin]

Fixed in the fork https://github.com/Kuzyashin/mattermost-plugin-alertmanager

And some improvements also