Version 0.28.0 always logs probe failures (as 'ERROR' level failures) with no way to turn it off

[siebenmann]

We have some Blackbox probes that are expected to always fail (they exist to make sure that certain TCP ports are not responding). As of Blackbox 0.28.0, it appears impossible to silence the logs for these probes, because 'failed' probes are always logged at level=ERROR and you cannot set the new --log.prober to anything above error (or to 'none' or the equivalent). We don't want our logs to be full of constant failure notices from blackbox, especially not logged at level ERROR right beside serious issues with Blackbox in general.

[electron0zero]

we updated the log levels and changed the logging behaviour in v0.28.0 to be more accurate, please see release notes: https://github.com/prometheus/blackbox_exporter/releases/tag/v0.28.0

A failure should be logged at error level, and if your probe is always expected to fail then you should adjust the probe config to expect the failure so probe will pass when server returns the expected error.

see https://github.com/prometheus/blackbox_exporter/blob/master/example.yml and https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md for more details.

[SuperQ]

@electron0zero, I think we have a missing feature here. TCP probes don't have a configuration for expected errors.

[electron0zero]

yup, @siebenmann can you share more about the probe and the config you have?

[siebenmann]

The probe module is simply:

  tcp_connect_fail:
    prober: tcp
    tcp:
      preferred_ip_protocol: "ip4"

It is used to check that port 22, 80, and 443 on some devices are not connecting at all, because we have had problems where these particular devices will activate these ports on their own under various circumstances. We probe them as normal and write alerts that use probe_success{probe="tcp_connect_fail"} == 1 as the alert rule.

It would be nice if there was a way to specify a TCP probe that was expected to fail to connect, but when we set these up many years ago I assumed that Blackbox was deliberately minimal here since you could do this with Prometheus alert rules.

[siebenmann]

It's also my personal view as a system administrator that system logs at level 'error' are for actual, significant errors that should attract attention from people. In some environments, probes failing every now and then is an extremely routine event when, for example, an ICMP ping packet is dropped; this is why Prometheus alert rules have 'for:' expressions and so on. Making logging every probe failure mandatory results in verbose logs full of things that aren't true 'get the system administrator to look and fix the problem' errors; on a modest configuration without probes that are expected to fail I have 137 error logs from failed probes in the past 8 hours or so (partly because ipv6.google.com seems to drop a lot of ICMP6 pings). Probe failures are not system log level 'errors' as ordinarily understood in my view.

[SuperQ]

Yes, it was intended that we only log probe failures as an opt-in.

As for TCP probes that are intended to fail, the module configuration needs an option to have a list of possible connect statuses. Similar to how the http probe has a list of passing HTTP status codes. Rather than an additional label on probe_success, we want something like:

modules:
  tcp_refused:
    prober: tcp
    tcp:
      connect:
        - refused
        - unreachable

Figuring out how to parse the Go Dialer return errors is going to be fun, but should be possible.

[roidelapluie]

I do believe as well that those should not be logged as errors as they are not errors of the blackbox exporter. In the same way that prometheus does not log scrape failures are errors.

[electron0zero]

@siebenmann can you share the log lines there you are seeing at error level but expect them to be at Info or debug level?

I am making some changes in #1517 to address #1509. if it makes sense to revert the log level to Info, I can include that change in my PR.

[siebenmann]

Typical logged as errors routine probe failures include:

level=ERROR source=dns.go:273 msg="Error while sending a DNS query" module=dns_cslab_soa target=149.112.121.10:53 err="read udp4 128.100.3.51:41544->149.112.121.10:53: i/o timeout"
level=ERROR source=handler.go:130 msg="Probe failed" module=dns_cslab_soa target=149.112.121.10:53 duration_seconds=30.000167009
level=ERROR source=http.go:673 msg="Failed to read HTTP response body" module=http_2xx target=https://facebook.com/ err="stream error: stream ID 1; INTERNAL_ERROR; received from peer"
level=ERROR source=handler.go:130 msg="Probe failed" module=http_2xx target=https://facebook.com/ duration_seconds=0.292511101
level=ERROR source=handler.go:130 msg="Probe failed" module=icmp target=1.1.1.1 duration_seconds=9.500590546
level=ERROR source=handler.go:130 msg="Probe failed" module=icmp6 target=2602:fde5:2a::13 duration_seconds=9.502109727