Prometheus metric handler is exposing 500 server error

[geet]

I have instrumented custom prometheus metrics from my application. I have configured prometheus to scrape the target. I have started seeing 500 error in the prometheus and due to that the scrape failed. I tried to manually hit the /metrics endpoint in my application and got the error saying

<metric> [label_values] was collected before with the same name and label values

It works initially but after a few minutes I start seeing the error.

I don't think it should be considered a duplicate metric but if the metric with same label and values are added,then it should increase the count.

go client version: github.com/prometheus/client_golang v1.12.2

[geet]

Here is the code snippet I am using to provide prometheus handler with gofiber.

import (
“github.com/gofiber/adaptor/v2](http://github.com/gofiber/adaptor/v2”
“github.com/prometheus/client_golang/prometheus/promhttp"

)

func MetricsRoutes(api fiber.Router) {
api.Get(“/metrics”, adaptor.HTTPHandler(promhttp.Handler()))
}

[geet]

The problematic metric is a custom GaugeVec registerd by the following code.

func RegisterMetric(metricNames []string) {
	for i, metricName := range metricNames {
		metricGauge := prometheus.NewGaugeVec(prometheus.GaugeOpts{
			Name: metricName,
		}, getLabels(metricName))
		metricsList[i] = metricName
		metrics[i] = metricGauge
		prometheus.MustRegister(metricGauge)
	}
}

getLabels returns the labels supported.

I see the metric being collected fine for the first few scrapes and suddenly I start seeing 500 server error on the scrapes.

The error thrown is like this.

* collected metric "en_devices" { label:<name:"_storage_" value:"3" > label:<name:"ctype" value:"public" > label:<name:"en_destination" value:"iOS " > label:<name:"en_device_type" value:"A" > label:<name:"location" value:"us" > label:<name:"scope" value:"<some_string>" > label:<name:"service_instance" value:"<some_guid>" > label:<name:"service_name" value:"<some_string>" > gauge:<value:1 > } was collected before with the same name and label values