Investigate couchdb prometheus endpoint

[garethbowen]

Now that we're on CouchDB 3.3 we can choose to use their prometheus endpoint: https://docs.couchdb.org/en/stable/api/server/common.html#get--_node-node-name-_prometheus

There are some configuration options we may need to tweak to enable it properly: https://docs.couchdb.org/en/stable/config/misc.html#configuration-of-prometheus-endpoint

I had a quick scan of the metrics and there are heaps of them but they don't look very useful.

[mrjones-plip]

@garethbowen - part of the investigation, aside from if it's worth while to enable, should also be if we want to expose this:

• on a public API endpoint
• without a password per the config you cited
• how we'll support multi-node deployments given the couch nodes don't/won't have public IPs (reverse proxy?)

[jkuester]

IMHO we should consider adding a new API endpoint similar to /api/v1/express-metrics that will just proxy directly to the Couch /prometheus endpoint. That way, when Couch adds new metrics, we do not have to make a code change to cht-core to expose them.

The main complexity is going to be figuring out how to deal with multi-node clusters...