Unable to setup monitoring for patroni (etcd3)

[MichalBartos]

Hello Everyone,

Could I please ask for some advice? I'm having trouble configuring monitoring for Patroni with etcd3 and can't figure out what I'm doing wrong. Any guidance would be greatly appreciated.

I'm getting following error:
[ERROR] [error:unknown DCS] could not refresh active sources, using last valid cache

pgwatch config:

Type: patroni-continuous-discovery
Connection string: etcd://<server_1>:2379,<server_2>:2379/service/postgresql-cluster?ca_file=<path_to_ca>&cert_file=<path_to_cert>&key_file=<path_to_key>

patroni yaml:

scope: postgresql-cluster
namespace: /service/
name: postgresql-01
...

scope: postgresql-cluster
namespace: /service/
name: postgresql-02
...

etcd config:

ETCD_NAME="postgresql-01"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_INITIAL_CLUSTER="postgresql-01=https://<server_1>:2380,postgresql-02=https://<server_2>:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://<server_1>:2380"
ETCD_LISTEN_PEER_URLS="https://<ip_server_1>:2380"
ETCD_LISTEN_CLIENT_URLS="https://<ip_server_1>:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://<server_1>:2379"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="<path_to_ca>"
ETCD_CERT_FILE="<path_to_cert>"
ETCD_KEY_FILE="<path_to_key>"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="<path_to_ca>"
ETCD_PEER_CERT_FILE="<path_to_cert>"
ETCD_PEER_KEY_FILE="<path_to_key>"

ETCD_NAME="postgresql-02"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_INITIAL_CLUSTER="postgresql-01=https://<server_1>:2380,postgresql-02=https://<server_2>:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://<server_2>:2380"
ETCD_LISTEN_PEER_URLS="https://<ip_server_2>:2380"
ETCD_LISTEN_CLIENT_URLS="https://<ip_server_2>:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://<server_2>:2379"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="<path_to_ca>"
ETCD_CERT_FILE="<path_to_cert>"
ETCD_KEY_FILE="<path_to_key>"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="<path_to_ca>"
ETCD_PEER_CERT_FILE="<path_to_cert>"
ETCD_PEER_KEY_FILE="<path_to_key>"

Note: <...> are placeholders for actuals server names, IPs and paths.

[pashagolub]

what version of pgwatch is involved?

[MichalBartos]

4.0.0 - but it was same for older versions as well.

When I tested connection in WebUI I got:

From some reason it i not parsing data from etcd

2025-10-10 10:28:55.445 [ERROR] [database:] [err:failed to connect to `user=postgres database=`: /var/run/postgresql/.s.PGSQL.5432 (/var/run/postgresql): dial error: dial unix /var/run/postgresql/.s.PGSQL.5432: connect: no such file or directory] [host:/var/run/postgresql] [port:5432] [time:69.6µs] Connect
2025-10-10 10:28:55.445 [ERROR] [err:failed to connect to `user=postgres database=`: /var/run/postgresql/.s.PGSQL.5432 (/var/run/postgresql): dial error: dial unix /var/run/postgresql/.s.PGSQL.5432: connect: no such file or directory] [time:123.57µs] Acquire
2025-10-10 10:28:55.445 [WARNING] [source:pat_test] could not init connection, retrying on next iteration

Port and server is configured in patroni.yml

postgresql:
  listen: 0.0.0.0:9400
  connect_address: <server_1>:9400
  data_dir: /<path>/patroni17/data

Testing connection with etcdctl works fine.

I know that I'm doing something wrong, but I can't figure out what :D.

[pashagolub]

It seems I fixed similar error in #991. Do you have an opportunity to build a patched binary to check?

[MichalBartos]

Hi,
sorry for waiting. I'll build it and let you know.

[MichalBartos]

Hi,

I build it from git but the error remains. Here is full debug log:

2025-10-22 07:47:24.778 [DEBUG] wake up after 120 seconds (reaper/reaper.go:201 reaper.(*Reaper).Reap)
2025-10-22 07:47:24.779 [DEBUG] Alloc: 1921 Kb, TotalAlloc: 7330 Kb, Sys: 14934 Kb, NumGC: 8, HeapAlloc: 1921 Kb, HeapSys: 7360 Kb (reaper/reaper.go:70 reaper.(*Reaper).PrintMemStats)
2025-10-22 07:47:24.790 [DEBUG] [database:pgwatch] [host:<pgwatch_db_host>] [pid:3704760] [port:<pgwatch_db_port>] [time:11.237485ms] Connect
2025-10-22 07:47:24.790 [DEBUG] [pid:3704760] [time:11.375212ms] Acquire
2025-10-22 07:47:24.791 [DEBUG] [sql:select /* pgwatch_generated */
        name,
        "group",
        dbtype,
        connstr,
        coalesce(config, '{}'::jsonb) as config,
        coalesce(config_standby, '{}'::jsonb) as config_standby,
        coalesce(preset_config, '') as preset_config,
        coalesce(preset_config_standby, '') as preset_config_standby,
        coalesce(include_pattern, '') as include_pattern,
        coalesce(exclude_pattern, '') as exclude_pattern,
        coalesce(custom_tags, '{}'::jsonb) as custom_tags,
        only_if_master,
        is_enabled
from
        pgwatch.source] [alreadyPrepared:false] [name:stmtcache_37fa59bb0cd8aa1f372ee1720a0dac7caff08787cceb9e71] [pid:3704760] [time:804.11µs] Prepare
2025-10-22 07:47:24.792 [DEBUG] [sql:select /* pgwatch_generated */
        name,
        "group",
        dbtype,
        connstr,
        coalesce(config, '{}'::jsonb) as config,
        coalesce(config_standby, '{}'::jsonb) as config_standby,
        coalesce(preset_config, '') as preset_config,
        coalesce(preset_config_standby, '') as preset_config_standby,
        coalesce(include_pattern, '') as include_pattern,
        coalesce(exclude_pattern, '') as exclude_pattern,
        coalesce(custom_tags, '{}'::jsonb) as custom_tags,
        only_if_master,
        is_enabled
from
        pgwatch.source] [args:[]] [commandTag:SELECT 14] [pid:3704760] [time:1.626324ms] Query
2025-10-22 07:47:24.792 [DEBUG] [pid:3704760] Release
2025-10-22 07:47:24.792 [INFO] [sources:1] sources refreshed (reaper/reaper.go:406 reaper.(*Reaper).LoadSources)
2025-10-22 07:47:24.813 [DEBUG] [database:pgwatch] [host:<pgwatch_db_host>] [pid:3704762] [port:<pgwatch_db_port>] [time:21.144979ms] Connect
2025-10-22 07:47:24.813 [DEBUG] [pid:3704762] [time:21.222434ms] Acquire
2025-10-22 07:47:24.814 [DEBUG] [sql:SELECT name, sqls, init_sql, description, node_status, gauges, is_instance_level, storage_name FROM pgwatch.metric] [alreadyPrepared:false] [name:stmtcache_341035664dcd9797523413f17c7ac1ae8a05be77189a3daf] [pid:3704762] [time:794.161µs] Prepare
2025-10-22 07:47:24.816 [DEBUG] [sql:SELECT name, sqls, init_sql, description, node_status, gauges, is_instance_level, storage_name FROM pgwatch.metric] [args:[]] [commandTag:SELECT 75] [pid:3704762] [time:2.807558ms] Query
2025-10-22 07:47:24.816 [DEBUG] [pid:3704762] Release
2025-10-22 07:47:24.816 [DEBUG] [pid:3704762] [time:842ns] Acquire
2025-10-22 07:47:24.817 [DEBUG] [sql:SELECT name, description, metrics FROM pgwatch.preset] [alreadyPrepared:false] [name:stmtcache_02da66cc46effd0fa0fe1758463c81dafaf980f82938e49a] [pid:3704762] [time:257.69µs] Prepare
2025-10-22 07:47:24.817 [DEBUG] [sql:SELECT name, description, metrics FROM pgwatch.preset] [args:[]] [commandTag:SELECT 21] [pid:3704762] [time:859.132µs] Query
2025-10-22 07:47:24.817 [DEBUG] [pid:3704762] Release
2025-10-22 07:47:24.817 [INFO] [metrics:75] [presets:21] metrics and presets refreshed (reaper/metric.go:98 reaper.(*Reaper).LoadMetrics)
2025-10-22 07:47:24.818 [ERROR] [source:pat_test] [database:] [err:failed to connect to `user=postgres database=`: /var/run/postgresql/.s.PGSQL.5432 (/var/run/postgresql): dial error: dial unix /var/run/postgresql/.s.PGSQL.5432: connect: no such file or directory] [host:/var/run/postgresql] [port:5432] [time:265.426µs] Connect
2025-10-22 07:47:24.818 [ERROR] [source:pat_test] [err:failed to connect to `user=postgres database=`: /var/run/postgresql/.s.PGSQL.5432 (/var/run/postgresql): dial error: dial unix /var/run/postgresql/.s.PGSQL.5432: connect: no such file or directory] [time:341.046µs] Acquire
2025-10-22 07:47:24.818 [WARNING] [source:pat_test] could not init connection, retrying on next iteration (reaper/reaper.go:105 reaper.(*Reaper).Reap)
2025-10-22 07:47:24.818 [DEBUG] checking if any workers need to be shut down... (reaper/reaper.go:239 reaper.(*Reaper).ShutdownOldWorkers)

It is strange that it is not parsing information from etcd. I made a fork and this or next weekend I'll try to make a build with more debug.

[pashagolub]

Would you please specify kind as patroni and check? It might be the case