feat: add monitoring stack

[sxd]

Every cluster now has the podMonitor enabled and the script scripts/setup_monitoring.sh was created to deploy Prometheus Operator and Grafana Operator, using these operators we create:

• Prometheus instance
• Grafana Deployment
• Grafana Datasource (connecting to Prometheus instance)
• Grafana Dashboard (from the project cloudnative-pg/grafana-dashboard)

After the cluster are updated the Grafana dashboard can be accessed redirecting the 3000 port of the Grafana deployment (namespace grafana) on any of the two kind clusters.

The dashboard is loaded from the URL of the json file in the repository, this means that if the content is changed also the dashboard will be updated.

[smiyc]

I tested the setup_monitoring.sh and it failed because my laptop didn't had kustomize installed.
Therefor I would add kustomize to the prereqs in the setup.sh script or add a prereq check into setup_monitoring.sh

[gbartolini]

I would like to keep the requirements to a minimal level, without requiring development tools. We shouldn't forget that the audience are not developers.

[smiyc]

so instead of kustomize, kubectl apply -f < the directory >?

[sxd]

@smiyc I did some minor changes and removed the needed of kustomize, can you check again please! :D

[smiyc]

@sxd the monitoring stack is set up correctly without kustomize.
The cloudnativepg-dashboard GrafanaDashboard is applied, but I don't see it in grafana

[sxd]

@smiyc it should be in the playlists section, can you check please? :D

[smiyc]

@smiyc it should be in the playlists section, can you check please? :D

nope

[sxd]

@smiyc will create a fresh environment to test, and check, but in the meantime, can you check the status of the grafana operator ? looking for errors or something.
None of the command failed right?

[smiyc]

@sxd

the operator wants to download the dashboard from my external IP address...

│ 2024-10-07T20:37:58Z    ERROR    GrafanaDashboardReconciler    error fetching dashboard    {"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"cloudnativepg-dashboard","namespace":"grafana"}, "namespace": "grafana", "na │
│ me": "cloudnativepg-dashboard", "reconcileID": "818bb948-ce37-4bdb-8507-9b5730b23cb8", "dashboard": "cloudnativepg-dashboard", "error": "dial tcp XXX:443: i/o timeout"}   

a nslookup raw.githubusercontent.com from inside the pod as well outside docker is looking good

...after connecting via hotspot, the dashboard was downloaded...
so the problem is in my setup, nevermind

[gbartolini]

@sxd can you please review this? please do not merge as we need to add Leonardo as co author.

[sxd]

@gbartolini on it!! will modify the commit message too ;) BTW nice changes!

[smiyc]

there is a variable mismatch context / region in the grafana deployment

I can't commit to this PR, do you fix it on your side or should I create a fork and commit there?

[ardentperf]

what's the latest status on this PR?

[ardentperf]

I gave this PR an end-to-end test, twice in a row and then a third time on a completely clean setup (starting with a complete fresh/clean OS install before installing the all prereqs and then the CNPG playground). Consistent results in all three attempts.

EU cluster: only the prometheus operator pod starts
US cluster: four pods - prometheus operator, prometheus deployment, grafana operator, grafana deployment

After forwarding the port, I can connect to grafana in the US. It has the CNPG dashboard however it does not have any metrics. I haven't debugged it yet, but wanted to leave the initial feedback here FYI.

nb. when I cut and past the helm-based instructions from the quick start in the official docs, these work - everything starts in both EU and US, and the metrics appear on the CNPG dashboard. My overall environment seems to work; I need to look closer to see what's happening with the monitoring script in this PR

Three other notes:

• missing a teardown script
• default user/password should be explicitly set, i would suggest same password that's used in official quick start documentation. also should add the user/password to the README here
• similar to cert-manager and cnpg operator can run on app node instead of infra node #29 - i also saw this running on app nodes, when it probably should be on infra or control plane