Send Prometheus metrics to Grafana Cloud #2994

ndr-ds · 2024-12-02T14:40:34Z

Motivation

We're migrating our internal validators to send data to Grafana Cloud.

Proposal

Starting with the local setup. This was useful to test the required changes that we'll likely need for the actual internal validator setup. We probably won't want to send metrics to Grafana Cloud from local runs all the time (specially for cost reasons), so this is disabled by default.

Grafana Cloud credentials are stored in a secret within GCP's Secret Manager. Since Grafana Cloud will be for our internal validators, then I figured storing the secret on GCP made sense.

Test Plan

Ran locally with linera net up --kubernetes, and saw the metrics in our Grafana Cloud instance:

Release Plan

If we want to start sending metrics to Grafana Cloud from our devnet/testnet, then:

These changes should be backported to the latest devnet branch
These changes should be backported to the latest testnet branch, then

ndr-ds · 2024-12-02T14:40:47Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

christos-h

Thanks for the cleanup.

I'm a bit confused on two things:

Do we not still want the internal Grafana which is validator specific? In which case why is the dashboard disabled? If you don't want the dashboard why not remove Grafana entirely?
The reference to the GCP secret should not be hard-coded in linera-protocol but instead be parameterised and injected by the caller in linera-infra (probably lineractl).

ndr-ds · 2024-12-03T20:00:44Z

The way I did it, either you do Grafana Cloud (writeToGrafanaCloud set to true), or you do the same existing setup we have right now. With Grafana Cloud, we don't need the Grafana pod, or the dashboards or any of that. So the idea is that for internal validators we run it with writeToGrafanaCloud set to true, and for external validators we run it with writeToGrafanaCloud set to false.
Ok, sure, I can do that. I thought it was fine since you need to be authenticated with gcloud for the secret to be successfully fetched 🤔

christos-h

Left some comments. Also the Kubernetes workflow is not passing. Although this may be unrelated to these changes.

christos-h · 2025-01-10T15:31:19Z

kubernetes/linera-validator/templates/grafana-cloud-auth-secret.yaml

+  name: grafana-cloud-auth-secret
+type: kubernetes.io/basic-auth
+stringData:
+  username: {{ .Values.grafanaCloudUsername | quote }}


What happens if this isn't specified (for example external validators which use Kuberentes?).

I'm currently changing this to only be created if writeToGrafanaCloud is true, which will only happen for internal validators

christos-h · 2025-01-10T15:32:23Z

kubernetes/linera-validator/values-local.yaml.gotmpl


 # Linera
-lineraImage: "" # Is set by helmfile.
+lineraImage: {{ env "LINERA_HELMFILE_LINERA_IMAGE" | default "linera:latest" }}


Thanks for this. I'm not entirely sure if this is better or worse to be honest. On the one hand there is less indirection on the other hand configurable variables are now in more places.

I think it definitely looks better/less confusing than just an empty string, tbh 🤔 at least now you know what the default for this is supposed to be

ndr-ds · 2025-01-10T16:12:38Z

It's related, actually. Working on a fix right now!

ndr-ds · 2025-01-29T10:43:07Z

Merge activity

Jan 29, 5:43 AM EST: A user started a stack merge that includes this pull request via Graphite.
Jan 29, 5:43 AM EST: A user merged this pull request with Graphite.

We're migrating our internal validators to send data to Grafana Cloud. Starting with the local setup. This was useful to test the required changes that we'll likely need for the actual internal validator setup. We probably won't want to send metrics to Grafana Cloud from local runs all the time (specially for cost reasons), so this is disabled by default. Grafana Cloud credentials are stored in a secret within GCP's Secret Manager. Since Grafana Cloud will be for our internal validators, then I figured storing the secret on GCP made sense. Ran locally with `linera net up --kubernetes`, and saw the metrics in our Grafana Cloud instance: ![Screenshot 2024-12-02 at 11.37.04.png](https://graphite-user-uploaded-assets-prod.s3.amazonaws.com/HlciHFAoHZW62zn13apJ/4b46c8bc-a5d3-4aad-9e3f-151cf44be947.png) If we want to start sending metrics to Grafana Cloud from our devnet/testnet, then: - These changes should be backported to the latest `devnet` branch - These changes should be backported to the latest `testnet` branch, then

## Motivation We need this to deploy one validator with Grafana Cloud ## Proposal Port to `testnet_archimedes` ## Test Plan CI

ndr-ds mentioned this pull request Dec 2, 2024

Avoid build on linera net up --kubernetes #2986

Merged

ndr-ds requested review from Twey, christos-h, jvff and ma2bd December 2, 2024 14:48

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from 00f28a0 to 4aa0e5d Compare December 2, 2024 17:52

ndr-ds force-pushed the 11-28-avoid_build_on_linera_net_up_--kubernetes branch from a872d17 to 37758e3 Compare December 2, 2024 21:11

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from 4aa0e5d to e11c5b7 Compare December 2, 2024 21:11

This was referenced Dec 3, 2024

RemoteNet -> ExistingNet #2998

Merged

Run existing-net-test in a Kubernetes local network #2999

Closed

Remove SharedLocalKubernetesNetTestingConfig #3000

Closed

christos-h suggested changes Dec 3, 2024

View reviewed changes

ndr-ds force-pushed the 11-28-avoid_build_on_linera_net_up_--kubernetes branch from 37758e3 to 1b117c3 Compare December 3, 2024 21:33

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from e11c5b7 to 188dc33 Compare December 3, 2024 21:34

ndr-ds force-pushed the 11-28-avoid_build_on_linera_net_up_--kubernetes branch 2 times, most recently from 6266ccd to 3650595 Compare December 4, 2024 14:13

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from 188dc33 to 17b1c8c Compare December 4, 2024 14:13

ndr-ds changed the base branch from 11-28-avoid_build_on_linera_net_up_--kubernetes to graphite-base/2994 December 4, 2024 14:16

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from 17b1c8c to ec0f9af Compare December 4, 2024 14:16

ndr-ds changed the base branch from graphite-base/2994 to main December 4, 2024 14:17

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from ec0f9af to e9fee28 Compare December 4, 2024 14:17

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from e9fee28 to cc8f74c Compare January 8, 2025 20:55

ndr-ds requested a review from christos-h January 8, 2025 20:57

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from cc8f74c to f682033 Compare January 10, 2025 14:50

christos-h suggested changes Jan 10, 2025

View reviewed changes

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch 2 times, most recently from 2001d5c to d158401 Compare January 10, 2025 19:27

ndr-ds mentioned this pull request Jan 10, 2025

Support path option on net up kubernetes #3113

Closed

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch 3 times, most recently from eb982c2 to 7852057 Compare January 14, 2025 14:09

ndr-ds requested a review from christos-h January 14, 2025 14:29

christos-h approved these changes Jan 14, 2025

View reviewed changes

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from 7852057 to dd685cc Compare January 14, 2025 23:37

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch 3 times, most recently from f8bcc30 to 93c1017 Compare January 28, 2025 18:28

Send Prometheus metrics to Grafana Cloud

4a85cd9

ndr-ds force-pushed the 12-02-send_prometheus_metrics_to_grafana_cloud branch from 93c1017 to 4a85cd9 Compare January 29, 2025 04:21

ndr-ds merged commit ffef3ee into main Jan 29, 2025
1 check passed

ndr-ds deleted the 12-02-send_prometheus_metrics_to_grafana_cloud branch January 29, 2025 10:43

ndr-ds added a commit that referenced this pull request Feb 21, 2025

Port of #2994 - Send Prometheus metrics to Grafana Cloud (#3373)

d021729

## Motivation We need this to deploy one validator with Grafana Cloud ## Proposal Port to `testnet_archimedes` ## Test Plan CI

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Send Prometheus metrics to Grafana Cloud #2994

Send Prometheus metrics to Grafana Cloud #2994

Uh oh!

ndr-ds commented Dec 2, 2024 •

edited

Loading

Uh oh!

ndr-ds commented Dec 2, 2024 •

edited

Loading

Uh oh!

christos-h left a comment

Uh oh!

ndr-ds commented Dec 3, 2024

Uh oh!

christos-h left a comment

Uh oh!

christos-h Jan 10, 2025

Uh oh!

ndr-ds Jan 10, 2025

Uh oh!

christos-h Jan 10, 2025

Uh oh!

ndr-ds Jan 10, 2025

Uh oh!

ndr-ds commented Jan 10, 2025

Uh oh!

ndr-ds commented Jan 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Send Prometheus metrics to Grafana Cloud #2994

Send Prometheus metrics to Grafana Cloud #2994

Uh oh!

Conversation

ndr-ds commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Proposal

Test Plan

Release Plan

Uh oh!

ndr-ds commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

christos-h left a comment

Choose a reason for hiding this comment

Uh oh!

ndr-ds commented Dec 3, 2024

Uh oh!

christos-h left a comment

Choose a reason for hiding this comment

Uh oh!

christos-h Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

ndr-ds Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

christos-h Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

ndr-ds Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

ndr-ds commented Jan 10, 2025

Uh oh!

ndr-ds commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ndr-ds commented Dec 2, 2024 •

edited

Loading

ndr-ds commented Dec 2, 2024 •

edited

Loading

ndr-ds commented Jan 29, 2025 •

edited

Loading