Skip to content

Add Guidance wrt Labelling to Naming and Rules Best Practices #2691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion docs/practices/naming.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,18 @@ the underlying metric type and unit you work with.
* **Metric collisions**: With growing adoption and metric changes over time, there are cases where lack
of unit and type information in the metric name will cause certain series to collide (e.g. `process_cpu` for seconds and milliseconds).

## Labels
### Important Labels

* `job`
* The `job` label is a primary key to differentiate metrics from eaach other.
* If not specified in PromQL expressions, they will match unrelated metrics with the same name. This is especially true in a multi system or multi tenant installation

WARNING: When using `without`, be careful not to strip out the `job` label accidentally.

* `instance`
* The `instance` label will include the `ip:port` what was scraped, providing a crucial breadcrumb for debugging scrape time issues

### General Labelling Advice

Use labels to differentiate the characteristics of the thing that is being measured:

Expand Down
18 changes: 14 additions & 4 deletions docs/practices/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ This page documents proper naming conventions and aggregation for recording rule
Keeping the metric name unchanged makes it easy to know what a metric is and
easy to find in the codebase.

IMPORTANT: `job` label acts as a primary key. It is **strongly** recommended that you use it to scope your PromQL expressions to the system you are monitoring.

To keep the operations clean, `_sum` is omitted if there are other operations,
as `sum()`. Associative operations can be merged (for example `min_min` is the
same as `min`).
Expand All @@ -27,6 +29,18 @@ If there is no obvious operation to use, use `sum`. When taking a ratio by
doing division, separate the metrics using `_per_` and call the operation
`ratio`.

## Labels

NOTE: Omitting a label in a PromQL expression is the functional equivalent of specifying `label=*`

* In both recorded rules and alerting expressions, always specify a `job` label to prevent expression mismatches from occuring.
This is especially important in multi-tenant systems where the same metric names may be exported by different jobs or the
same job (e.g `node_exporter) in multiple, distinct deployments

* Always specify a `without` clause with the labels you are aggregating away.
This is to preserve all the other labels such as `job`, which will avoid
conflicts and give you more useful metrics and alerts.

## Aggregation

* When aggregating up ratios, aggregate up the numerator and denominator
Expand All @@ -40,10 +54,6 @@ Instead keep the metric name without the `_count` or `_sum` suffix and replace
the `rate` in the operation with `mean`. This represents the average
observation size over that time period.

* Always specify a `without` clause with the labels you are aggregating away.
This is to preserve all the other labels such as `job`, which will avoid
conflicts and give you more useful metrics and alerts.

## Examples

_Note the indentation style with outdented operators on their own line between
Expand Down