-
Notifications
You must be signed in to change notification settings - Fork 478
feat(otelcol.connector): adding connector count #4913
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
hhertout
wants to merge
3
commits into
grafana:main
Choose a base branch
from
hhertout:feature/otel-connector-count
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,152
−26
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
370 changes: 370 additions & 0 deletions
370
docs/sources/reference/components/otelcol/otelcol.connector.count.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,370 @@ | ||
| --- | ||
| canonical: https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.connector.count/ | ||
| description: Learn about otelcol.connector.count | ||
| labels: | ||
| stage: experimental | ||
| products: | ||
| - oss | ||
| title: otelcol.connector.count | ||
| --- | ||
|
|
||
| # `otelcol.connector.count` | ||
|
|
||
| `otelcol.connector.count` accepts spans, span events, metrics, data points, and log records from other `otelcol` components and generates metrics that count the received telemetry data. | ||
|
|
||
| {{< admonition type="note" >}} | ||
| `otelcol.connector.count` is a wrapper over the upstream OpenTelemetry Collector Contrib `count` connector. | ||
| Bug reports or feature requests are redirected to the upstream repository if necessary. | ||
| {{< /admonition >}} | ||
|
|
||
| You can specify multiple `otelcol.connector.count` components by giving them different labels. | ||
|
|
||
| ## Usage | ||
|
|
||
| ```alloy | ||
| otelcol.connector.count "<LABEL>" { | ||
| output { | ||
| metrics = [...] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Arguments | ||
|
|
||
| The `otelcol.connector.count` component doesn't support any arguments. You can configure this component with blocks. | ||
|
|
||
| ## Blocks | ||
|
|
||
| You can use the following blocks with `otelcol.connector.count`: | ||
|
|
||
| | Block | Description | Required | | ||
| | ----------------------------------------- | -------------------------------------------------------------------------- | -------- | | ||
| | [`output`][output] | Configures where to send received telemetry data. | yes | | ||
| | [`datapoints`][datapoints] | Configures custom metrics for counting data points. | no | | ||
| | `datapoints` > [`attributes`][attributes] | Groups data point counts by attribute values. | no | | ||
| | [`debug_metrics`][debug_metrics] | Configures the metrics that this component generates to monitor its state. | no | | ||
| | [`logs`][logs] | Configures custom metrics for counting log records. | no | | ||
| | `logs` > [`attributes`][attributes] | Groups log counts by attribute values. | no | | ||
| | [`metrics`][metrics] | Configures custom metrics for counting metrics. | no | | ||
| | `metrics` > [`attributes`][attributes] | Groups metric counts by attribute values. | no | | ||
| | [`spanevents`][spanevents] | Configures custom metrics for counting span events. | no | | ||
| | `spanevents` > [`attributes`][attributes] | Groups span event counts by attribute values. | no | | ||
| | [`spans`][spans] | Configures custom metrics for counting spans. | no | | ||
| | `spans` > [`attributes`][attributes] | Groups span counts by attribute values. | no | | ||
|
|
||
| The `>` symbol indicates deeper levels of nesting. | ||
| For example, `spans` > `attributes` refers to an `attributes` block defined inside a `spans` block. | ||
|
|
||
| [spans]: #spans | ||
| [spanevents]: #spanevents | ||
| [metrics]: #metrics | ||
| [datapoints]: #datapoints | ||
| [logs]: #logs | ||
| [attributes]: #attributes | ||
| [output]: #output | ||
| [debug_metrics]: #debug_metrics | ||
|
|
||
| ### `spans` | ||
|
|
||
| The `spans` block configures a custom metric for counting spans. | ||
|
|
||
| You can specify multiple `spans` blocks to define different metrics. | ||
|
|
||
| If you don't define any `spans` blocks, the connector emits a default metric named `trace.span.count`. | ||
|
|
||
| | Name | Type | Description | Default | Required | | ||
| | ------------- | -------------- | ------------------------------------- | ------------------------------------------ | -------- | | ||
| | `conditions` | `list(string)` | OTTL expressions for filtering spans. | `[]` | no | | ||
| | `description` | `string` | Metric description. | `""` | no | | ||
| | `name` | `string` | Metric name. | Uses default name based on telemetry type. | no | | ||
|
|
||
| The `conditions` argument accepts a list of [OTTL][] expressions that filter which spans to count. | ||
| If any condition matches, the connector counts the span. | ||
| If you don't specify conditions, the connector counts all spans. | ||
|
|
||
| [OTTL]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl | ||
|
|
||
| ### `spanevents` | ||
|
|
||
| The `spanevents` block configures a custom metric for counting span events. | ||
|
|
||
| You can specify multiple `spanevents` blocks to define different metrics. | ||
|
|
||
| If you don't define any `spanevents` blocks, the connector emits a default metric named `trace.span.event.count`. | ||
|
|
||
| This block shares the same configuration structure as [`spans`][spans]. | ||
| Refer to the [`spans`][spans] block documentation for the complete list of supported arguments and blocks. | ||
|
|
||
| ### `metrics` | ||
|
|
||
| The `metrics` block configures a custom metric for counting metrics. | ||
|
|
||
| You can specify multiple `metrics` blocks to define different metrics. | ||
|
|
||
| If you don't define any `metrics` blocks, the connector emits a default metric named `metric.count`. | ||
|
|
||
| This block shares the same configuration structure as [`spans`][spans]. | ||
| Refer to the [`spans`][spans] block documentation for the complete list of supported arguments and blocks. | ||
|
|
||
| ### `datapoints` | ||
|
|
||
| The `datapoints` block configures a custom metric for counting data points. | ||
|
|
||
| You can specify multiple `datapoints` blocks to define different metrics. | ||
|
|
||
| If you don't define any `datapoints` blocks, the connector emits a default metric named `metric.datapoint.count`. | ||
|
|
||
| This block shares the same configuration structure as [`spans`][spans]. | ||
| Refer to the [`spans`][spans] block documentation for the complete list of supported arguments and blocks. | ||
|
|
||
| ### `logs` | ||
|
|
||
| The `logs` block configures a custom metric for counting log records. | ||
|
|
||
| You can specify multiple `logs` blocks to define different metrics. | ||
|
|
||
| If you don't define any `logs` blocks, the connector emits a default metric named `log.record.count`. | ||
|
|
||
| This block shares the same configuration structure as [`spans`][spans]. | ||
| Refer to the [`spans`][spans] block documentation for the complete list of supported arguments and blocks. | ||
|
|
||
| ### `attributes` | ||
|
|
||
| The `attributes` block specifies an attribute to use for grouping counted telemetry data. | ||
|
|
||
| Each unique combination of attribute values generates a separate data point on the metric. | ||
|
|
||
| You can specify multiple `attributes` blocks within `spans`, `spanevents`, `metrics`, `datapoints`, or `logs` blocks to group by multiple attributes. | ||
|
|
||
| The following arguments are supported: | ||
|
|
||
| | Name | Type | Description | Default | Required | | ||
| | --------------- | -------- | --------------------------------------------- | ------- | -------- | | ||
| | `key` | `string` | Attribute key name. | | yes | | ||
| | `default_value` | `any` | Default value if the attribute doesn't exist. | | no | | ||
|
|
||
| Attribute precedence: span/log/metric attributes > scope attributes > resource attributes. | ||
|
|
||
| ### `output` | ||
|
|
||
| {{< badge text="Required" >}} | ||
|
|
||
| {{< docs/shared lookup="reference/components/output-block.md" source="alloy" version="<ALLOY_VERSION>" >}} | ||
|
|
||
| ### `debug_metrics` | ||
|
|
||
| {{< docs/shared lookup="reference/components/otelcol-debug-metrics-block.md" source="alloy" version="<ALLOY_VERSION>" >}} | ||
|
|
||
| ## Exported fields | ||
|
|
||
| The following fields are exported and can be referenced by other components: | ||
|
|
||
| | Name | Type | Description | | ||
| | ------- | ------------------ | ---------------------------------------------------------------- | | ||
| | `input` | `otelcol.Consumer` | A value that other components can use to send telemetry data to. | | ||
|
|
||
| `input` accepts `otelcol.Consumer` data for traces, metrics, and logs. | ||
|
|
||
| he component counts all telemetry received through `input` according to the configured blocks and emits metrics to the components specified in `output`. | ||
|
|
||
| ## Component health | ||
|
|
||
| `otelcol.connector.count` is only reported as unhealthy if given an invalid configuration. | ||
|
|
||
| ## Debug information | ||
|
|
||
| `otelcol.connector.count` doesn't expose any component-specific debug information. | ||
|
|
||
| ## Examples | ||
|
|
||
| ### Default configuration | ||
|
|
||
| Use the count connector with minimal configuration to count all telemetry data using default metric names. | ||
|
|
||
| ```alloy | ||
| otelcol.connector.count "default" { | ||
| output { | ||
| metrics = [otelcol.exporter.otlp.default.input] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| This configuration generates the following default metrics: | ||
|
|
||
| - `trace.span.count` - Count of all spans | ||
| - `trace.span.event.count` - Count of all span events | ||
| - `metric.count` - Count of all metrics | ||
| - `metric.datapoint.count` - Count of all data points | ||
| - `log.record.count` - Count of all log records | ||
|
|
||
| ### Custom metrics with conditions and attributes | ||
|
|
||
| Create custom count metrics with filtering conditions and group counts by specific attributes. | ||
|
|
||
| ```alloy | ||
| otelcol.connector.count "default" { | ||
| // Count only HTTP GET spans | ||
| spans { | ||
| name = "http_get_requests" | ||
| description = "Number of HTTP GET requests" | ||
| conditions = [ | ||
| "attributes[\"http.method\"] == \"GET\"", | ||
| ] | ||
| } | ||
|
|
||
| // Count spans grouped by service and environment | ||
| spans { | ||
| name = "spans_by_service" | ||
| description = "Spans per service and environment" | ||
| attributes { | ||
| key = "service.name" | ||
| default_value = "unknown" | ||
| } | ||
| attributes { | ||
| key = "env" | ||
| default_value = "production" | ||
| } | ||
| } | ||
|
|
||
| // Count error and fatal logs only | ||
| logs { | ||
| name = "error_logs" | ||
| description = "Error and fatal log records" | ||
| conditions = [ | ||
| "severity_number >= 17", | ||
| ] | ||
| } | ||
|
|
||
| // Count logs by environment | ||
| logs { | ||
| name = "logs_by_env" | ||
| description = "Log records per environment" | ||
| attributes { | ||
| key = "env" | ||
| } | ||
| } | ||
|
|
||
| output { | ||
| metrics = [otelcol.exporter.otlp.default.input] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Complete pipeline with Prometheus export | ||
|
|
||
| Count spans, logs, and metrics, then export to Prometheus using the delta to cumulative processor. | ||
|
|
||
| ```alloy | ||
| otelcol.receiver.otlp "default" { | ||
| grpc { | ||
| endpoint = "127.0.0.1:4317" | ||
| } | ||
|
|
||
| http { | ||
| endpoint = "127.0.0.1:4318" | ||
| } | ||
|
|
||
| output { | ||
| traces = [otelcol.connector.count.default.input] | ||
| metrics = [otelcol.connector.count.default.input] | ||
| logs = [otelcol.connector.count.default.input] | ||
| } | ||
| } | ||
|
|
||
| otelcol.connector.count "default" { | ||
| spans { | ||
| name = "traces_total" | ||
| description = "Total number of spans received" | ||
| } | ||
|
|
||
| spans { | ||
| name = "http_get_spans" | ||
| description = "HTTP GET requests" | ||
| conditions = [ | ||
| "attributes[\"http.method\"] == \"GET\"", | ||
| ] | ||
| } | ||
|
|
||
| spans { | ||
| name = "spans_by_service" | ||
| description = "Spans grouped by service" | ||
| attributes { | ||
| key = "service.name" | ||
| default_value = "unknown" | ||
| } | ||
| } | ||
|
|
||
| logs { | ||
| name = "logs_total" | ||
| description = "Total number of logs received" | ||
| } | ||
|
|
||
| logs { | ||
| name = "error_logs" | ||
| description = "Error level logs" | ||
| conditions = [ | ||
| "severity_number >= 17", | ||
| ] | ||
| } | ||
|
|
||
| metrics { | ||
| name = "metrics_total" | ||
| description = "Total number of metrics received" | ||
| } | ||
|
|
||
| output { | ||
| metrics = [otelcol.processor.deltatocumulative.default.input] | ||
| } | ||
| } | ||
|
|
||
| // Convert delta metrics to cumulative for Prometheus compatibility | ||
| otelcol.processor.deltatocumulative "default" { | ||
| output { | ||
| metrics = [otelcol.exporter.prometheus.default.input] | ||
| } | ||
| } | ||
|
|
||
| otelcol.exporter.prometheus "default" { | ||
| forward_to = [prometheus.remote_write.default.receiver] | ||
| add_metric_suffixes = false | ||
| resource_to_telemetry_conversion = true | ||
| } | ||
|
|
||
| prometheus.remote_write "default" { | ||
| endpoint { | ||
| url = "http://localhost:9090/api/v1/write" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Technical details | ||
|
|
||
| `otelcol.connector.count` uses the count connector from OpenTelemetry Collector Contrib. | ||
|
|
||
| All generated metrics use the Sum data type with Delta aggregation temporality. | ||
|
|
||
| {{< admonition type="note" >}} | ||
| Prometheus doesn't natively support delta metrics. | ||
| Use [`otelcol.processor.deltatocumulative`][otelcol.processor.deltatocumulative] to convert delta metrics to cumulative before sending to Prometheus. | ||
|
|
||
| [otelcol.processor.deltatocumulative]: ../otelcol.processor.deltatocumulative/ | ||
| {{< /admonition >}} | ||
|
|
||
| <!-- START GENERATED COMPATIBLE COMPONENTS --> | ||
|
|
||
| ## Compatible components | ||
|
|
||
| `otelcol.connector.count` can accept arguments from the following components: | ||
|
|
||
| - Components that export [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-exporters) | ||
|
|
||
| `otelcol.connector.count` has exports that can be consumed by the following components: | ||
|
|
||
| - Components that consume [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-consumers) | ||
|
|
||
| {{< admonition type="note" >}} | ||
| Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. | ||
| Refer to the linked documentation for more details. | ||
| {{< /admonition >}} | ||
|
|
||
| <!-- END GENERATED COMPATIBLE COMPONENTS --> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this is an experimental component, we need to add the shared element for experimental.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you a lot for you review ! I've made the changes.
Though I've a question about this comment:
I don't know if I have to do it tbh... what's the default recommendation for this ? Should I mark it as experimental ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a couple of things here we should verify.
experimental? If yes, then we need the lable.stage in the metadata (already there) and the shared text block include (suggested). If no, then we need to remove or change the label.stage in the metadata and then skip this suggestion.I'll defer to @grafana/grafana-alloy-maintainers for the decision on which way to go, and I can provide the doc suggestions to align.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your answer !