From 67dfe22607f07afb895ef104ccfb201ae3299fe7 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Wed, 2 Jul 2025 14:31:58 -0600 Subject: [PATCH 01/24] Initial draft. --- content/en/opentelemetry/correlate/_index.md | 311 ++++++++++++++++++- 1 file changed, 306 insertions(+), 5 deletions(-) diff --git a/content/en/opentelemetry/correlate/_index.md b/content/en/opentelemetry/correlate/_index.md index 55b94feb129b4..06f03b3a9fc69 100644 --- a/content/en/opentelemetry/correlate/_index.md +++ b/content/en/opentelemetry/correlate/_index.md @@ -1,5 +1,6 @@ --- title: Correlate Data +description: Learn how to correlate your OpenTelemetry traces, metrics, logs, and other telemetry in Datadog to get a unified view of your application's performance. aliases: - /opentelemetry/otel_logs/ further_reading: @@ -10,12 +11,312 @@ further_reading: ## Overview -Link your telemetry data for full-stack observability: +Getting a unified view of your application's performance requires connecting its traces, metrics, logs, and user interactions. This guide provides the steps to correlate your OpenTelemetry data in Datadog, allowing you to seamlessly navigate between all related telemetry in a single view. -{{< whatsnext desc=" " >}} - {{< nextlink href="/opentelemetry/correlate/logs_and_traces/" >}}Connect Logs and Traces{{< /nextlink >}} - {{< nextlink href="/opentelemetry/correlate/rum_and_traces/" >}}Connect RUM and Traces{{< /nextlink >}} -{{< /whatsnext >}} +Foundational Tagging: The Prerequisite for All Correlation +To tie your telemetry together, Datadog uses three standard tags: env, service, and version. Configuring these tags is the most critical step for enabling correlation across the Datadog platform. + +When using OpenTelemetry, you set these tags by defining a standard set of resource attributes. Datadog automatically maps these attributes to the correct tags. + +OpenTelemetry Resource Attribute + +Datadog Tag + +service.name + +service + +deployment.environment + +env + +service.version + +version + +You can set these attributes in your application's environment variables, SDK, or in the OpenTelemetry Collector. + +Configuration +{{< tabs >}} +{{% tab "Environment Variables" %}} + +Set the OTEL_RESOURCE_ATTRIBUTES environment variable with your service's information: + +export OTEL_RESOURCE_ATTRIBUTES="service.name=my-service,deployment.environment=production,service.version=1.2.3" + +{{% /tab %}} +{{% tab "SDK" %}} + +Create a Resource with the required attributes and associate it with your TracerProvider in your application code. + +Here's an example using the OpenTelemetry SDK for Python: + +from opentelemetry.sdk.resources import Resource +from opentelemetry.sdk.trace import TracerProvider + +resource = Resource(attributes={ + "service.name": "", + "deployment.environment": "", + "service.version": "" +}) +tracer_provider = TracerProvider(resource=resource) + +{{% /tab %}} +{{% tab "Collector" %}} + +Use the transform processor in your Collector configuration to set the resource attributes on your telemetry data: + +processors: + transform: + trace_statements: + - context: resource + statements: + - set(attributes["service.name"], "my-service") + - set(attributes["deployment.environment"], "production") + - set(attributes["service.version"], "1.2.3") + +{{% /tab %}} +{{< /tabs >}} + +Correlating Core Telemetry +Traces and Metrics +Why it matters: Connecting traces to infrastructure metrics allows you to pivot from a slow request directly to the CPU and memory metrics of the host it ran on, helping you instantly determine if resource contention was the root cause. + +Essential Attributes +Correlation between traces and infrastructure metrics relies on the following resource attributes: + +host.name: For correlating with host metrics (CPU, memory, disk). + +container.id: For correlating with container metrics. + +Setup Instructions +Enable the Host Metrics Receiver +To collect system metrics, enable the hostmetrics receiver in your OpenTelemetry Collector configuration. This receiver gathers metrics like CPU, memory, disk, and network usage. + +Add the following to your Collector configuration file (config.yaml): + +receivers: + hostmetrics: + collection_interval: 10s + scrapers: + cpu: + disk: + load: + filesystem: + memory: + network: + paging: + processes: + +service: + pipelines: + metrics: + receivers: [hostmetrics] + # ... other processors and exporters + +Ensure Consistent Host and Container Tagging +For correlation to work, the host.name (or container.id) attribute on your traces must match the corresponding attribute on the metrics collected by the hostmetrics receiver. The OpenTelemetry Collector running as an agent on the host typically discovers this automatically. + +Traces and Logs +Why it matters: Connecting traces and logs allows you to jump from a specific span in a trace to the exact logs that were generated during that operation, making debugging faster and more intuitive. + +{{< alert "warning" >}} +Previous documentation that described complex, language-specific conversions of span_id is outdated and no longer necessary. +{{< /alert >}} + +Connecting traces and logs has been simplified. The only requirement is to inject the active trace_id and span_id from your application's trace context into your logs. + +Setup Instructions +Configure Automatic Context Injection +Most OpenTelemetry logging instrumentation can automatically inject the active trace context into your logs. Ensure this feature is enabled for your logging library. Your logs must be sent in JSON format for Datadog to parse the attributes correctly. + +Verify the Log Output +After configuration, your JSON logs should contain the trace_id and span_id attributes, which Datadog uses to link the log to the exact trace and span that was active when the log was generated. + +Here is an example of a properly formatted JSON log entry: + +{ + "timestamp": 1720557413000, + "level": "INFO", + "message": "Processing user request", + "service": "my-service", + "env": "production", + "version": "1.2.3", + "trace_id": "8738839999436033394", + "span_id": "1006654203791334032" +} + +{{< alert "info" >}} +For complete, working examples of trace and log correlation for various languages, see the Datadog OpenTelemetry Examples repository. +{{< /alert >}} + +Correlating with Datadog Products +Real User Monitoring (RUM) +Why it matters: By connecting RUM and APM, you can follow a user's journey from a slow button click on your website all the way to the specific backend database query that caused the delay. + +This is achieved by configuring the RUM SDK to inject trace context headers into requests made to your backend. The default injection style is tracecontext, Datadog. + +{{< tabs >}} +{{% tab "Browser" %}} + +Ensure you have completed the standard Browser RUM setup. + +Modify allowedTracingUrls in your RUM SDK initialization to specify which propagator to use for which backend endpoint. + +import { datadogRum } from '@datadog/browser-rum' + +datadogRum.init({ + // ...otherConfig, + allowedTracingUrls: [ + { match: "[https://api.example.com](https://api.example.com)", propagatorTypes: ["tracecontext"]} + ] +}) +```propagatorTypes` accepts a list of desired propagators: + - `datadog`: Datadog's propagator (`x-datadog-*`) + - `tracecontext`: [W3C Trace Context](https://www.w3.org/TR/trace-context/) (`traceparent`, `tracestate`) + - `b3`: [B3 single header](https://github.com/openzipkin/b3-propagation#single-header) (`b3`) + - `b3multi`: [B3 multiple headers](https://github.com/openzipkin/b3-propagation#multiple-headers) (`X-B3-*`) + + +{{% /tab %}} +{{% tab "iOS" %}} + +Ensure you have completed the standard iOS RUM setup. + +Use .traceWithHeaders(hostsWithHeaders:sampleRate:) to specify which tracing headers to inject for requests to your backend hosts. + +RUM.enable( + with: RUM.Configuration( + applicationID: "", + urlSessionTracking: .init( + firstPartyHostsTracing: .traceWithHeaders( + hostsWithHeaders: [ + "api.example.com": [.tracecontext] + ], + sampleRate: 100 + ) + ) + ) +) +```TracingHeaderType` is an enum that includes `.datadog`, `.tracecontext`, `.b3`, and `.b3multi`. + + +{{% /tab %}} +{{% tab "Android" %}} + +Ensure you have completed the standard Android RUM setup. + +Configure your OkHttpClient interceptor with a map of your backend hosts and the desired TracingHeaderType for each. + +val tracedHosts = mapOf("example.com" to setOf(TracingHeaderType.TRACECONTEXT), + "example.eu" to setOf(TracingHeaderType.DATADOG)) + +val okHttpClient = OkHttpClient.Builder() + .addInterceptor(DatadogInterceptor.Builder(tracedHosts).build()) + .addNetworkInterceptor(TracingInterceptor.Builder(tracedHosts).build()) + .eventListenerFactory(DatadogEventListener.Factory()) + .build() +```TracingHeaderType` is an enum that includes `.DATADOG`, `.TRACECONTEXT`, `.B3`, and `.B3MULTI`. + + +{{% /tab %}} +{{< /tabs >}} + +For React Native, Flutter, and Kotlin Multiplatform, refer to the platform-specific RUM documentation for configuring firstPartyHosts, firstPartyHostsWithTracingHeaders, or the datadogKtorPlugin respectively. + +Database Monitoring (DBM) +Why it matters: Connect backend traces to detailed database performance data in Datadog, including query metrics and execution plans, to identify the exact queries that are slowing down your application. + +Step 1: Instrument Your Database Spans +For DBM correlation to work, your database spans must include the following attributes. + +Attribute + +Description + +Example + +span.type + +Required. The type of span. Must be sql, postgres, mysql, or sql.query. + +sql + +span.service + +Required. The name of the service executing the query. This should match your foundational service tag. + +my-web-app + +span.resource + +Required. The raw SQL query text. This is critical for Datadog's obfuscation and normalization. + +SELECT * FROM users WHERE id = ? + +db.name + +The logical database or schema name being queried. + +user_accounts + +db.system + +The database technology. + +postgres + +Step 2: Configure Your Ingest Path +Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. + +{{< tabs >}} +{{% tab "OTel Collector" %}} + +OpenTelemetry Collector with Datadog Exporter + +When starting the Collector, enable the datadog.EnableOperationAndResourceNameV2 feature gate. This gate is available in v0.118.0 and later. + +otelcontribcol --config=config.yaml \ +--feature-gates=datadog.EnableOperationAndResourceNameV2 + +{{% /tab %}} +{{% tab "Datadog Agent (OTLP Ingest)" %}} + +Datadog Agent with OTLP Ingest + +In your Datadog Agent configuration, ensure the DD_APM_FEATURES environment variable includes enable_operation_and_resource_name_logic_v2. For more information, see the Agent Configuration Documentation. + +{{% /tab %}} +{{% tab "Datadog Agent (Embedded OTel)" %}} + +Datadog Agent with embedded OTel Collector + +If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gates in your values.yaml. For more information, see the Datadog Helm Chart Documentation. + +datadog: + otelCollector: + featureGates: datadog.EnableOperationAndResourceNameV2 + +{{% /tab %}} +{{< /tabs >}} + +APM and Infrastructure +Why it matters: Connecting your APM traces to Kubernetes metrics allows you to view pod and container performance directly in the Infrastructure tab of the APM Service Page. + +This correlation requires configuring specific receivers in your OpenTelemetry Collector to gather Kubernetes metrics. + +{{< alert "info" >}} +This feature is in active development. Detailed documentation on configuring the required Kubernetes receivers is forthcoming. +{{< /alert >}} + +What's Next? +Now that your telemetry is correlated, you can explore the unified views in Datadog: + +Service Catalog: View the health of all your services in one place. + +Trace Explorer: Search and analyze individual requests as they travel across your services. + +Log Explorer: Search and filter all your logs, and pivot to related traces with a single click. ## Further reading From ffb9f0b3ec0daee5035f2a809f170c859c1833cc Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 8 Jul 2025 17:26:37 -0600 Subject: [PATCH 02/24] Flesh out content into second draft. --- content/en/opentelemetry/correlate/_index.md | 326 ++++++++++--------- 1 file changed, 179 insertions(+), 147 deletions(-) diff --git a/content/en/opentelemetry/correlate/_index.md b/content/en/opentelemetry/correlate/_index.md index 06f03b3a9fc69..bcdd79eca94ae 100644 --- a/content/en/opentelemetry/correlate/_index.md +++ b/content/en/opentelemetry/correlate/_index.md @@ -1,5 +1,5 @@ --- -title: Correlate Data +title: Correlate OpenTelemetry Data description: Learn how to correlate your OpenTelemetry traces, metrics, logs, and other telemetry in Datadog to get a unified view of your application's performance. aliases: - /opentelemetry/otel_logs/ @@ -11,38 +11,31 @@ further_reading: ## Overview -Getting a unified view of your application's performance requires connecting its traces, metrics, logs, and user interactions. This guide provides the steps to correlate your OpenTelemetry data in Datadog, allowing you to seamlessly navigate between all related telemetry in a single view. +Getting a unified view of your application's performance requires connecting its traces, metrics, logs, user interactions, and more. This guide provides the steps to correlate your OpenTelemetry data in Datadog, allowing you to navigate between all related telemetry in a single view. -Foundational Tagging: The Prerequisite for All Correlation -To tie your telemetry together, Datadog uses three standard tags: env, service, and version. Configuring these tags is the most critical step for enabling correlation across the Datadog platform. +## Prerequisite: Unified service tagging -When using OpenTelemetry, you set these tags by defining a standard set of resource attributes. Datadog automatically maps these attributes to the correct tags. - -OpenTelemetry Resource Attribute - -Datadog Tag - -service.name - -service +Datadog uses three standard tags to link telemetry together: `env`, `service`, and `version`. To apply these tags, you must configure the corresponding resource attributes in OpenTelemetry. -deployment.environment - -env - -service.version +When using OpenTelemetry, you set these tags by defining a standard set of resource attributes. Datadog automatically maps these attributes to the correct tags. -version +| OpenTelemetry Resource Attribute | Datadog Tag | Notes | +|----------------------------------|-------------|---------------------------------------------------------------------------------------------------------| +| `deployment.environment.name` | `env` | **Recommended**. Supported in Agent v7.58.0+ and Collector Exporter v0.110.0+. | +| `deployment.environment` | `env` | Use if you are running an Agent version older than v7.58.0 or a Collector Exporter older than v0.110.0. | +| `service.name` | `service` | | +| `service.version` | `version` | | You can set these attributes in your application's environment variables, SDK, or in the OpenTelemetry Collector. -Configuration {{< tabs >}} {{% tab "Environment Variables" %}} -Set the OTEL_RESOURCE_ATTRIBUTES environment variable with your service's information: +Set the `OTEL_RESOURCE_ATTRIBUTES` environment variable with your service's information: -export OTEL_RESOURCE_ATTRIBUTES="service.name=my-service,deployment.environment=production,service.version=1.2.3" +```bash +export OTEL_RESOURCE_ATTRIBUTES="service.name=my-service,deployment.environment.name=production,service.version=1.2.3" +``` {{% /tab %}} {{% tab "SDK" %}} @@ -51,90 +44,151 @@ Create a Resource with the required attributes and associate it with your Tracer Here's an example using the OpenTelemetry SDK for Python: +```python from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider resource = Resource(attributes={ "service.name": "", - "deployment.environment": "", + "deployment.environment.name": "", "service.version": "" }) tracer_provider = TracerProvider(resource=resource) +``` {{% /tab %}} {{% tab "Collector" %}} Use the transform processor in your Collector configuration to set the resource attributes on your telemetry data: +```yaml processors: transform: trace_statements: - context: resource statements: - set(attributes["service.name"], "my-service") - - set(attributes["deployment.environment"], "production") + - set(attributes["deployment.environment.name"], "production") - set(attributes["service.version"], "1.2.3") +``` {{% /tab %}} {{< /tabs >}} -Correlating Core Telemetry -Traces and Metrics -Why it matters: Connecting traces to infrastructure metrics allows you to pivot from a slow request directly to the CPU and memory metrics of the host it ran on, helping you instantly determine if resource contention was the root cause. +## Correlating telemetry + +### Traces and metrics + +Correlating traces with infrastructure metrics allows you to pivot from a slow request directly to the CPU and memory metrics of the host or container it ran on. This helps you determine if resource contention was the root cause of a performance issue. -Essential Attributes Correlation between traces and infrastructure metrics relies on the following resource attributes: -host.name: For correlating with host metrics (CPU, memory, disk). +- `host.name`: For correlating with host metrics (CPU, memory, disk). +- `container.id`: For correlating with container metrics. -container.id: For correlating with container metrics. +#### Setup -Setup Instructions -Enable the Host Metrics Receiver -To collect system metrics, enable the hostmetrics receiver in your OpenTelemetry Collector configuration. This receiver gathers metrics like CPU, memory, disk, and network usage. +##### Enable the host metrics receiver -Add the following to your Collector configuration file (config.yaml): +To collect system-level metrics, enable the `hostmetrics` receiver in your OpenTelemetry Collector configuration. This receiver gathers metrics like CPU, memory, disk, and network usage. +```yaml receivers: hostmetrics: - collection_interval: 10s + collection_interval: 30s scrapers: cpu: - disk: - load: - filesystem: memory: - network: - paging: - processes: + +processors: + batch: + +exporters: + datadog: {} service: pipelines: metrics: receivers: [hostmetrics] - # ... other processors and exporters + processors: [batch] + exporters: [datadog] +``` + +For detailed setup instructions, see [Host Metrics][2] documentation. + +##### Ensure consistent host and container tagging + +For correlation to work, the `host.name` (or `container.id`) attribute on your traces must match the corresponding attribute on the metrics collected by the `hostmetrics` receiver. When running the OpenTelemetry Collector as an agent on the host, it typically discovers this information automatically. + +### Traces and logs + +Correlating traces with logs allows you to jump from a specific span in a trace to the exact logs that were generated during that operation, making debugging faster and more intuitive. + +The only requirement for connecting traces and logs is to inject the active `trace_id` and `span_id` from your application's trace context into your logs. + +#### Setup -Ensure Consistent Host and Container Tagging -For correlation to work, the host.name (or container.id) attribute on your traces must match the corresponding attribute on the metrics collected by the hostmetrics receiver. The OpenTelemetry Collector running as an agent on the host typically discovers this automatically. +Most OpenTelemetry logging instrumentation can automatically inject the active trace context into your logs. Ensure this feature is enabled for your logging library. For Datadog to parse the attributes correctly, your logs must be sent in JSON format. -Traces and Logs -Why it matters: Connecting traces and logs allows you to jump from a specific span in a trace to the exact logs that were generated during that operation, making debugging faster and more intuitive. +For complete example applications, see the [Datadog OpenTelemetry Examples repository][1]. -{{< alert "warning" >}} -Previous documentation that described complex, language-specific conversions of span_id is outdated and no longer necessary. -{{< /alert >}} +{{< tabs >}} +{{% tab "Go" %}} + +For Go applications using a structured logger such as `zap`, the recommended approach is to use a bridge such as **`otelzap`**. This wraps your logger and automatically injects the active `trace_id` and `span_id` into every log message. + +First, ensure you have an initialized OpenTelemetry `LoggerProvider`. Then, use it to create your `zap` logger instance: + +```go +import "go.opentelemetry.io/contrib/bridges/otelzap" + +// Replace your standard zap logger with the otelzap-backed one +logger := zap.New(otelzap.NewCore( + "my-service-name", + otelzap.WithLoggerProvider(loggerProvider), +)) + +// Now, logs written with this logger are automatically correlated +logger.Info("Processing user request") +``` + +To see how the `LoggerProvider` is configured in a complete application, see the [full Go example in the examples repository][100]. + +[100]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/golang/calendar/main.go + +{{% /tab %}} -Connecting traces and logs has been simplified. The only requirement is to inject the active trace_id and span_id from your application's trace context into your logs. +{{% tab "Java" %}} -Setup Instructions -Configure Automatic Context Injection -Most OpenTelemetry logging instrumentation can automatically inject the active trace context into your logs. Ensure this feature is enabled for your logging library. Your logs must be sent in JSON format for Datadog to parse the attributes correctly. +To inject the trace context in Java, you can use the OpenTelemetry Logback Appender. Add the `io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0` dependency to your project and configure it in your `logback.xml`: -Verify the Log Output -After configuration, your JSON logs should contain the trace_id and span_id attributes, which Datadog uses to link the log to the exact trace and span that was active when the log was generated. +```xml + + + true + + + + + + +``` + +For complete, working example configuration, see the [full Java example in the examples repository][200]. + +[200]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/java/calendar/src/main/resources/logback.xml + +{{% /tab %}} +{{< /tabs >}} + +#### Verify the log output + +After configuration, your JSON logs should contain `trace_id` and `span_id` attributes. Datadog uses these to link the log to the exact trace and span that was active when the log was generated. Here is an example of a properly formatted JSON log entry: +```json { "timestamp": 1720557413000, "level": "INFO", @@ -145,46 +199,42 @@ Here is an example of a properly formatted JSON log entry: "trace_id": "8738839999436033394", "span_id": "1006654203791334032" } +``` + +
For complete, working examples of trace and log correlation for various languages, see the Datadog OpenTelemetry Examples repository.
+ +## Correlating with Datadog products -{{< alert "info" >}} -For complete, working examples of trace and log correlation for various languages, see the Datadog OpenTelemetry Examples repository. -{{< /alert >}} +### Real User Monitoring (RUM) -Correlating with Datadog Products -Real User Monitoring (RUM) -Why it matters: By connecting RUM and APM, you can follow a user's journey from a slow button click on your website all the way to the specific backend database query that caused the delay. +Correlating RUM and APM allows you to follow a user's journey from a slow button click on your website all the way to the specific backend database query that caused the delay. -This is achieved by configuring the RUM SDK to inject trace context headers into requests made to your backend. The default injection style is tracecontext, Datadog. +This correlation is achieved by configuring the RUM SDK to inject trace context headers into requests made to your backend. The default injection style includes `tracecontext` and `datadog` headers. {{< tabs >}} {{% tab "Browser" %}} -Ensure you have completed the standard Browser RUM setup. - -Modify allowedTracingUrls in your RUM SDK initialization to specify which propagator to use for which backend endpoint. +After completing the standard Browser RUM setup, modify `allowedTracingUrls` in your RUM SDK initialization to specify which propagator to use for each backend endpoint. +```javascript import { datadogRum } from '@datadog/browser-rum' datadogRum.init({ // ...otherConfig, allowedTracingUrls: [ - { match: "[https://api.example.com](https://api.example.com)", propagatorTypes: ["tracecontext"]} + { match: "https://api.example.com", propagatorTypes: ["tracecontext"] } ] }) -```propagatorTypes` accepts a list of desired propagators: - - `datadog`: Datadog's propagator (`x-datadog-*`) - - `tracecontext`: [W3C Trace Context](https://www.w3.org/TR/trace-context/) (`traceparent`, `tracestate`) - - `b3`: [B3 single header](https://github.com/openzipkin/b3-propagation#single-header) (`b3`) - - `b3multi`: [B3 multiple headers](https://github.com/openzipkin/b3-propagation#multiple-headers) (`X-B3-*`) +``` +`propagatorTypes` accepts a list of desired propagators: `datadog`, `tracecontext`, `b3`, and `b3multi`. {{% /tab %}} {{% tab "iOS" %}} -Ensure you have completed the standard iOS RUM setup. - -Use .traceWithHeaders(hostsWithHeaders:sampleRate:) to specify which tracing headers to inject for requests to your backend hosts. +After completing the standard [iOS RUM setup][200], use `.traceWithHeaders(hostsWithHeaders:sampleRate:)` to specify which tracing headers to inject for requests to your backend hosts. +```swift RUM.enable( with: RUM.Configuration( applicationID: "", @@ -198,126 +248,108 @@ RUM.enable( ) ) ) -```TracingHeaderType` is an enum that includes `.datadog`, `.tracecontext`, `.b3`, and `.b3multi`. +``` + +`TracingHeaderType` is an enum that includes `.datadog`, `.tracecontext`, `.b3`, and `.b3multi`. +[200]: /real_user_monitoring/ios/ {{% /tab %}} {{% tab "Android" %}} -Ensure you have completed the standard Android RUM setup. +After completing the standard [Android RUM setup][300], configure your `OkHttpClient` interceptor with a map of your backend hosts and the desired `TracingHeaderType` for each. -Configure your OkHttpClient interceptor with a map of your backend hosts and the desired TracingHeaderType for each. - -val tracedHosts = mapOf("example.com" to setOf(TracingHeaderType.TRACECONTEXT), - "example.eu" to setOf(TracingHeaderType.DATADOG)) +```kotlin +val tracedHosts = mapOf( + "example.com" to setOf(TracingHeaderType.TRACECONTEXT), + "example.eu" to setOf(TracingHeaderType.DATADOG) +) val okHttpClient = OkHttpClient.Builder() .addInterceptor(DatadogInterceptor.Builder(tracedHosts).build()) .addNetworkInterceptor(TracingInterceptor.Builder(tracedHosts).build()) .eventListenerFactory(DatadogEventListener.Factory()) .build() -```TracingHeaderType` is an enum that includes `.DATADOG`, `.TRACECONTEXT`, `.B3`, and `.B3MULTI`. +``` +`TracingHeaderType` is an enum that includes `.DATADOG`, `.TRACECONTEXT`, `.B3`, and `.B3MULTI`. + +[300]: /real_user_monitoring/android/ {{% /tab %}} {{< /tabs >}} -For React Native, Flutter, and Kotlin Multiplatform, refer to the platform-specific RUM documentation for configuring firstPartyHosts, firstPartyHostsWithTracingHeaders, or the datadogKtorPlugin respectively. - -Database Monitoring (DBM) -Why it matters: Connect backend traces to detailed database performance data in Datadog, including query metrics and execution plans, to identify the exact queries that are slowing down your application. - -Step 1: Instrument Your Database Spans -For DBM correlation to work, your database spans must include the following attributes. - -Attribute +For **React Native**, **Flutter**, and **Kotlin Multiplatform**, see the platform-specific RUM documentation for configuring `firstPartyHosts`, `firstPartyHostsWithTracingHeaders`, or the `datadogKtorPlugin` respectively: -Description +- [React Native RUM Documentation][3] +- [Flutter RUM Documentation][4] +- [Kotlin Multiplatform RUM Documentation][5] -Example +### Database Monitoring (DBM) -span.type +Correlate backend traces to detailed database performance data in Datadog, including query metrics and execution plans, to identify the exact queries that are slowing down your application. -Required. The type of span. Must be sql, postgres, mysql, or sql.query. +#### Step 1: Instrument your database spans -sql - -span.service - -Required. The name of the service executing the query. This should match your foundational service tag. - -my-web-app - -span.resource - -Required. The raw SQL query text. This is critical for Datadog's obfuscation and normalization. - -SELECT * FROM users WHERE id = ? +For DBM correlation to work, your database spans must include the following attributes. -db.name +| Attribute | Description | Example | +|----------------|---------------------------------------------------------------------------------------|------------------------------------| +| `db.system` | **Required.** The database technology, such as `postgres`, `mysql`, or `sqlserver`. | `postgres` | +| `db.statement` | **Required.** The raw SQL query text. This is used for obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` | +| `db.name` | The logical database or schema name being queried. | `user_accounts` | +| `span.type` | **Required (Datadog-specific).** The type of span. Must be `sql`. | `sql` | -The logical database or schema name being queried. +#### Step 2: Configure your ingest path -user_accounts +Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. -db.system +{{< tabs >}} +{{% tab "Datadog Agent (DDOT Collector)" %}} -The database technology. -postgres +If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`: -Step 2: Configure Your Ingest Path -Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. +```yaml +datadog: + otelCollector: + featureGates: datadog.EnableOperationAndResourceNameV2 +``` -{{< tabs >}} +{{% /tab %}} {{% tab "OTel Collector" %}} -OpenTelemetry Collector with Datadog Exporter - -When starting the Collector, enable the datadog.EnableOperationAndResourceNameV2 feature gate. This gate is available in v0.118.0 and later. +When starting the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate. This is available in Collector v0.118.0 and later. +```bash otelcontribcol --config=config.yaml \ --feature-gates=datadog.EnableOperationAndResourceNameV2 +``` {{% /tab %}} -{{% tab "Datadog Agent (OTLP Ingest)" %}} -Datadog Agent with OTLP Ingest +{{% tab "Datadog Agent (OTLP Ingest)" %}} -In your Datadog Agent configuration, ensure the DD_APM_FEATURES environment variable includes enable_operation_and_resource_name_logic_v2. For more information, see the Agent Configuration Documentation. +In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`. {{% /tab %}} -{{% tab "Datadog Agent (Embedded OTel)" %}} - -Datadog Agent with embedded OTel Collector - -If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gates in your values.yaml. For more information, see the Datadog Helm Chart Documentation. -datadog: - otelCollector: - featureGates: datadog.EnableOperationAndResourceNameV2 - -{{% /tab %}} {{< /tabs >}} -APM and Infrastructure -Why it matters: Connecting your APM traces to Kubernetes metrics allows you to view pod and container performance directly in the Infrastructure tab of the APM Service Page. +## What's next? -This correlation requires configuring specific receivers in your OpenTelemetry Collector to gather Kubernetes metrics. - -{{< alert "info" >}} -This feature is in active development. Detailed documentation on configuring the required Kubernetes receivers is forthcoming. -{{< /alert >}} - -What's Next? Now that your telemetry is correlated, you can explore the unified views in Datadog: -Service Catalog: View the health of all your services in one place. - -Trace Explorer: Search and analyze individual requests as they travel across your services. - -Log Explorer: Search and filter all your logs, and pivot to related traces with a single click. +- **Service Catalog**: View the health of all your services in one place. +- **Trace Explorer**: Search and analyze individual requests as they travel across your services. +- **Log Explorer**: Search and filter all your logs, and pivot to related traces. ## Further reading -{{< partial name="whats-next/whats-next.html" >}} \ No newline at end of file +{{< partial name="whats-next/whats-next.html" >}} + +[1]: https://github.com/DataDog/opentelemetry-examples +[2]: /opentelemetry/integrations/host_metrics/ +[3]: /real_user_monitoring/reactnative/ +[4]: /real_user_monitoring/mobile_and_tv_monitoring/flutter/ +[5]: /real_user_monitoring/mobile_and_tv_monitoring/kotlin_multiplatform/ \ No newline at end of file From 3ee6847a475d059cd8924b839b0df6d1e0d6907a Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 8 Jul 2025 18:43:42 -0600 Subject: [PATCH 03/24] Tweak examples. --- content/en/opentelemetry/correlate/_index.md | 26 ++++++++++++-------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/content/en/opentelemetry/correlate/_index.md b/content/en/opentelemetry/correlate/_index.md index bcdd79eca94ae..638d3534dca18 100644 --- a/content/en/opentelemetry/correlate/_index.md +++ b/content/en/opentelemetry/correlate/_index.md @@ -33,8 +33,9 @@ You can set these attributes in your application's environment variables, SDK, o Set the `OTEL_RESOURCE_ATTRIBUTES` environment variable with your service's information: -```bash -export OTEL_RESOURCE_ATTRIBUTES="service.name=my-service,deployment.environment.name=production,service.version=1.2.3" +```sh +export OTEL_SERVICE_NAME="my-service" +export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.version=1.2.3" ``` {{% /tab %}} @@ -59,17 +60,22 @@ tracer_provider = TracerProvider(resource=resource) {{% /tab %}} {{% tab "Collector" %}} -Use the transform processor in your Collector configuration to set the resource attributes on your telemetry data: +Use the `resource` processor in your Collector configuration to set the resource attributes on your telemetry data: ```yaml processors: - transform: - trace_statements: - - context: resource - statements: - - set(attributes["service.name"], "my-service") - - set(attributes["deployment.environment.name"], "production") - - set(attributes["service.version"], "1.2.3") + resource: + attributes: + - key: service.name + value: "my-service" + action: upsert + - key: deployment.environment.name + value: "production" + action: upsert + - key: service.version + value: "1.2.3" + action: upsert +... ``` {{% /tab %}} From 409da0ecbf61d2d7e84e6eccaecd9199b101dee4 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 8 Jul 2025 18:49:06 -0600 Subject: [PATCH 04/24] Remove vague statement. --- content/en/opentelemetry/correlate/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/opentelemetry/correlate/_index.md b/content/en/opentelemetry/correlate/_index.md index 638d3534dca18..d986a3deea190 100644 --- a/content/en/opentelemetry/correlate/_index.md +++ b/content/en/opentelemetry/correlate/_index.md @@ -124,7 +124,7 @@ For detailed setup instructions, see [Host Metrics][2] documentation. ##### Ensure consistent host and container tagging -For correlation to work, the `host.name` (or `container.id`) attribute on your traces must match the corresponding attribute on the metrics collected by the `hostmetrics` receiver. When running the OpenTelemetry Collector as an agent on the host, it typically discovers this information automatically. +For correlation to work, the `host.name` (or `container.id`) attribute on your traces must match the corresponding attribute on the metrics collected by the `hostmetrics` receiver. ### Traces and logs From 69dc93ec94e8f7d197a6849ab0350c8d8dd6d78c Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Mon, 14 Jul 2025 11:13:41 -0600 Subject: [PATCH 05/24] Create new Data Correlation parent page with UST. --- .../en/opentelemetry/correlate/_index copy.md | 361 ++++++++++++++++++ content/en/opentelemetry/correlate/_index.md | 288 +------------- 2 files changed, 375 insertions(+), 274 deletions(-) create mode 100644 content/en/opentelemetry/correlate/_index copy.md diff --git a/content/en/opentelemetry/correlate/_index copy.md b/content/en/opentelemetry/correlate/_index copy.md new file mode 100644 index 0000000000000..d986a3deea190 --- /dev/null +++ b/content/en/opentelemetry/correlate/_index copy.md @@ -0,0 +1,361 @@ +--- +title: Correlate OpenTelemetry Data +description: Learn how to correlate your OpenTelemetry traces, metrics, logs, and other telemetry in Datadog to get a unified view of your application's performance. +aliases: + - /opentelemetry/otel_logs/ +further_reading: +- link: "https://www.datadoghq.com/blog/opentelemetry-instrumentation/" + tag: "Blog" + text: "Datadog's partnership with OpenTelemetry" +--- + +## Overview + +Getting a unified view of your application's performance requires connecting its traces, metrics, logs, user interactions, and more. This guide provides the steps to correlate your OpenTelemetry data in Datadog, allowing you to navigate between all related telemetry in a single view. + +## Prerequisite: Unified service tagging + +Datadog uses three standard tags to link telemetry together: `env`, `service`, and `version`. To apply these tags, you must configure the corresponding resource attributes in OpenTelemetry. + +When using OpenTelemetry, you set these tags by defining a standard set of resource attributes. Datadog automatically maps these attributes to the correct tags. + +| OpenTelemetry Resource Attribute | Datadog Tag | Notes | +|----------------------------------|-------------|---------------------------------------------------------------------------------------------------------| +| `deployment.environment.name` | `env` | **Recommended**. Supported in Agent v7.58.0+ and Collector Exporter v0.110.0+. | +| `deployment.environment` | `env` | Use if you are running an Agent version older than v7.58.0 or a Collector Exporter older than v0.110.0. | +| `service.name` | `service` | | +| `service.version` | `version` | | + +You can set these attributes in your application's environment variables, SDK, or in the OpenTelemetry Collector. + +{{< tabs >}} +{{% tab "Environment Variables" %}} + +Set the `OTEL_RESOURCE_ATTRIBUTES` environment variable with your service's information: + +```sh +export OTEL_SERVICE_NAME="my-service" +export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.version=1.2.3" +``` + +{{% /tab %}} +{{% tab "SDK" %}} + +Create a Resource with the required attributes and associate it with your TracerProvider in your application code. + +Here's an example using the OpenTelemetry SDK for Python: + +```python +from opentelemetry.sdk.resources import Resource +from opentelemetry.sdk.trace import TracerProvider + +resource = Resource(attributes={ + "service.name": "", + "deployment.environment.name": "", + "service.version": "" +}) +tracer_provider = TracerProvider(resource=resource) +``` + +{{% /tab %}} +{{% tab "Collector" %}} + +Use the `resource` processor in your Collector configuration to set the resource attributes on your telemetry data: + +```yaml +processors: + resource: + attributes: + - key: service.name + value: "my-service" + action: upsert + - key: deployment.environment.name + value: "production" + action: upsert + - key: service.version + value: "1.2.3" + action: upsert +... +``` + +{{% /tab %}} +{{< /tabs >}} + +## Correlating telemetry + +### Traces and metrics + +Correlating traces with infrastructure metrics allows you to pivot from a slow request directly to the CPU and memory metrics of the host or container it ran on. This helps you determine if resource contention was the root cause of a performance issue. + +Correlation between traces and infrastructure metrics relies on the following resource attributes: + +- `host.name`: For correlating with host metrics (CPU, memory, disk). +- `container.id`: For correlating with container metrics. + +#### Setup + +##### Enable the host metrics receiver + +To collect system-level metrics, enable the `hostmetrics` receiver in your OpenTelemetry Collector configuration. This receiver gathers metrics like CPU, memory, disk, and network usage. + +```yaml +receivers: + hostmetrics: + collection_interval: 30s + scrapers: + cpu: + memory: + +processors: + batch: + +exporters: + datadog: {} + +service: + pipelines: + metrics: + receivers: [hostmetrics] + processors: [batch] + exporters: [datadog] +``` + +For detailed setup instructions, see [Host Metrics][2] documentation. + +##### Ensure consistent host and container tagging + +For correlation to work, the `host.name` (or `container.id`) attribute on your traces must match the corresponding attribute on the metrics collected by the `hostmetrics` receiver. + +### Traces and logs + +Correlating traces with logs allows you to jump from a specific span in a trace to the exact logs that were generated during that operation, making debugging faster and more intuitive. + +The only requirement for connecting traces and logs is to inject the active `trace_id` and `span_id` from your application's trace context into your logs. + +#### Setup + +Most OpenTelemetry logging instrumentation can automatically inject the active trace context into your logs. Ensure this feature is enabled for your logging library. For Datadog to parse the attributes correctly, your logs must be sent in JSON format. + +For complete example applications, see the [Datadog OpenTelemetry Examples repository][1]. + +{{< tabs >}} +{{% tab "Go" %}} + +For Go applications using a structured logger such as `zap`, the recommended approach is to use a bridge such as **`otelzap`**. This wraps your logger and automatically injects the active `trace_id` and `span_id` into every log message. + +First, ensure you have an initialized OpenTelemetry `LoggerProvider`. Then, use it to create your `zap` logger instance: + +```go +import "go.opentelemetry.io/contrib/bridges/otelzap" + +// Replace your standard zap logger with the otelzap-backed one +logger := zap.New(otelzap.NewCore( + "my-service-name", + otelzap.WithLoggerProvider(loggerProvider), +)) + +// Now, logs written with this logger are automatically correlated +logger.Info("Processing user request") +``` + +To see how the `LoggerProvider` is configured in a complete application, see the [full Go example in the examples repository][100]. + +[100]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/golang/calendar/main.go + +{{% /tab %}} + +{{% tab "Java" %}} + +To inject the trace context in Java, you can use the OpenTelemetry Logback Appender. Add the `io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0` dependency to your project and configure it in your `logback.xml`: + +```xml + + + true + + + + + + +``` + +For complete, working example configuration, see the [full Java example in the examples repository][200]. + +[200]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/java/calendar/src/main/resources/logback.xml + +{{% /tab %}} +{{< /tabs >}} + +#### Verify the log output + +After configuration, your JSON logs should contain `trace_id` and `span_id` attributes. Datadog uses these to link the log to the exact trace and span that was active when the log was generated. + +Here is an example of a properly formatted JSON log entry: + +```json +{ + "timestamp": 1720557413000, + "level": "INFO", + "message": "Processing user request", + "service": "my-service", + "env": "production", + "version": "1.2.3", + "trace_id": "8738839999436033394", + "span_id": "1006654203791334032" +} +``` + +
For complete, working examples of trace and log correlation for various languages, see the Datadog OpenTelemetry Examples repository.
+ +## Correlating with Datadog products + +### Real User Monitoring (RUM) + +Correlating RUM and APM allows you to follow a user's journey from a slow button click on your website all the way to the specific backend database query that caused the delay. + +This correlation is achieved by configuring the RUM SDK to inject trace context headers into requests made to your backend. The default injection style includes `tracecontext` and `datadog` headers. + +{{< tabs >}} +{{% tab "Browser" %}} + +After completing the standard Browser RUM setup, modify `allowedTracingUrls` in your RUM SDK initialization to specify which propagator to use for each backend endpoint. + +```javascript +import { datadogRum } from '@datadog/browser-rum' + +datadogRum.init({ + // ...otherConfig, + allowedTracingUrls: [ + { match: "https://api.example.com", propagatorTypes: ["tracecontext"] } + ] +}) +``` + +`propagatorTypes` accepts a list of desired propagators: `datadog`, `tracecontext`, `b3`, and `b3multi`. + +{{% /tab %}} +{{% tab "iOS" %}} + +After completing the standard [iOS RUM setup][200], use `.traceWithHeaders(hostsWithHeaders:sampleRate:)` to specify which tracing headers to inject for requests to your backend hosts. + +```swift +RUM.enable( + with: RUM.Configuration( + applicationID: "", + urlSessionTracking: .init( + firstPartyHostsTracing: .traceWithHeaders( + hostsWithHeaders: [ + "api.example.com": [.tracecontext] + ], + sampleRate: 100 + ) + ) + ) +) +``` + +`TracingHeaderType` is an enum that includes `.datadog`, `.tracecontext`, `.b3`, and `.b3multi`. + +[200]: /real_user_monitoring/ios/ + +{{% /tab %}} +{{% tab "Android" %}} + +After completing the standard [Android RUM setup][300], configure your `OkHttpClient` interceptor with a map of your backend hosts and the desired `TracingHeaderType` for each. + +```kotlin +val tracedHosts = mapOf( + "example.com" to setOf(TracingHeaderType.TRACECONTEXT), + "example.eu" to setOf(TracingHeaderType.DATADOG) +) + +val okHttpClient = OkHttpClient.Builder() + .addInterceptor(DatadogInterceptor.Builder(tracedHosts).build()) + .addNetworkInterceptor(TracingInterceptor.Builder(tracedHosts).build()) + .eventListenerFactory(DatadogEventListener.Factory()) + .build() +``` + +`TracingHeaderType` is an enum that includes `.DATADOG`, `.TRACECONTEXT`, `.B3`, and `.B3MULTI`. + +[300]: /real_user_monitoring/android/ + +{{% /tab %}} +{{< /tabs >}} + +For **React Native**, **Flutter**, and **Kotlin Multiplatform**, see the platform-specific RUM documentation for configuring `firstPartyHosts`, `firstPartyHostsWithTracingHeaders`, or the `datadogKtorPlugin` respectively: + +- [React Native RUM Documentation][3] +- [Flutter RUM Documentation][4] +- [Kotlin Multiplatform RUM Documentation][5] + +### Database Monitoring (DBM) + +Correlate backend traces to detailed database performance data in Datadog, including query metrics and execution plans, to identify the exact queries that are slowing down your application. + +#### Step 1: Instrument your database spans + +For DBM correlation to work, your database spans must include the following attributes. + +| Attribute | Description | Example | +|----------------|---------------------------------------------------------------------------------------|------------------------------------| +| `db.system` | **Required.** The database technology, such as `postgres`, `mysql`, or `sqlserver`. | `postgres` | +| `db.statement` | **Required.** The raw SQL query text. This is used for obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` | +| `db.name` | The logical database or schema name being queried. | `user_accounts` | +| `span.type` | **Required (Datadog-specific).** The type of span. Must be `sql`. | `sql` | + +#### Step 2: Configure your ingest path + +Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. + +{{< tabs >}} +{{% tab "Datadog Agent (DDOT Collector)" %}} + + +If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`: + +```yaml +datadog: + otelCollector: + featureGates: datadog.EnableOperationAndResourceNameV2 +``` + +{{% /tab %}} +{{% tab "OTel Collector" %}} + +When starting the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate. This is available in Collector v0.118.0 and later. + +```bash +otelcontribcol --config=config.yaml \ +--feature-gates=datadog.EnableOperationAndResourceNameV2 +``` + +{{% /tab %}} + +{{% tab "Datadog Agent (OTLP Ingest)" %}} + +In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`. + +{{% /tab %}} + +{{< /tabs >}} + +## What's next? + +Now that your telemetry is correlated, you can explore the unified views in Datadog: + +- **Service Catalog**: View the health of all your services in one place. +- **Trace Explorer**: Search and analyze individual requests as they travel across your services. +- **Log Explorer**: Search and filter all your logs, and pivot to related traces. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: https://github.com/DataDog/opentelemetry-examples +[2]: /opentelemetry/integrations/host_metrics/ +[3]: /real_user_monitoring/reactnative/ +[4]: /real_user_monitoring/mobile_and_tv_monitoring/flutter/ +[5]: /real_user_monitoring/mobile_and_tv_monitoring/kotlin_multiplatform/ \ No newline at end of file diff --git a/content/en/opentelemetry/correlate/_index.md b/content/en/opentelemetry/correlate/_index.md index d986a3deea190..966ea87bc7458 100644 --- a/content/en/opentelemetry/correlate/_index.md +++ b/content/en/opentelemetry/correlate/_index.md @@ -11,13 +11,14 @@ further_reading: ## Overview -Getting a unified view of your application's performance requires connecting its traces, metrics, logs, user interactions, and more. This guide provides the steps to correlate your OpenTelemetry data in Datadog, allowing you to navigate between all related telemetry in a single view. +Getting a unified view of your application's performance requires connecting its traces, metrics, logs, user interactions, and more. By correlating your OpenTelemetry data in Datadog, you can navigate between all related telemetry in a single view, allowing you to diagnose and resolve issues faster. + ## Prerequisite: Unified service tagging -Datadog uses three standard tags to link telemetry together: `env`, `service`, and `version`. To apply these tags, you must configure the corresponding resource attributes in OpenTelemetry. +Datadog uses three standard tags to link telemetry together: `env`, `service`, and `version`. -When using OpenTelemetry, you set these tags by defining a standard set of resource attributes. Datadog automatically maps these attributes to the correct tags. +To ensure your OpenTelemetry data is properly correlated, you must configure your application or system to use these tags by setting a standard set of OpenTelemetry resource attributes. Datadog automatically maps these attributes to the correct tags. | OpenTelemetry Resource Attribute | Datadog Tag | Notes | |----------------------------------|-------------|---------------------------------------------------------------------------------------------------------| @@ -81,281 +82,20 @@ processors: {{% /tab %}} {{< /tabs >}} -## Correlating telemetry - -### Traces and metrics - -Correlating traces with infrastructure metrics allows you to pivot from a slow request directly to the CPU and memory metrics of the host or container it ran on. This helps you determine if resource contention was the root cause of a performance issue. - -Correlation between traces and infrastructure metrics relies on the following resource attributes: - -- `host.name`: For correlating with host metrics (CPU, memory, disk). -- `container.id`: For correlating with container metrics. - -#### Setup - -##### Enable the host metrics receiver - -To collect system-level metrics, enable the `hostmetrics` receiver in your OpenTelemetry Collector configuration. This receiver gathers metrics like CPU, memory, disk, and network usage. - -```yaml -receivers: - hostmetrics: - collection_interval: 30s - scrapers: - cpu: - memory: - -processors: - batch: - -exporters: - datadog: {} - -service: - pipelines: - metrics: - receivers: [hostmetrics] - processors: [batch] - exporters: [datadog] -``` - -For detailed setup instructions, see [Host Metrics][2] documentation. - -##### Ensure consistent host and container tagging - -For correlation to work, the `host.name` (or `container.id`) attribute on your traces must match the corresponding attribute on the metrics collected by the `hostmetrics` receiver. - -### Traces and logs - -Correlating traces with logs allows you to jump from a specific span in a trace to the exact logs that were generated during that operation, making debugging faster and more intuitive. - -The only requirement for connecting traces and logs is to inject the active `trace_id` and `span_id` from your application's trace context into your logs. - -#### Setup - -Most OpenTelemetry logging instrumentation can automatically inject the active trace context into your logs. Ensure this feature is enabled for your logging library. For Datadog to parse the attributes correctly, your logs must be sent in JSON format. - -For complete example applications, see the [Datadog OpenTelemetry Examples repository][1]. - -{{< tabs >}} -{{% tab "Go" %}} - -For Go applications using a structured logger such as `zap`, the recommended approach is to use a bridge such as **`otelzap`**. This wraps your logger and automatically injects the active `trace_id` and `span_id` into every log message. - -First, ensure you have an initialized OpenTelemetry `LoggerProvider`. Then, use it to create your `zap` logger instance: - -```go -import "go.opentelemetry.io/contrib/bridges/otelzap" - -// Replace your standard zap logger with the otelzap-backed one -logger := zap.New(otelzap.NewCore( - "my-service-name", - otelzap.WithLoggerProvider(loggerProvider), -)) - -// Now, logs written with this logger are automatically correlated -logger.Info("Processing user request") -``` - -To see how the `LoggerProvider` is configured in a complete application, see the [full Go example in the examples repository][100]. - -[100]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/golang/calendar/main.go - -{{% /tab %}} - -{{% tab "Java" %}} - -To inject the trace context in Java, you can use the OpenTelemetry Logback Appender. Add the `io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0` dependency to your project and configure it in your `logback.xml`: - -```xml - - - true - - - - - - -``` - -For complete, working example configuration, see the [full Java example in the examples repository][200]. - -[200]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/java/calendar/src/main/resources/logback.xml - -{{% /tab %}} -{{< /tabs >}} - -#### Verify the log output - -After configuration, your JSON logs should contain `trace_id` and `span_id` attributes. Datadog uses these to link the log to the exact trace and span that was active when the log was generated. - -Here is an example of a properly formatted JSON log entry: - -```json -{ - "timestamp": 1720557413000, - "level": "INFO", - "message": "Processing user request", - "service": "my-service", - "env": "production", - "version": "1.2.3", - "trace_id": "8738839999436033394", - "span_id": "1006654203791334032" -} -``` - -
For complete, working examples of trace and log correlation for various languages, see the Datadog OpenTelemetry Examples repository.
- -## Correlating with Datadog products - -### Real User Monitoring (RUM) - -Correlating RUM and APM allows you to follow a user's journey from a slow button click on your website all the way to the specific backend database query that caused the delay. - -This correlation is achieved by configuring the RUM SDK to inject trace context headers into requests made to your backend. The default injection style includes `tracecontext` and `datadog` headers. - -{{< tabs >}} -{{% tab "Browser" %}} - -After completing the standard Browser RUM setup, modify `allowedTracingUrls` in your RUM SDK initialization to specify which propagator to use for each backend endpoint. - -```javascript -import { datadogRum } from '@datadog/browser-rum' - -datadogRum.init({ - // ...otherConfig, - allowedTracingUrls: [ - { match: "https://api.example.com", propagatorTypes: ["tracecontext"] } - ] -}) -``` - -`propagatorTypes` accepts a list of desired propagators: `datadog`, `tracecontext`, `b3`, and `b3multi`. - -{{% /tab %}} -{{% tab "iOS" %}} - -After completing the standard [iOS RUM setup][200], use `.traceWithHeaders(hostsWithHeaders:sampleRate:)` to specify which tracing headers to inject for requests to your backend hosts. - -```swift -RUM.enable( - with: RUM.Configuration( - applicationID: "", - urlSessionTracking: .init( - firstPartyHostsTracing: .traceWithHeaders( - hostsWithHeaders: [ - "api.example.com": [.tracecontext] - ], - sampleRate: 100 - ) - ) - ) -) -``` - -`TracingHeaderType` is an enum that includes `.datadog`, `.tracecontext`, `.b3`, and `.b3multi`. - -[200]: /real_user_monitoring/ios/ - -{{% /tab %}} -{{% tab "Android" %}} - -After completing the standard [Android RUM setup][300], configure your `OkHttpClient` interceptor with a map of your backend hosts and the desired `TracingHeaderType` for each. - -```kotlin -val tracedHosts = mapOf( - "example.com" to setOf(TracingHeaderType.TRACECONTEXT), - "example.eu" to setOf(TracingHeaderType.DATADOG) -) - -val okHttpClient = OkHttpClient.Builder() - .addInterceptor(DatadogInterceptor.Builder(tracedHosts).build()) - .addNetworkInterceptor(TracingInterceptor.Builder(tracedHosts).build()) - .eventListenerFactory(DatadogEventListener.Factory()) - .build() -``` - -`TracingHeaderType` is an enum that includes `.DATADOG`, `.TRACECONTEXT`, `.B3`, and `.B3MULTI`. - -[300]: /real_user_monitoring/android/ - -{{% /tab %}} -{{< /tabs >}} - -For **React Native**, **Flutter**, and **Kotlin Multiplatform**, see the platform-specific RUM documentation for configuring `firstPartyHosts`, `firstPartyHostsWithTracingHeaders`, or the `datadogKtorPlugin` respectively: - -- [React Native RUM Documentation][3] -- [Flutter RUM Documentation][4] -- [Kotlin Multiplatform RUM Documentation][5] - -### Database Monitoring (DBM) - -Correlate backend traces to detailed database performance data in Datadog, including query metrics and execution plans, to identify the exact queries that are slowing down your application. - -#### Step 1: Instrument your database spans - -For DBM correlation to work, your database spans must include the following attributes. - -| Attribute | Description | Example | -|----------------|---------------------------------------------------------------------------------------|------------------------------------| -| `db.system` | **Required.** The database technology, such as `postgres`, `mysql`, or `sqlserver`. | `postgres` | -| `db.statement` | **Required.** The raw SQL query text. This is used for obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` | -| `db.name` | The logical database or schema name being queried. | `user_accounts` | -| `span.type` | **Required (Datadog-specific).** The type of span. Must be `sql`. | `sql` | - -#### Step 2: Configure your ingest path - -Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. - -{{< tabs >}} -{{% tab "Datadog Agent (DDOT Collector)" %}} - - -If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`: - -```yaml -datadog: - otelCollector: - featureGates: datadog.EnableOperationAndResourceNameV2 -``` - -{{% /tab %}} -{{% tab "OTel Collector" %}} - -When starting the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate. This is available in Collector v0.118.0 and later. - -```bash -otelcontribcol --config=config.yaml \ ---feature-gates=datadog.EnableOperationAndResourceNameV2 -``` - -{{% /tab %}} - -{{% tab "Datadog Agent (OTLP Ingest)" %}} - -In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`. - -{{% /tab %}} - -{{< /tabs >}} - -## What's next? +## Correlate telemetry -Now that your telemetry is correlated, you can explore the unified views in Datadog: +After unified service tagging is configured, you can connect your various telemetry streams. Select a guide below for platform-specific instructions. -- **Service Catalog**: View the health of all your services in one place. -- **Trace Explorer**: Search and analyze individual requests as they travel across your services. -- **Log Explorer**: Search and filter all your logs, and pivot to related traces. +- [Correlate logs and traces][1] +- [Correlate metrics and traces][2] +- [Correlate RUM and traces][3] +- [Correlate DBM and traces][4] ## Further reading {{< partial name="whats-next/whats-next.html" >}} -[1]: https://github.com/DataDog/opentelemetry-examples -[2]: /opentelemetry/integrations/host_metrics/ -[3]: /real_user_monitoring/reactnative/ -[4]: /real_user_monitoring/mobile_and_tv_monitoring/flutter/ -[5]: /real_user_monitoring/mobile_and_tv_monitoring/kotlin_multiplatform/ \ No newline at end of file +[1]: /opentelemetry/correlate/logs_and_traces +[1]: /opentelemetry/correlate/metrics_and_traces +[1]: /opentelemetry/correlate/rum_and_traces +[1]: /opentelemetry/correlate/dbm_and_traces \ No newline at end of file From 9dabb1a6a4f2d4a0b7cc248f884945aa48a316d7 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Mon, 14 Jul 2025 12:05:07 -0600 Subject: [PATCH 06/24] Update correlate logs and traces page. --- .../connect_logs_and_traces/opentelemetry.md | 333 +++--------------- 1 file changed, 57 insertions(+), 276 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index da0868d97d46d..7868ef2d88112 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -21,321 +21,102 @@ further_reading: text: 'Ease troubleshooting with cross product correlation.' --- -Connecting OpenTelemetry language SDK logs and traces within Datadog is similar to connecting [Datadog SDK logs and traces][1], with a few additional steps: +## Overview -1. Datadog supports the OpenTelemetry standard of 128-bit Trace IDs by default, but the `SpanId` formats differ between vendors. Therefore, it's necessary to translate the `SpanId` from its OpenTelemetry format ([a 16-hex-character lowercase string][2]) into its Datadog format ([a 64bit unsigned int][3]). +Correlating traces and logs allows you to investigate issues by navigating from a specific span in a trace directly to the logs that were generated during that operation. This makes debugging faster and more intuitive by providing the exact context needed to understand an error or performance problem. -2. Ensure your logs are sent as JSON, because your language level logs must be turned into Datadog attributes for trace-log correlation to work. +## Requirements -See the following examples for language-specific information about how to correlate your OpenTelemetry traces and logs. +Before you begin, ensure you have configured [unified service tagging][1]. This is required for all data correlation in Datadog. -{{< tabs >}} -{{% tab "Python" %}} - -To manually correlate your traces with your logs, patch the logging module you are using with a processor that translates the OpenTelemetry formatted `span_id` into Datadog format. The following example uses the [structlog logging library][1]. For other logging libraries, it may be more appropriate to [modify the Datadog SDK examples][2]. You can also find [an example OpenTelemetry instrumented Python application with trace and log correlation][3] in the `trace-examples` GitHub repository. - -```python -# ########## injection.py -from opentelemetry import trace - -class CustomDatadogLogProcessor(object): - def __call__(self, logger, method_name, event_dict): - # An example of adding datadog formatted trace context to logs - # from: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/b53b9a012f76c4fc883c3c245fddc29142706d0d/exporter/opentelemetry-exporter-datadog/src/opentelemetry/exporter/datadog/propagator.py#L127-L129 - current_span = trace.get_current_span() - if not current_span.is_recording(): - return event_dict - - context = current_span.get_span_context() if current_span is not None else None - if context is not None: - # if trace_id > 2**64 store as hex otherwise store the id as an integer - event_dict["dd.trace_id"] = str(context.trace_id) if context.trace_id < 2**64 else f"{context.trace_id:032x}" - event_dict["dd.span_id"] = str(context.span_id) - event_dict["dd.service"] = span.attributes.get("service.name") - event_dict["dd.env"] = span.attributes.get("deployment.environment") - event_dict["dd.version"] = span.attributes.get("service.version") - - return event_dict -# ########## - -# ########## app.py -import .injection -import logging -import structlog -# Add custom formatting to inject datadog formatted trace ids into logs -structlog.configure( - processors=[ - injection.CustomDatadogLogProcessor(), - structlog.processors.JSONRenderer(sort_keys=True) - ], -) - -log = structlog.getLogger() - -log.info("Example log line with trace correlation info") -``` - -#### Alternative approach - -You can also use unstructured logs with OpenTelemetry SDK logic to correlate logs and traces in your application. - -**Note**: This approach uses OpenTelemetry-native trace context fields (not `dd.trace_id` or `dd.span_id`), which can still be correlated in Datadog using remappers. See below for details. - -1. Set up OpenTelemetry logging instrumentation in your Python application: - - ```python - import logging - from opentelemetry.instrumentation.logging import LoggingInstrumentor - - # Initialize the OpenTelemetry logging instrumentation - LoggingInstrumentor().instrument(set_logging_format=True) - - # Create a logger instance - logger = logging.getLogger(__name__) - - # Log a message with automatic trace context injection - logger.info("This is a log message") - ``` - -2. Verify your log format contains the trace context information: - - Your logs should now include trace context information and look similar to: - ``` - 2025-03-25 10:31:52,116 INFO [__main__] [test-logging.py:9] [trace_id=0 span_id=0 resource.service.name= trace_sampled=False] - This is a log message - ``` - - Or in a real-world scenario: - ``` - 2025-03-20 12:45:10,123 INFO [jobs.scheduler.task_runner.execute] [task_runner.py:123] [trace_id=123abc456def789ghi012jkl345mno67 span_id=89ab01cd23ef45gh resource.service.name=job_scheduler trace_sampled=True] - STARTED JOB TASK. Success - ``` - -3. Create a Log Pipeline in Datadog with this Grok Parser Rule: - - ``` - # Define prefix components - _timestamp %{date("yyyy-MM-dd HH:mm:ss','SSS"):timestamp} - _level %{word:level} - _module \[%{notSpace:module}\] - _file_location \[%{notSpace:file_location}\] - _trace_info \[trace_id=%{notSpace:trace_id} span_id=%{notSpace:span_id} resource\.service\.name=%{notSpace:service_name} trace_sampled=% {notSpace:trace_sampled}\] - - # Complete rule - custom_format %{_timestamp} %{_level} %{_module} %{_file_location} %{_trace_info} - %{data:message} - ``` - -After setting up the Grok Parser Rule, add the `Trace Id Remapper` and `Span Id Remapper` processors to your pipeline to extract the `trace_id` and `span_id` values, respectively. This configuration allows your logs to appear properly correlated with traces in Datadog. - -[1]: https://www.structlog.org/en/stable/standard-library.html -[2]: /tracing/other_telemetry/connect_logs_and_traces/python/#manually-inject-trace-and-span-ids -[3]: https://github.com/DataDog/trace-examples/blob/98626d924f82666de60d6b2d6a65d87eebebdff1/opentelemetry/python-microservice/ddlogging/injection.py#L3 -{{% /tab %}} +## Setup -{{% tab "Node.js" %}} +To correlate OpenTelemetry traces and logs in Datadog, you must: -To manually correlate your traces with your logs, patch the logging module you are using with a processor that translates the OpenTelemetry formatted `span_id` into Datadog format. The following example uses the [winston logging library][1]. For other logging libraries, it may be more appropriate to [modify the Datadog SDK examples][2]. You can also find [an example OpenTelemetry instrumented Node.js application with trace and log correlation][3] in the `trace-examples` GitHub repository. +- **Inject Trace Context**: Your application's logger must be configured to inject the `trace_id` and `span_id` of the active trace into your logs. -```js -// ########## logger.js +- **Use JSON Formatting**: Your logs must be sent in JSON format so Datadog can automatically parse the trace context attributes. -// convert to dd with: -// https://github.com/DataDog/dd-trace-js/blob/master/packages/dd-trace/src/id.js -const opentelemetry = require('@opentelemetry/api'); -const winston = require('winston') +
For logs collected directly by the Datadog Agent (instead of through the OpenTelemetry Collector), automatic correlation relies on the dd.trace_id attribute.
-const tracingFormat = function () { - return winston.format(info => { - const span = opentelemetry.trace.getSpan(opentelemetry.context.active()); - if (span) { - const { spanId } = span.spanContext(); - info['dd.span_id'] = BigInt(`0x${spanId}`).toString(); - } - return info; - })(); -} +#### 1. Configure your logging library -module.exports = winston.createLogger({ - transports: [new winston.transports.Console], - format: winston.format.combine(tracingFormat(), winston.format.json()) -}); - -// ########## - -// ########## index.js -// -// ... -// initialize your tracer -// ... -// -const logger = require('./logger') -// -// use the logger in your application -logger.info("Example log line with trace correlation info") -``` +The recommended approach is to configure your logging library to automatically inject the standard OpenTelemetry `trace_id` and `span_id` attributes into your logs. +The following examples show how to configure logging instrumentation to automatically inject trace context. For complete, working applications, see the [Datadog OpenTelemetry Examples repository][2]. +{{< tabs >}} +{{% tab "Go" %}} -[1]: https://github.com/winstonjs/winston -[2]: /tracing/other_telemetry/connect_logs_and_traces/nodejs/#manually-inject-trace-and-span-ids -[3]: https://github.com/DataDog/trace-examples/blob/98626d924f82666de60d6b2d6a65d87eebebdff1/opentelemetry/node-microservice/logger.js#L86 -{{% /tab %}} - -{{% tab "Ruby" %}} - -To manually correlate your traces with your logs, patch the logging module you are using with a processor that translates the OpenTelemetry formatted `span_id` into Datadog format. The following example uses the [Ruby Standard Logging Library][1]. For Rails applications or other logging libraries, it may be more appropriate to [modify the Datadog SDK examples][2]. You can also find [an example OpenTelemetry instrumented Ruby application with trace and log correlation][3] in the `trace-examples` GitHub repository. - -```ruby -logger = Logger.new(STDOUT) -logger.progname = 'multivac' -original_formatter = Logger::Formatter.new -logger.formatter = proc do |severity, datetime, progname, msg| - current_span = OpenTelemetry::Trace.current_span(OpenTelemetry::Context.current).context - - trace_id_int = current_span.trace_id.unpack1('H*').to_i(16) - dd_trace_id = trace_id_int < 2**64 ? trace_id_int.to_s : format("%032x", trace_id_int) - dd_span_id = current_span.span_id.unpack1('H*').to_i(16).to_s - - if current_span - "#{{datetime: datetime, progname: progname, severity: severity, msg: msg, 'dd.trace_id': dd_trace_id, 'dd.span_id': dd_span_id}.to_json}\n" - else - "#{{datetime: datetime, progname: progname, severity: severity, msg: msg}.to_json}\n" - end -end - -logger.info("Example log line with trace correlation info") -``` - - - - -[1]: https://ruby-doc.org/stdlib-3.0.0/libdoc/logger/rdoc/index.html -[2]: /tracing/other_telemetry/connect_logs_and_traces/ruby/#manually-inject-trace-and-span-ids -[3]: https://github.com/DataDog/trace-examples/blob/98626d924f82666de60d6b2d6a65d87eebebdff1/opentelemetry/ruby-microservice/app.rb#L21-L35 -{{% /tab %}} - -{{% tab "Java" %}} - -To manually correlate your traces with your logs, first enable the [openTelemetry-java-instrumentation Logger MDC Instrumentation][1]. Then, patch the logging module you are using with a processor that translates the OpenTelemetry formatted `span_id` into Datadog format. The following example uses [Spring Boot and Logback][2]. For other logging libraries, it may be more appropriate to [modify the Datadog SDK examples][3]. +For Go applications using a structured logger such as `zap`, use a bridge such as `otelzap`. This wraps your logger and automatically injects the active `trace_id` and `span_id` into every log message. -```java -String traceIdValue = Span.current().getSpanContext().getTraceId(); -String traceIdHexString = traceIdValue.substring(traceIdValue.length() - 16 ); +First, ensure you have an initialized OpenTelemetry `LoggerProvider`. Then, use it to create your `zap` logger instance: -String spanIdHexString = Span.current().getSpanContext().getSpanId(); -long datadogSpanId = Long.parseUnsignedLong(spanIdHexString, 16); -String datadogSpanIdString = Long.toUnsignedString(datadogSpanId); +```go +import "go.opentelemetry.io/contrib/bridges/otelzap" -MDC.put("dd.trace_id", traceIdHexString); -MDC.put("dd.span_id", datadogSpanIdString); +// Replace your standard zap logger with the otelzap-backed one +logger := zap.New(otelzap.NewCore( + "my-service-name", + otelzap.WithLoggerProvider(loggerProvider), +)) -logger.info("Log Message"); +// Now, logs written with this logger are automatically correlated +logger.Info("Processing user request") ``` -See [Java Log Collection][4] on how to send your Java logs to Datadog. - -[1]: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/logger-mdc-instrumentation.md -[2]: https://docs.spring.io/spring-boot/docs/2.1.18.RELEASE/reference/html/boot-features-logging.html -[3]: /tracing/other_telemetry/connect_logs_and_traces/java/?tab=log4j2#manually-inject-trace-and-span-ids -[4]: /logs/log_collection/java/?tab=logback -{{% /tab %}} - -{{% tab "PHP" %}} - -For trace and log correlation in PHP, modify the [Datadog SDK PHP examples][1] to include the additional steps discussed above. - -[Contact Datadog support][2] with any questions. - - +To see how the `LoggerProvider` is configured in a complete application, see the [full Go example in the examples repository][100]. +[100]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/golang/calendar/main.go -[1]: /tracing/other_telemetry/connect_logs_and_traces/php/ -[2]: /help/ {{% /tab %}} -{{% tab "Go" %}} - -To manually correlate your traces with your logs, patch the logging module you are using with a function that translates the OpenTelemetry formatted `span_id` into Datadog format. The following example uses the [logrus Library][1]. - -```go -package main - -import ( - "context" - log "github.com/sirupsen/logrus" - "go.opentelemetry.io/otel" - "strconv" -) - -func main() { - ctx := context.Background() - tracer := otel.Tracer("example/main") - ctx, span := tracer.Start(ctx, "example") - defer span.End() - - log.SetFormatter(&log.JSONFormatter{}) - - standardFields := log.Fields{ - "dd.trace_id": span.SpanContext().TraceID().String(), - "dd.span_id": convertSpanID(span.SpanContext().SpanID().String()), - "dd.service": "serviceName", - "dd.env": "serviceEnv", - "dd.version": "serviceVersion", - } - - log.WithFields(standardFields).WithContext(ctx).Info("hello world") -} +{{% tab "Java" %}} -func convertSpanID(id string) string { - if len(id) < 16 { - return "" - } - if len(id) > 16 { - id = id[16:] - } - intValue, err := strconv.ParseUint(id, 16, 64) - if err != nil { - return "" - } - return strconv.FormatUint(intValue, 10) -} +To inject the trace context in Java, you can use the OpenTelemetry Logback Appender. Add the `io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0` dependency to your project and configure it in your `logback.xml`: +```xml + + + true + + + + + ``` -[Contact Datadog support][2] with any questions. - +For complete, working example configuration, see the [full Java example in the examples repository][200]. +[200]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/java/calendar/src/main/resources/logback.xml -[1]: https://github.com/sirupsen/logrus -[2]: /help/ {{% /tab %}} +{{< /tabs >}} -{{% tab ".NET" %}} - -To manually correlate traces with logs, convert the OpenTelemetry `SpanId` into the format used by Datadog. Add the trace ID to your logs along with the converted span ID, under `dd.trace_id` and `dd.span_id` attributes, respectively. The following example uses the [Serilog library][1], and shows how to convert the OpenTelemetry (`System.DiagnosticSource.Activity`) span ID into Datadog's required format: +### 2. Verify the log output -```csharp -var ddTraceId = Activity.Current.TraceId.ToString(); +After configuring your logging library, your JSON logs should contain `trace_id` and `span_id` attributes. Datadog uses these to link the log to the exact trace and span that was active when the log was generated. -var stringSpanId = Activity.Current.SpanId.ToString(); -var ddSpanId = Convert.ToUInt64(stringSpanId, 16).ToString(); +Here is an example of a properly formatted JSON log entry: -using (LogContext.PushProperty("dd.trace_id", ddTraceId)) -using (LogContext.PushProperty("dd.span_id", ddSpanId)) +```json { - Serilog.Log.Logger.Information("Example log line with trace correlation info"); + "timestamp": 1720557413000, + "level": "INFO", + "message": "Processing user request", + "service": "my-service", + "env": "production", + "version": "1.2.3", + "trace_id": "8738839999436033394", + "span_id": "1006654203791334032" } ``` - -[1]: https://serilog.net/ -{{% /tab %}} - -{{< /tabs >}} - -## Further Reading +## Further reading {{< partial name="whats-next/whats-next.html" >}} -[1]: /tracing/other_telemetry/connect_logs_and_traces/ -[2]: https://github.com/open-telemetry/opentelemetry-specification/blob/eeef21259a12d61100804eff2e12ba06523821c3/specification/trace/api.md#retrieving-the-traceid-and-spanid -[3]: /api/latest/tracing/#send-traces +[1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging +[2]: https://github.com/DataDog/opentelemetry-examples From d1251d3e7f401cd643485680b33ae4059f4f56c9 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Mon, 14 Jul 2025 17:23:39 -0600 Subject: [PATCH 07/24] Add separate pages for metrics and dbm. --- .../en/opentelemetry/correlate/_index copy.md | 361 ------------------ content/en/opentelemetry/correlate/_index.md | 6 +- .../opentelemetry/correlate/dbm_and_traces.md | 82 ++++ .../correlate/metrics_and_traces.md | 79 ++++ .../integrations/host_metrics.md | 4 +- 5 files changed, 166 insertions(+), 366 deletions(-) delete mode 100644 content/en/opentelemetry/correlate/_index copy.md create mode 100644 content/en/opentelemetry/correlate/dbm_and_traces.md create mode 100644 content/en/opentelemetry/correlate/metrics_and_traces.md diff --git a/content/en/opentelemetry/correlate/_index copy.md b/content/en/opentelemetry/correlate/_index copy.md deleted file mode 100644 index d986a3deea190..0000000000000 --- a/content/en/opentelemetry/correlate/_index copy.md +++ /dev/null @@ -1,361 +0,0 @@ ---- -title: Correlate OpenTelemetry Data -description: Learn how to correlate your OpenTelemetry traces, metrics, logs, and other telemetry in Datadog to get a unified view of your application's performance. -aliases: - - /opentelemetry/otel_logs/ -further_reading: -- link: "https://www.datadoghq.com/blog/opentelemetry-instrumentation/" - tag: "Blog" - text: "Datadog's partnership with OpenTelemetry" ---- - -## Overview - -Getting a unified view of your application's performance requires connecting its traces, metrics, logs, user interactions, and more. This guide provides the steps to correlate your OpenTelemetry data in Datadog, allowing you to navigate between all related telemetry in a single view. - -## Prerequisite: Unified service tagging - -Datadog uses three standard tags to link telemetry together: `env`, `service`, and `version`. To apply these tags, you must configure the corresponding resource attributes in OpenTelemetry. - -When using OpenTelemetry, you set these tags by defining a standard set of resource attributes. Datadog automatically maps these attributes to the correct tags. - -| OpenTelemetry Resource Attribute | Datadog Tag | Notes | -|----------------------------------|-------------|---------------------------------------------------------------------------------------------------------| -| `deployment.environment.name` | `env` | **Recommended**. Supported in Agent v7.58.0+ and Collector Exporter v0.110.0+. | -| `deployment.environment` | `env` | Use if you are running an Agent version older than v7.58.0 or a Collector Exporter older than v0.110.0. | -| `service.name` | `service` | | -| `service.version` | `version` | | - -You can set these attributes in your application's environment variables, SDK, or in the OpenTelemetry Collector. - -{{< tabs >}} -{{% tab "Environment Variables" %}} - -Set the `OTEL_RESOURCE_ATTRIBUTES` environment variable with your service's information: - -```sh -export OTEL_SERVICE_NAME="my-service" -export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.version=1.2.3" -``` - -{{% /tab %}} -{{% tab "SDK" %}} - -Create a Resource with the required attributes and associate it with your TracerProvider in your application code. - -Here's an example using the OpenTelemetry SDK for Python: - -```python -from opentelemetry.sdk.resources import Resource -from opentelemetry.sdk.trace import TracerProvider - -resource = Resource(attributes={ - "service.name": "", - "deployment.environment.name": "", - "service.version": "" -}) -tracer_provider = TracerProvider(resource=resource) -``` - -{{% /tab %}} -{{% tab "Collector" %}} - -Use the `resource` processor in your Collector configuration to set the resource attributes on your telemetry data: - -```yaml -processors: - resource: - attributes: - - key: service.name - value: "my-service" - action: upsert - - key: deployment.environment.name - value: "production" - action: upsert - - key: service.version - value: "1.2.3" - action: upsert -... -``` - -{{% /tab %}} -{{< /tabs >}} - -## Correlating telemetry - -### Traces and metrics - -Correlating traces with infrastructure metrics allows you to pivot from a slow request directly to the CPU and memory metrics of the host or container it ran on. This helps you determine if resource contention was the root cause of a performance issue. - -Correlation between traces and infrastructure metrics relies on the following resource attributes: - -- `host.name`: For correlating with host metrics (CPU, memory, disk). -- `container.id`: For correlating with container metrics. - -#### Setup - -##### Enable the host metrics receiver - -To collect system-level metrics, enable the `hostmetrics` receiver in your OpenTelemetry Collector configuration. This receiver gathers metrics like CPU, memory, disk, and network usage. - -```yaml -receivers: - hostmetrics: - collection_interval: 30s - scrapers: - cpu: - memory: - -processors: - batch: - -exporters: - datadog: {} - -service: - pipelines: - metrics: - receivers: [hostmetrics] - processors: [batch] - exporters: [datadog] -``` - -For detailed setup instructions, see [Host Metrics][2] documentation. - -##### Ensure consistent host and container tagging - -For correlation to work, the `host.name` (or `container.id`) attribute on your traces must match the corresponding attribute on the metrics collected by the `hostmetrics` receiver. - -### Traces and logs - -Correlating traces with logs allows you to jump from a specific span in a trace to the exact logs that were generated during that operation, making debugging faster and more intuitive. - -The only requirement for connecting traces and logs is to inject the active `trace_id` and `span_id` from your application's trace context into your logs. - -#### Setup - -Most OpenTelemetry logging instrumentation can automatically inject the active trace context into your logs. Ensure this feature is enabled for your logging library. For Datadog to parse the attributes correctly, your logs must be sent in JSON format. - -For complete example applications, see the [Datadog OpenTelemetry Examples repository][1]. - -{{< tabs >}} -{{% tab "Go" %}} - -For Go applications using a structured logger such as `zap`, the recommended approach is to use a bridge such as **`otelzap`**. This wraps your logger and automatically injects the active `trace_id` and `span_id` into every log message. - -First, ensure you have an initialized OpenTelemetry `LoggerProvider`. Then, use it to create your `zap` logger instance: - -```go -import "go.opentelemetry.io/contrib/bridges/otelzap" - -// Replace your standard zap logger with the otelzap-backed one -logger := zap.New(otelzap.NewCore( - "my-service-name", - otelzap.WithLoggerProvider(loggerProvider), -)) - -// Now, logs written with this logger are automatically correlated -logger.Info("Processing user request") -``` - -To see how the `LoggerProvider` is configured in a complete application, see the [full Go example in the examples repository][100]. - -[100]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/golang/calendar/main.go - -{{% /tab %}} - -{{% tab "Java" %}} - -To inject the trace context in Java, you can use the OpenTelemetry Logback Appender. Add the `io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0` dependency to your project and configure it in your `logback.xml`: - -```xml - - - true - - - - - - -``` - -For complete, working example configuration, see the [full Java example in the examples repository][200]. - -[200]: https://github.com/DataDog/opentelemetry-examples/blob/main/apps/rest-services/java/calendar/src/main/resources/logback.xml - -{{% /tab %}} -{{< /tabs >}} - -#### Verify the log output - -After configuration, your JSON logs should contain `trace_id` and `span_id` attributes. Datadog uses these to link the log to the exact trace and span that was active when the log was generated. - -Here is an example of a properly formatted JSON log entry: - -```json -{ - "timestamp": 1720557413000, - "level": "INFO", - "message": "Processing user request", - "service": "my-service", - "env": "production", - "version": "1.2.3", - "trace_id": "8738839999436033394", - "span_id": "1006654203791334032" -} -``` - -
For complete, working examples of trace and log correlation for various languages, see the Datadog OpenTelemetry Examples repository.
- -## Correlating with Datadog products - -### Real User Monitoring (RUM) - -Correlating RUM and APM allows you to follow a user's journey from a slow button click on your website all the way to the specific backend database query that caused the delay. - -This correlation is achieved by configuring the RUM SDK to inject trace context headers into requests made to your backend. The default injection style includes `tracecontext` and `datadog` headers. - -{{< tabs >}} -{{% tab "Browser" %}} - -After completing the standard Browser RUM setup, modify `allowedTracingUrls` in your RUM SDK initialization to specify which propagator to use for each backend endpoint. - -```javascript -import { datadogRum } from '@datadog/browser-rum' - -datadogRum.init({ - // ...otherConfig, - allowedTracingUrls: [ - { match: "https://api.example.com", propagatorTypes: ["tracecontext"] } - ] -}) -``` - -`propagatorTypes` accepts a list of desired propagators: `datadog`, `tracecontext`, `b3`, and `b3multi`. - -{{% /tab %}} -{{% tab "iOS" %}} - -After completing the standard [iOS RUM setup][200], use `.traceWithHeaders(hostsWithHeaders:sampleRate:)` to specify which tracing headers to inject for requests to your backend hosts. - -```swift -RUM.enable( - with: RUM.Configuration( - applicationID: "", - urlSessionTracking: .init( - firstPartyHostsTracing: .traceWithHeaders( - hostsWithHeaders: [ - "api.example.com": [.tracecontext] - ], - sampleRate: 100 - ) - ) - ) -) -``` - -`TracingHeaderType` is an enum that includes `.datadog`, `.tracecontext`, `.b3`, and `.b3multi`. - -[200]: /real_user_monitoring/ios/ - -{{% /tab %}} -{{% tab "Android" %}} - -After completing the standard [Android RUM setup][300], configure your `OkHttpClient` interceptor with a map of your backend hosts and the desired `TracingHeaderType` for each. - -```kotlin -val tracedHosts = mapOf( - "example.com" to setOf(TracingHeaderType.TRACECONTEXT), - "example.eu" to setOf(TracingHeaderType.DATADOG) -) - -val okHttpClient = OkHttpClient.Builder() - .addInterceptor(DatadogInterceptor.Builder(tracedHosts).build()) - .addNetworkInterceptor(TracingInterceptor.Builder(tracedHosts).build()) - .eventListenerFactory(DatadogEventListener.Factory()) - .build() -``` - -`TracingHeaderType` is an enum that includes `.DATADOG`, `.TRACECONTEXT`, `.B3`, and `.B3MULTI`. - -[300]: /real_user_monitoring/android/ - -{{% /tab %}} -{{< /tabs >}} - -For **React Native**, **Flutter**, and **Kotlin Multiplatform**, see the platform-specific RUM documentation for configuring `firstPartyHosts`, `firstPartyHostsWithTracingHeaders`, or the `datadogKtorPlugin` respectively: - -- [React Native RUM Documentation][3] -- [Flutter RUM Documentation][4] -- [Kotlin Multiplatform RUM Documentation][5] - -### Database Monitoring (DBM) - -Correlate backend traces to detailed database performance data in Datadog, including query metrics and execution plans, to identify the exact queries that are slowing down your application. - -#### Step 1: Instrument your database spans - -For DBM correlation to work, your database spans must include the following attributes. - -| Attribute | Description | Example | -|----------------|---------------------------------------------------------------------------------------|------------------------------------| -| `db.system` | **Required.** The database technology, such as `postgres`, `mysql`, or `sqlserver`. | `postgres` | -| `db.statement` | **Required.** The raw SQL query text. This is used for obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` | -| `db.name` | The logical database or schema name being queried. | `user_accounts` | -| `span.type` | **Required (Datadog-specific).** The type of span. Must be `sql`. | `sql` | - -#### Step 2: Configure your ingest path - -Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. - -{{< tabs >}} -{{% tab "Datadog Agent (DDOT Collector)" %}} - - -If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`: - -```yaml -datadog: - otelCollector: - featureGates: datadog.EnableOperationAndResourceNameV2 -``` - -{{% /tab %}} -{{% tab "OTel Collector" %}} - -When starting the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate. This is available in Collector v0.118.0 and later. - -```bash -otelcontribcol --config=config.yaml \ ---feature-gates=datadog.EnableOperationAndResourceNameV2 -``` - -{{% /tab %}} - -{{% tab "Datadog Agent (OTLP Ingest)" %}} - -In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`. - -{{% /tab %}} - -{{< /tabs >}} - -## What's next? - -Now that your telemetry is correlated, you can explore the unified views in Datadog: - -- **Service Catalog**: View the health of all your services in one place. -- **Trace Explorer**: Search and analyze individual requests as they travel across your services. -- **Log Explorer**: Search and filter all your logs, and pivot to related traces. - -## Further reading - -{{< partial name="whats-next/whats-next.html" >}} - -[1]: https://github.com/DataDog/opentelemetry-examples -[2]: /opentelemetry/integrations/host_metrics/ -[3]: /real_user_monitoring/reactnative/ -[4]: /real_user_monitoring/mobile_and_tv_monitoring/flutter/ -[5]: /real_user_monitoring/mobile_and_tv_monitoring/kotlin_multiplatform/ \ No newline at end of file diff --git a/content/en/opentelemetry/correlate/_index.md b/content/en/opentelemetry/correlate/_index.md index 966ea87bc7458..a4d4e9b4e8a7d 100644 --- a/content/en/opentelemetry/correlate/_index.md +++ b/content/en/opentelemetry/correlate/_index.md @@ -96,6 +96,6 @@ After unified service tagging is configured, you can connect your various teleme {{< partial name="whats-next/whats-next.html" >}} [1]: /opentelemetry/correlate/logs_and_traces -[1]: /opentelemetry/correlate/metrics_and_traces -[1]: /opentelemetry/correlate/rum_and_traces -[1]: /opentelemetry/correlate/dbm_and_traces \ No newline at end of file +[2]: /opentelemetry/correlate/metrics_and_traces +[3]: /opentelemetry/correlate/rum_and_traces +[4]: /opentelemetry/correlate/dbm_and_traces \ No newline at end of file diff --git a/content/en/opentelemetry/correlate/dbm_and_traces.md b/content/en/opentelemetry/correlate/dbm_and_traces.md new file mode 100644 index 0000000000000..c6a92fc32913c --- /dev/null +++ b/content/en/opentelemetry/correlate/dbm_and_traces.md @@ -0,0 +1,82 @@ +--- +title: Correlate OpenTelemetry Traces and DBM +further_reading: +- link: "/opentelemetry/otel_tracing/" + tag: "Documentation" + text: "Send OpenTelemetry Traces to Datadog" +--- + +## Overview + +Correlate backend traces to detailed database performance data in Datadog Database Monitoring (DBM). This allows you to link spans from your OpenTelemetry-instrumented application directly to query metrics and execution plans to identify the exact queries that are slowing down your application. + +## Requirements + +Before you begin, ensure you have configured [unified service tagging][1]. This is required for all data correlation in Datadog. + +## Setup + +To correlate traces and metrics, you must: + +1. **Instrument database spans**: Add specific OpenTelemetry attributes to your database spans to enable correlation with DBM. + +2. **Configure trace ingestion path**: Enable the correct feature gate on your Collector or Agent to ensure database spans are properly processed for DBM. + +### Step 1: Instrument your database spans + +For DBM correlation to work, your database spans must include the following attributes. + +| Attribute | Description | Example | +|----------------|-----------------------------------------------------------------------------------------------------|------------------------------------| +| `db.system` | **Required.** The database technology, such as `postgres`, `mysql`, or `sqlserver`. | `postgres` | +| `db.statement` | **Required.** The raw SQL query text. This is used for obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` | +| `db.name` | The logical database or schema name being queried. | `user_accounts` | +| `span.type` | **Required (Datadog-specific).** The type of span such as `sql`,`postgres`, `mysql`, or `sql.query` | `sql` | + +### Step 2: Configure your ingest path + +Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. + +{{< tabs >}} +{{% tab "Datadog Agent (DDOT Collector)" %}} + + +If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`: + +```yaml +datadog: + otelCollector: + featureGates: datadog.EnableOperationAndResourceNameV2 +``` + +{{% /tab %}} +{{% tab "OTel Collector" %}} + +When starting the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate. This is available in Collector v0.118.0 and later. + +```sh +otelcontribcol --config=config.yaml \ +--feature-gates=datadog.EnableOperationAndResourceNameV2 +``` + +{{% /tab %}} + +{{% tab "Datadog Agent (OTLP Ingest)" %}} + +In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`. + +{{% /tab %}} + +{{< /tabs >}} + +### View correlated data in Datadog + +TODO + + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging +[2]: /opentelemetry/integrations/host_metrics \ No newline at end of file diff --git a/content/en/opentelemetry/correlate/metrics_and_traces.md b/content/en/opentelemetry/correlate/metrics_and_traces.md new file mode 100644 index 0000000000000..ffc927418863a --- /dev/null +++ b/content/en/opentelemetry/correlate/metrics_and_traces.md @@ -0,0 +1,79 @@ +--- +title: Correlate OpenTelemetry Traces and Metrics +further_reading: +- link: "/opentelemetry/otel_tracing/" + tag: "Documentation" + text: "Send OpenTelemetry Traces to Datadog" +--- + +## Overview + +Correlating traces with host metrics allows you to pivot from a slow request directly to the CPU and memory metrics of the host or container it ran on. This helps you determine if resource contention was the root cause of a performance issue. + +Correlation between traces and metrics relies on the following resource attributes: + +- `host.name`: For correlating with host metrics (CPU, memory, disk). +- `container.id`: For correlating with container metrics. + +## Requirements + +Before you begin, ensure you have configured [unified service tagging][1]. This is required for all data correlation in Datadog. + +## Setup + +To correlate traces and metrics, you must: + +1. **Collect Host Metrics**: You must have the OpenTelemetry Collector configured to collect and send host metrics to Datadog. + +2. **Ensure Consistent Tagging**: Your traces and metrics must share a consistent `host.name` (for hosts) or `container.id` (for containers) attribute for Datadog to link them. + + +### 1. Collect host metrics + +To collect system-level metrics from your infrastructure, enable the `hostmetrics` receiver in your OpenTelemetry Collector configuration. This receiver gathers metrics like CPU, memory, disk, and network usage. + +Add the `hostmetrics` receiver to the `receivers` section of your Collector configuration and enable it in your `metrics` pipeline: + + +```yaml +receivers: + hostmetrics: + collection_interval: 10s + scrapers: + cpu: + memory: + disk: + ... + +service: + pipelines: + metrics: + receivers: [hostmetrics, ...] + processors: [...] + exporters: [...] +``` + +For the complete, working configuration, including Kubernetes-specific setup, see the [Host Metrics][2] documentation. + +### 2. Ensure consistent host and container tagging + +For correlation to work, the `host.name` (or `container.id`) attribute on your traces must match the corresponding attribute on the metrics collected by the `hostmetrics` receiver. + +## View correlated data in Datadog + +After your application is sending traces and the Collector is sending host metrics, you can see the correlation in the APM Trace View. + +1. Navigate to [**APM** > **Traces**][3]. +2. Find and click on a trace from your instrumented service. +3. In the trace's flame graph, select a span that ran on the instrumented host. +4. In the details panel, click the **Infrastructure** tab. You should see the host metrics, like CPU and memory utilization, from the host that executed that part of the request. + +This allows you to immediately determine if a spike in host metrics corresponds with the performance of a specific request. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging +[2]: /opentelemetry/integrations/host_metrics +[3]: https://app.datadoghq.com/apm/traces \ No newline at end of file diff --git a/content/en/opentelemetry/integrations/host_metrics.md b/content/en/opentelemetry/integrations/host_metrics.md index b11285bd3534a..d7a839318d47f 100644 --- a/content/en/opentelemetry/integrations/host_metrics.md +++ b/content/en/opentelemetry/integrations/host_metrics.md @@ -99,7 +99,7 @@ The metrics, mapped to Datadog metrics, are used in the following views: - [Host default dashboards][8] - [APM Trace view Host info][9] -**Note**: To correlate trace and host metrics, configure [Universal Service Monitoring attributes][10] for each service, and set the `host.name` resource attribute to the corresponding underlying host for both service and collector instances. +**Note**: To correlate trace and host metrics, configure [Unified Service Tagging attributes][10] for each service, and set the `host.name` resource attribute to the corresponding underlying host for both service and collector instances. The following table shows which Datadog host metric names are associated with corresponding OpenTelemetry host metric names, and, if applicable, what math is applied to the OTel host metric to transform it to Datadog units during the mapping. @@ -168,6 +168,6 @@ Value: 1153183744 [7]: https://app.datadoghq.com/infrastructure [8]: /opentelemetry/collector_exporter/#out-of-the-box-dashboards [9]: /tracing/trace_explorer/trace_view/?tab=hostinfo -[10]: /universal_service_monitoring/setup/ +[10]: /opentelemetry/correlate/#prerequisite-unified-service-tagging From 75e126ca825692187a6dd9324b078354bdd9356d Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Mon, 14 Jul 2025 17:34:40 -0600 Subject: [PATCH 08/24] Add new pages to left-nav. --- config/_default/menus/main.en.yaml | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index f8d3c368833b0..ad54d56ef5cbd 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -785,11 +785,21 @@ menu: identifier: otel_logs parent: otel_explore weight: 701 + - name: Correlate Metrics and Traces + url: /opentelemetry/correlate/metrics_and_traces/ + identifier: otel_metrics_traces + parent: otel_explore + weight: 702 - name: Correlate RUM and Traces url: /opentelemetry/correlate/rum_and_traces/ identifier: otel_rum parent: otel_explore - weight: 702 + weight: 703 + - name: Correlate DBM and Traces + url: /opentelemetry/correlate/dbm_and_traces/ + identifier: otel_dbm + parent: otel_explore + weight: 704 - name: Integrations url: opentelemetry/integrations/ identifier: otel_integrations From e0dd27df2ede7c45021f859d1988615c6d551d15 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Wed, 16 Jul 2025 10:43:33 -0600 Subject: [PATCH 09/24] Clean up. --- .../opentelemetry/correlate/dbm_and_traces.md | 9 +++++-- .../connect_logs_and_traces/opentelemetry.md | 25 +++++++++++++++++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/content/en/opentelemetry/correlate/dbm_and_traces.md b/content/en/opentelemetry/correlate/dbm_and_traces.md index c6a92fc32913c..4b00268dee5f3 100644 --- a/content/en/opentelemetry/correlate/dbm_and_traces.md +++ b/content/en/opentelemetry/correlate/dbm_and_traces.md @@ -71,12 +71,17 @@ In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment va ### View correlated data in Datadog -TODO +After your application is sending traces, you can see the correlation in the APM Trace View: +1. Navigate to [**APM** > **Traces**][3]. +2. Find and click on a trace from your instrumented service. +3. In the trace's flame graph, select a database span (for example, a span with `span.type: sql`) +4. In the details panel, click the **SQL Queries** tab. You should see the host metrics, like CPU and memory utilization, from the host that executed that part of the request. ## Further reading {{< partial name="whats-next/whats-next.html" >}} [1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging -[2]: /opentelemetry/integrations/host_metrics \ No newline at end of file +[2]: /opentelemetry/integrations/host_metrics +[3]: https://app.datadoghq.com/apm/traces \ No newline at end of file diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index 7868ef2d88112..b33a7544d3bf2 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -114,9 +114,34 @@ Here is an example of a properly formatted JSON log entry: } ``` +## View correlated data in Datadog + +After your application is sending traces, you can navigate between them in Datadog. + +### From a trace to logs + +1. Navigate to [**APM** > **Traces**][3]. +2. Find and click on a trace from your instrumented service. +3. Select any span in the flame graph to view its details. +4. Click the **Logs** tab. + +Here, you can see all the logs generated during the execution of that specific span. + +### From a log to a trace + +1. Navigate to [**Logs** > **Explorer**][4]. +2. Find and click a log entry from your instrumented service. +3. Click the **Trace** tab. + +Here, you can see a flame graph of the associated trace, with the span that generated the log. + +Click **View Trace in APM** to pivot directly to the full APM trace associated with that log event, allowing you to see the context of the entire request. + ## Further reading {{< partial name="whats-next/whats-next.html" >}} [1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging [2]: https://github.com/DataDog/opentelemetry-examples +[3]: https://app.datadoghq.com/apm/traces +[4]: https://app.datadoghq.com/logs From dd49a7b4ddb3472b9234be6eef7509aa13e8b8fd Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Wed, 16 Jul 2025 11:51:40 -0600 Subject: [PATCH 10/24] Add example to DBM page. --- .../opentelemetry/correlate/dbm_and_traces.md | 26 ++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/content/en/opentelemetry/correlate/dbm_and_traces.md b/content/en/opentelemetry/correlate/dbm_and_traces.md index 4b00268dee5f3..3e8782ce04eb8 100644 --- a/content/en/opentelemetry/correlate/dbm_and_traces.md +++ b/content/en/opentelemetry/correlate/dbm_and_traces.md @@ -33,6 +33,29 @@ For DBM correlation to work, your database spans must include the following attr | `db.name` | The logical database or schema name being queried. | `user_accounts` | | `span.type` | **Required (Datadog-specific).** The type of span such as `sql`,`postgres`, `mysql`, or `sql.query` | `sql` | +#### Example + +The method for adding these attributes depends on your setup. If you are using an OpenTelemetry auto-instrumentation library for your database client, see its documentation for configuration options. If you are manually creating spans with the OpenTelemetry SDK, you can set the attributes directly in your code. For more information, see the [OpenTelemetry documentation][4]. + +The following is a conceptual example of manual instrumentation using Python's OpenTelemetry SDK: + +```python +from opentelemetry import trace + +tracer = trace.get_tracer("my-app.instrumentation") + +# When making a database call, create a span and set attributes +with tracer.start_as_current_span("postgres.query") as span: + # Set attributes required for DBM correlation + span.set_attribute("span.type", "sql") + span.set_attribute("db.system", "postgres") + span.set_attribute("db.statement", "SELECT * FROM users WHERE id = ?") + span.set_attribute("db.name", "user_accounts") + + # Your actual database call would go here + # db_cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,)) +``` + ### Step 2: Configure your ingest path Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. @@ -84,4 +107,5 @@ After your application is sending traces, you can see the correlation in the APM [1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging [2]: /opentelemetry/integrations/host_metrics -[3]: https://app.datadoghq.com/apm/traces \ No newline at end of file +[3]: https://app.datadoghq.com/apm/traces +[4]: https://opentelemetry.io/docs/languages/ \ No newline at end of file From e974e04f9f4bc73c16b1e042b8a1e57c38a2707f Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Thu, 17 Jul 2025 10:51:55 -0600 Subject: [PATCH 11/24] Restructure into two distinct steps with a few log pipelines. --- .../connect_logs_and_traces/opentelemetry.md | 141 +++++++++++++++--- 1 file changed, 119 insertions(+), 22 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index b33a7544d3bf2..39e06d901b3f3 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -35,15 +35,15 @@ To correlate OpenTelemetry traces and logs in Datadog, you must: - **Inject Trace Context**: Your application's logger must be configured to inject the `trace_id` and `span_id` of the active trace into your logs. -- **Use JSON Formatting**: Your logs must be sent in JSON format so Datadog can automatically parse the trace context attributes. +- **Send Logs to Datadog**: Your logs, now enriched with trace context, must be collected and sent to Datadog.
For logs collected directly by the Datadog Agent (instead of through the OpenTelemetry Collector), automatic correlation relies on the dd.trace_id attribute.
-#### 1. Configure your logging library +#### 1. Inject trace context into your logs -The recommended approach is to configure your logging library to automatically inject the standard OpenTelemetry `trace_id` and `span_id` attributes into your logs. +The recommended approach is to configure your logging library to automatically inject the standard OpenTelemetry `trace_id` and `span_id` attributes into your logs. The following examples show how to configure common logging libraries to do this. -The following examples show how to configure logging instrumentation to automatically inject trace context. For complete, working applications, see the [Datadog OpenTelemetry Examples repository][2]. +For complete, working applications, see the [Datadog OpenTelemetry Examples repository][2]. {{< tabs >}} {{% tab "Go" %}} @@ -95,24 +95,121 @@ For complete, working example configuration, see the [full Java example in the e {{% /tab %}} {{< /tabs >}} -### 2. Verify the log output - -After configuring your logging library, your JSON logs should contain `trace_id` and `span_id` attributes. Datadog uses these to link the log to the exact trace and span that was active when the log was generated. - -Here is an example of a properly formatted JSON log entry: - -```json -{ - "timestamp": 1720557413000, - "level": "INFO", - "message": "Processing user request", - "service": "my-service", - "env": "production", - "version": "1.2.3", - "trace_id": "8738839999436033394", - "span_id": "1006654203791334032" -} -``` +### 2. Choose your log pipeline + +Once your logs are instrumented with trace context, you need to send them to Datadog. The simplest approach is to send them directly from your application to the OpenTelemetry Collector using OTLP. However, you can also scrape logs from files or collect logs using the Datadog Agent. + +#### Send logs using OTLP + +This is the simplest and most direct method. Your application sends logs to the Collector over the network, avoiding the complexity of writing to and parsing local files. + +1. **Configure your Application to Export Logs using OTLP**: In your OpenTelemetry SDK setup, configure a `LogRecordProcessor` to use an `OTLPLogExporter`. The following example shows how to do this in Python: + ```python + # In your OTel SDK setup for Python + import logging + from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler + from opentelemetry.sdk._logs.export import BatchLogRecordProcessor + from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter + + # Configure the OTLP Log Exporter to send to your Collector + # Note: The endpoint should point to your OpenTelemetry Collector. + # The default port is 4317 for gRPC and 4318 for HTTP. + exporter = OTLPLogExporter(endpoint="localhost:4317", insecure=True) + + log_provider = LoggerProvider() + log_provider.add_log_record_processor(BatchLogRecordProcessor(exporter)) + + # Attach to the root logger + handler = LoggingHandler(level=logging.INFO, logger_provider=log_provider) + logging.getLogger().addHandler(handler) + ``` +2. **Configure the Collector to Receive OTLP Logs**: In your Collector's `config.yaml`, enable the `otlp` receiver and add it to your `logs` pipeline: + ```yaml + receivers: + otlp: + protocols: + grpc: + http: + + exporters: + datadog: + # ... your datadog exporter config + + service: + pipelines: + logs: + receivers: [otlp] + processors: [batch] + exporters: [datadog] + ``` + +#### Scrape logs from files + +This approach is useful if you have a requirement to keep local log files for compliance or other tooling. + +1. **Configure your Application to Output JSON Logs**: In your OpenTelemetry SDK setup, configure a `ConsoleLogRecordExporter` to format logs as JSON and write them to `stdout` or a file. + ```python + # In your OTel SDK setup for Python + import logging + from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler + from opentelemetry.sdk._logs.export import ConsoleLogRecordExporter, SimpleLogRecordProcessor + from opentelemetry.sdk.extension.opentelemetry_log_formatter import JSONLogRecordFormatter + + # Configure the exporter to write JSON to the console (stdout) + formatter = JSONLogRecordFormatter() + exporter = ConsoleLogRecordExporter(formatter=formatter) + + log_provider = LoggerProvider() + log_provider.add_log_record_processor(SimpleLogRecordProcessor(exporter)) + + # Attach to the root logger + handler = LoggingHandler(level=logging.INFO, logger_provider=log_provider) + logging.getLogger().addHandler(handler) + ``` + +2. **Verify the Log Output**: After configuring your logging library, your JSON logs should contain `trace_id` and `span_id` attributes. Datadog uses these to link the log to the exact trace and span that was active when the log was generated. Here is an example of a properly formatted JSON log entry: + + ```json + { + "timestamp": 1720557413000, + "level": "INFO", + "message": "Processing user request", + "service": "my-service", + "env": "production", + "version": "1.2.3", + "trace_id": "8738839999436033394", + "span_id": "1006654203791334032" + } + ``` +3. **Configure the Collector to Scrape Log Files**: In your Collector's `config.yaml`, enable the `filelog` receiver. Configure it to find your log files and parse them as JSON. + ```yaml + receivers: + filelog: + include: [ /var/log/my-app/*.log ] # Path to your log files + operators: + - type: json_parser # Parse the log line as JSON + # Note: The `parse_from` keys below must exactly match the + # field names present in your JSON log output. + timestamp: + parse_from: attributes.timestamp + layout_type: epoch + layout: ms + severity: + parse_from: attributes.level + + service: + pipelines: + logs: + receivers: [filelog] + # ... + ``` + +#### Collect logs using the Datadog Agent + +If you collect logs directly with the Datadog Agent instead of sending them through the OpenTelemetry Collector, two things change: + +- **Attribute Name**: Correlation relies on the `dd.trace_id` attribute instead of the standard OTel `trace_id`. +- **Attribute Mapping**: The Datadog Agent does not automatically perform the same OTel-to-Datadog convention conversions as the Datadog Exporter. You may need to manually remap resource attributes (like `service.name`) to Datadog's standard tags in the log processing pipeline. ## View correlated data in Datadog From aac4af74b876b7bfb4984f63bef959934ee4e7d3 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Thu, 17 Jul 2025 10:52:51 -0600 Subject: [PATCH 12/24] Remove repetition of correlate in nav. --- config/_default/menus/main.en.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 212ee56596710..7af6f34155693 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -780,22 +780,22 @@ menu: identifier: otel_explore parent: opentelemetry_top_level weight: 7 - - name: Correlate Logs and Traces + - name: Logs and Traces url: /opentelemetry/correlate/logs_and_traces/ identifier: otel_logs parent: otel_explore weight: 701 - - name: Correlate Metrics and Traces + - name: Metrics and Traces url: /opentelemetry/correlate/metrics_and_traces/ identifier: otel_metrics_traces parent: otel_explore weight: 702 - - name: Correlate RUM and Traces + - name: RUM and Traces url: /opentelemetry/correlate/rum_and_traces/ identifier: otel_rum parent: otel_explore weight: 703 - - name: Correlate DBM and Traces + - name: DBM and Traces url: /opentelemetry/correlate/dbm_and_traces/ identifier: otel_dbm parent: otel_explore From 780c3938528a17c25ebfd9055108f9d629f58277 Mon Sep 17 00:00:00 2001 From: Brett Blue <84536271+brett0000FF@users.noreply.github.com> Date: Tue, 22 Jul 2025 10:19:46 -0600 Subject: [PATCH 13/24] Apply suggestions from code review Co-authored-by: Jade Guiton --- .../other_telemetry/connect_logs_and_traces/opentelemetry.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index 39e06d901b3f3..d0533aaccdabd 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -194,8 +194,8 @@ This approach is useful if you have a requirement to keep local log files for co parse_from: attributes.timestamp layout_type: epoch layout: ms - severity: - parse_from: attributes.level + severity: + parse_from: attributes.level service: pipelines: From 52bee7992582b7c61bae5cf32f2876eb74c78f97 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 22 Jul 2025 10:26:47 -0600 Subject: [PATCH 14/24] Remove redundant alert. --- .../other_telemetry/connect_logs_and_traces/opentelemetry.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index d0533aaccdabd..dc39d31cd96bd 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -37,8 +37,6 @@ To correlate OpenTelemetry traces and logs in Datadog, you must: - **Send Logs to Datadog**: Your logs, now enriched with trace context, must be collected and sent to Datadog. -
For logs collected directly by the Datadog Agent (instead of through the OpenTelemetry Collector), automatic correlation relies on the dd.trace_id attribute.
- #### 1. Inject trace context into your logs The recommended approach is to configure your logging library to automatically inject the standard OpenTelemetry `trace_id` and `span_id` attributes into your logs. The following examples show how to configure common logging libraries to do this. From 5ebe9328f27b8f7a97530293ff3249bcfd21ee46 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 22 Jul 2025 10:29:18 -0600 Subject: [PATCH 15/24] Remove misleading description of attributes. --- .../other_telemetry/connect_logs_and_traces/opentelemetry.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index dc39d31cd96bd..9c18bfd23d35d 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -33,9 +33,9 @@ Before you begin, ensure you have configured [unified service tagging][1]. This To correlate OpenTelemetry traces and logs in Datadog, you must: -- **Inject Trace Context**: Your application's logger must be configured to inject the `trace_id` and `span_id` of the active trace into your logs. +- **Inject Trace Context**: Your application's logger must be configured to enrich your logs with the `trace_id` and `span_id` from the active trace. The recommended approach is to use an OpenTelemetry-aware logging library or appender. These tools automatically capture the active trace context and embed the `trace_id` and `span_id` as top-level fields in your log records, which is the standard method for correlation. -- **Send Logs to Datadog**: Your logs, now enriched with trace context, must be collected and sent to Datadog. +- **Send Logs to Datadog**: Your logs, enriched with trace context, must be collected and sent to Datadog. #### 1. Inject trace context into your logs From aefb382ce57176c5be915489e0dabf78a516eae0 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 22 Jul 2025 10:33:16 -0600 Subject: [PATCH 16/24] Clarify how trace context is injected using bridges. --- .../other_telemetry/connect_logs_and_traces/opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index 9c18bfd23d35d..1378dabd40a9a 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -46,7 +46,7 @@ For complete, working applications, see the [Datadog OpenTelemetry Examples repo {{< tabs >}} {{% tab "Go" %}} -For Go applications using a structured logger such as `zap`, use a bridge such as `otelzap`. This wraps your logger and automatically injects the active `trace_id` and `span_id` into every log message. +The following examples for Go and Java use logging bridges. These bridges intercept logs from common logging libraries (suchg as `zap` and `Logback`), convert them into the OpenTelemetry log data model, and forward them to the OpenTelemetry SDK. This process automatically enriches the logs with the active trace context. First, ensure you have an initialized OpenTelemetry `LoggerProvider`. Then, use it to create your `zap` logger instance: From fe0405e393ab7d9c802da55d0e426e61474b32d6 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 22 Jul 2025 10:36:12 -0600 Subject: [PATCH 17/24] Remove level param from LoggingHandler example. --- .../connect_logs_and_traces/opentelemetry.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index 1378dabd40a9a..e60c954cec1dd 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -99,7 +99,9 @@ Once your logs are instrumented with trace context, you need to send them to Dat #### Send logs using OTLP -This is the simplest and most direct method. Your application sends logs to the Collector over the network, avoiding the complexity of writing to and parsing local files. +This is the simplest and most direct method. Your application sends logs directly to an OTLP endpoint, avoiding the complexity of writing to and parsing local files. + +Both the Datadog Agent or the OpenTelemetry Collector can receive OTLP logs. 1. **Configure your Application to Export Logs using OTLP**: In your OpenTelemetry SDK setup, configure a `LogRecordProcessor` to use an `OTLPLogExporter`. The following example shows how to do this in Python: ```python @@ -118,7 +120,7 @@ This is the simplest and most direct method. Your application sends logs to the log_provider.add_log_record_processor(BatchLogRecordProcessor(exporter)) # Attach to the root logger - handler = LoggingHandler(level=logging.INFO, logger_provider=log_provider) + handler = LoggingHandler(logger_provider=log_provider) logging.getLogger().addHandler(handler) ``` 2. **Configure the Collector to Receive OTLP Logs**: In your Collector's `config.yaml`, enable the `otlp` receiver and add it to your `logs` pipeline: From 0e504faada68dfa1c53976535de91dc4d1f9f4c8 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 22 Jul 2025 10:48:11 -0600 Subject: [PATCH 18/24] Revise 'Scrape logs from files' approach. --- .../connect_logs_and_traces/opentelemetry.md | 96 ++++++++++--------- 1 file changed, 49 insertions(+), 47 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index e60c954cec1dd..73e9c44fa62a7 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -147,63 +147,65 @@ Both the Datadog Agent or the OpenTelemetry Collector can receive OTLP logs. This approach is useful if you have a requirement to keep local log files for compliance or other tooling. -1. **Configure your Application to Output JSON Logs**: In your OpenTelemetry SDK setup, configure a `ConsoleLogRecordExporter` to format logs as JSON and write them to `stdout` or a file. +1. **Configure your Application to Output JSON Logs**: Use a standard logging library to write logs as JSON to a file or `stdout`. The following Python example uses the standard `logging` library. +2. **Manually Inject Trace Context**: In your application code, retrieve the current span context from the OpenTelemetry tracer and add the `trace_id` and `span_id` to your log records. Datadog requires the IDs to be formatted as hexadecimal strings. +Here is a Python example showing how to create a custom logging.Filter to automatically add the active IDs: + ```python - # In your OTel SDK setup for Python import logging - from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler - from opentelemetry.sdk._logs.export import ConsoleLogRecordExporter, SimpleLogRecordProcessor - from opentelemetry.sdk.extension.opentelemetry_log_formatter import JSONLogRecordFormatter + import sys + from opentelemetry import trace + from pythonjsonlogger import jsonlogger - # Configure the exporter to write JSON to the console (stdout) - formatter = JSONLogRecordFormatter() - exporter = ConsoleLogRecordExporter(formatter=formatter) + # 1. Create a filter to inject trace context + class TraceContextFilter(logging.Filter): + def filter(self, record): + span = trace.get_current_span() + if span.is_recording(): + span_context = span.get_span_context() + record.trace_id = f'{span_context.trace_id:032x}' + record.span_id = f'{span_context.span_id:016x}' + else: + record.trace_id = "0" + record.span_id = "0" + return True - log_provider = LoggerProvider() - log_provider.add_log_record_processor(SimpleLogRecordProcessor(exporter)) + # 2. Configure a JSON logger + logger = logging.getLogger("my-json-logger") + logger.setLevel(logging.DEBUG) - # Attach to the root logger - handler = LoggingHandler(level=logging.INFO, logger_provider=log_provider) - logging.getLogger().addHandler(handler) - ``` - -2. **Verify the Log Output**: After configuring your logging library, your JSON logs should contain `trace_id` and `span_id` attributes. Datadog uses these to link the log to the exact trace and span that was active when the log was generated. Here is an example of a properly formatted JSON log entry: - - ```json - { - "timestamp": 1720557413000, - "level": "INFO", - "message": "Processing user request", - "service": "my-service", - "env": "production", - "version": "1.2.3", - "trace_id": "8738839999436033394", - "span_id": "1006654203791334032" - } + # 3. Add the filter to the logger + logger.addFilter(TraceContextFilter()) + + handler = logging.StreamHandler(sys.stdout) + formatter = jsonlogger.JsonFormatter( + '%(asctime)s %(name)s %(levelname)s %(message)s %(trace_id)s %(span_id)s' + ) + handler.setFormatter(formatter) + logger.addHandler(handler) + + # Logs will now contain the trace_id and span_id + logger.info("Processing user request with trace context.") ``` 3. **Configure the Collector to Scrape Log Files**: In your Collector's `config.yaml`, enable the `filelog` receiver. Configure it to find your log files and parse them as JSON. ```yaml - receivers: - filelog: - include: [ /var/log/my-app/*.log ] # Path to your log files - operators: - - type: json_parser # Parse the log line as JSON - # Note: The `parse_from` keys below must exactly match the - # field names present in your JSON log output. - timestamp: - parse_from: attributes.timestamp - layout_type: epoch - layout: ms - severity: - parse_from: attributes.level - - service: - pipelines: - logs: - receivers: [filelog] - # ... + receivers: + filelog: + include: [ /var/log/my-app/*.log ] # Path to your log files + operators: + - type: json_parser + # The timestamp and severity fields should match your JSON output + timestamp: + parse_from: attributes.asctime + layout: '%Y-%m-%d %H:%M:%S,%f' + severity: + parse_from: attributes.levelname + # ... your logs pipeline ... ``` +This manual approach gives you full control over the log format, ensuring it is clean and easily parsable by the Collector or Datadog Agent. + + #### Collect logs using the Datadog Agent If you collect logs directly with the Datadog Agent instead of sending them through the OpenTelemetry Collector, two things change: From 0f3e55af0cffeba51fa3765574e13045da912b7c Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 22 Jul 2025 10:52:58 -0600 Subject: [PATCH 19/24] Edits. --- .../connect_logs_and_traces/opentelemetry.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index 73e9c44fa62a7..07624dd9f8c21 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -43,11 +43,11 @@ The recommended approach is to configure your logging library to automatically i For complete, working applications, see the [Datadog OpenTelemetry Examples repository][2]. +The following examples for Go and Java use logging bridges. These bridges intercept logs from common logging libraries (such as `zap` and `Logback`), convert them into the OpenTelemetry log data model, and forward them to the OpenTelemetry SDK. This process automatically enriches the logs with the active trace context. + {{< tabs >}} {{% tab "Go" %}} -The following examples for Go and Java use logging bridges. These bridges intercept logs from common logging libraries (suchg as `zap` and `Logback`), convert them into the OpenTelemetry log data model, and forward them to the OpenTelemetry SDK. This process automatically enriches the logs with the active trace context. - First, ensure you have an initialized OpenTelemetry `LoggerProvider`. Then, use it to create your `zap` logger instance: ```go @@ -101,7 +101,7 @@ Once your logs are instrumented with trace context, you need to send them to Dat This is the simplest and most direct method. Your application sends logs directly to an OTLP endpoint, avoiding the complexity of writing to and parsing local files. -Both the Datadog Agent or the OpenTelemetry Collector can receive OTLP logs. +The OpenTelemetry Collector and the Datadog Agent can both receive OTLP logs. 1. **Configure your Application to Export Logs using OTLP**: In your OpenTelemetry SDK setup, configure a `LogRecordProcessor` to use an `OTLPLogExporter`. The following example shows how to do this in Python: ```python @@ -205,7 +205,6 @@ Here is a Python example showing how to create a custom logging.Filter to automa This manual approach gives you full control over the log format, ensuring it is clean and easily parsable by the Collector or Datadog Agent. - #### Collect logs using the Datadog Agent If you collect logs directly with the Datadog Agent instead of sending them through the OpenTelemetry Collector, two things change: From 2f31c00f7a5b0cfaf70ba5c6e82c51dbb0ae5f05 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 22 Jul 2025 11:26:34 -0600 Subject: [PATCH 20/24] Apply final feedback and clean up. --- .../connect_logs_and_traces/opentelemetry.md | 50 +++++++++++-------- 1 file changed, 29 insertions(+), 21 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index 07624dd9f8c21..09c4a4a952b71 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -39,12 +39,10 @@ To correlate OpenTelemetry traces and logs in Datadog, you must: #### 1. Inject trace context into your logs -The recommended approach is to configure your logging library to automatically inject the standard OpenTelemetry `trace_id` and `span_id` attributes into your logs. The following examples show how to configure common logging libraries to do this. +The following examples for Go and Java use logging bridges. These bridges intercept logs from common logging libraries (such as `zap` and `Logback`), convert them into the OpenTelemetry log data model, and forward them to the OpenTelemetry SDK. This process automatically enriches the logs with the active trace context. For complete, working applications, see the [Datadog OpenTelemetry Examples repository][2]. -The following examples for Go and Java use logging bridges. These bridges intercept logs from common logging libraries (such as `zap` and `Logback`), convert them into the OpenTelemetry log data model, and forward them to the OpenTelemetry SDK. This process automatically enriches the logs with the active trace context. - {{< tabs >}} {{% tab "Go" %}} @@ -143,13 +141,18 @@ The OpenTelemetry Collector and the Datadog Agent can both receive OTLP logs. exporters: [datadog] ``` -#### Scrape logs from files +#### Scrape logs from files This approach is useful if you have a requirement to keep local log files for compliance or other tooling. +**Required fields for logs** +For Datadog to correlate your logs and traces, your JSON log files must contain specific fields formatted correctly: +- `trace_id`: The ID of the trace. It must be a 32-character lowercase hexadecimal string. +- `span_id`: The ID of the span. It must be a 16-character lowercase hexadecimal string. +The OpenTelemetry SDK typically provides these as integers, which must be formatted into hexadecimal strings without any 0x prefix. + 1. **Configure your Application to Output JSON Logs**: Use a standard logging library to write logs as JSON to a file or `stdout`. The following Python example uses the standard `logging` library. -2. **Manually Inject Trace Context**: In your application code, retrieve the current span context from the OpenTelemetry tracer and add the `trace_id` and `span_id` to your log records. Datadog requires the IDs to be formatted as hexadecimal strings. -Here is a Python example showing how to create a custom logging.Filter to automatically add the active IDs: +2. **Manually Inject Trace Context**: In your application code, retrieve the current span context and add the `trace_id` and `span_id` to your log records. The following Python example shows how to create a custom logging.Filter to do this automatically: ```python import logging @@ -187,30 +190,34 @@ Here is a Python example showing how to create a custom logging.Filter to automa # Logs will now contain the trace_id and span_id logger.info("Processing user request with trace context.") ``` + 3. **Configure the Collector to Scrape Log Files**: In your Collector's `config.yaml`, enable the `filelog` receiver. Configure it to find your log files and parse them as JSON. ```yaml - receivers: - filelog: - include: [ /var/log/my-app/*.log ] # Path to your log files - operators: - - type: json_parser - # The timestamp and severity fields should match your JSON output - timestamp: - parse_from: attributes.asctime - layout: '%Y-%m-%d %H:%M:%S,%f' - severity: - parse_from: attributes.levelname - # ... your logs pipeline ... + receivers: + filelog: + include: [ /var/log/my-app/*.log ] # Path to your log files + operators: + - type: json_parser + # The timestamp and severity fields should match your JSON output + timestamp: + parse_from: attributes.asctime + layout: '%Y-%m-%d %H:%M:%S,%f' + severity: + parse_from: attributes.levelname + # ... your logs pipeline ... ``` This manual approach gives you full control over the log format, ensuring it is clean and easily parsable by the Collector or Datadog Agent. #### Collect logs using the Datadog Agent -If you collect logs directly with the Datadog Agent instead of sending them through the OpenTelemetry Collector, two things change: +If you collect logs directly with the Datadog Agent (without sending them through the OpenTelemetry Collector), you must ensure the trace IDs in your logs use the Datadog format. + +- **Trace ID format**: The Datadog Agent requires the trace ID to be in the `dd.trace_id` field. + - If you are using **Datadog's tracing libraries** (like `dd-trace-py`), this is handled for you automatically. + - If you are generating logs with OpenTelemetry `trace_id` and `span_id` (as shown in the [file-scraping example](#scrape-logs-from-files)), you must use a [Log Processing Rule][5] in Datadog to remap your `trace_id` attribute to `dd.trace_id`. -- **Attribute Name**: Correlation relies on the `dd.trace_id` attribute instead of the standard OTel `trace_id`. -- **Attribute Mapping**: The Datadog Agent does not automatically perform the same OTel-to-Datadog convention conversions as the Datadog Exporter. You may need to manually remap resource attributes (like `service.name`) to Datadog's standard tags in the log processing pipeline. +- **Attribute Mapping**: The Datadog Agent does not automatically convert OTel resource attributes (for example, `service.name`) to Datadog's standard tags. You may need to manually remap these attributes in your log processing pipeline to maintain unified service tagging. ## View correlated data in Datadog @@ -243,3 +250,4 @@ Click **View Trace in APM** to pivot directly to the full APM trace associated w [2]: https://github.com/DataDog/opentelemetry-examples [3]: https://app.datadoghq.com/apm/traces [4]: https://app.datadoghq.com/logs +[5]: /logs/log_configuration/processors \ No newline at end of file From b5de195aac141e0d8f0f5cfb3071c8418b5a1b6b Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Tue, 22 Jul 2025 11:39:36 -0600 Subject: [PATCH 21/24] Fix spacing. --- .../other_telemetry/connect_logs_and_traces/opentelemetry.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index 09c4a4a952b71..0307afd1a1090 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -145,10 +145,10 @@ The OpenTelemetry Collector and the Datadog Agent can both receive OTLP logs. This approach is useful if you have a requirement to keep local log files for compliance or other tooling. -**Required fields for logs** For Datadog to correlate your logs and traces, your JSON log files must contain specific fields formatted correctly: - `trace_id`: The ID of the trace. It must be a 32-character lowercase hexadecimal string. -- `span_id`: The ID of the span. It must be a 16-character lowercase hexadecimal string. +- `span_id`: The ID of the span. It must be a 16-character lowercase hexadecimal string. + The OpenTelemetry SDK typically provides these as integers, which must be formatted into hexadecimal strings without any 0x prefix. 1. **Configure your Application to Output JSON Logs**: Use a standard logging library to write logs as JSON to a file or `stdout`. The following Python example uses the standard `logging` library. From e9bae958d81b314b74a09d20dcf7b28bfae169a2 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Mon, 28 Jul 2025 16:07:08 -0600 Subject: [PATCH 22/24] Address final nits. --- .../other_telemetry/connect_logs_and_traces/opentelemetry.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md index 0307afd1a1090..13250c6a9ca3d 100644 --- a/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md +++ b/content/en/tracing/other_telemetry/connect_logs_and_traces/opentelemetry.md @@ -149,7 +149,7 @@ For Datadog to correlate your logs and traces, your JSON log files must contain - `trace_id`: The ID of the trace. It must be a 32-character lowercase hexadecimal string. - `span_id`: The ID of the span. It must be a 16-character lowercase hexadecimal string. -The OpenTelemetry SDK typically provides these as integers, which must be formatted into hexadecimal strings without any 0x prefix. +The OpenTelemetry SDK typically provides these in a raw format (such as an integer or byte array), which must be formatted into hexadecimal strings without any 0x prefix. 1. **Configure your Application to Output JSON Logs**: Use a standard logging library to write logs as JSON to a file or `stdout`. The following Python example uses the standard `logging` library. 2. **Manually Inject Trace Context**: In your application code, retrieve the current span context and add the `trace_id` and `span_id` to your log records. The following Python example shows how to create a custom logging.Filter to do this automatically: @@ -168,9 +168,6 @@ The OpenTelemetry SDK typically provides these as integers, which must be format span_context = span.get_span_context() record.trace_id = f'{span_context.trace_id:032x}' record.span_id = f'{span_context.span_id:016x}' - else: - record.trace_id = "0" - record.span_id = "0" return True # 2. Configure a JSON logger From 8e7da6b65f07cabfbd353dc7b372619cb1388830 Mon Sep 17 00:00:00 2001 From: Brett Blue Date: Thu, 31 Jul 2025 10:02:06 -0600 Subject: [PATCH 23/24] Remove DBM content; move to new branch. --- config/_default/menus/main.en.yaml | 5 - content/en/opentelemetry/correlate/_index.md | 4 +- .../opentelemetry/correlate/dbm_and_traces.md | 111 ------------------ 3 files changed, 1 insertion(+), 119 deletions(-) delete mode 100644 content/en/opentelemetry/correlate/dbm_and_traces.md diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 88f89db37edca..9e89ee4d15e2f 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -800,11 +800,6 @@ menu: identifier: otel_rum parent: otel_explore weight: 703 - - name: DBM and Traces - url: /opentelemetry/correlate/dbm_and_traces/ - identifier: otel_dbm - parent: otel_explore - weight: 704 - name: Integrations url: opentelemetry/integrations/ identifier: otel_integrations diff --git a/content/en/opentelemetry/correlate/_index.md b/content/en/opentelemetry/correlate/_index.md index a4d4e9b4e8a7d..c72d82043f7dc 100644 --- a/content/en/opentelemetry/correlate/_index.md +++ b/content/en/opentelemetry/correlate/_index.md @@ -89,7 +89,6 @@ After unified service tagging is configured, you can connect your various teleme - [Correlate logs and traces][1] - [Correlate metrics and traces][2] - [Correlate RUM and traces][3] -- [Correlate DBM and traces][4] ## Further reading @@ -97,5 +96,4 @@ After unified service tagging is configured, you can connect your various teleme [1]: /opentelemetry/correlate/logs_and_traces [2]: /opentelemetry/correlate/metrics_and_traces -[3]: /opentelemetry/correlate/rum_and_traces -[4]: /opentelemetry/correlate/dbm_and_traces \ No newline at end of file +[3]: /opentelemetry/correlate/rum_and_traces \ No newline at end of file diff --git a/content/en/opentelemetry/correlate/dbm_and_traces.md b/content/en/opentelemetry/correlate/dbm_and_traces.md deleted file mode 100644 index 3e8782ce04eb8..0000000000000 --- a/content/en/opentelemetry/correlate/dbm_and_traces.md +++ /dev/null @@ -1,111 +0,0 @@ ---- -title: Correlate OpenTelemetry Traces and DBM -further_reading: -- link: "/opentelemetry/otel_tracing/" - tag: "Documentation" - text: "Send OpenTelemetry Traces to Datadog" ---- - -## Overview - -Correlate backend traces to detailed database performance data in Datadog Database Monitoring (DBM). This allows you to link spans from your OpenTelemetry-instrumented application directly to query metrics and execution plans to identify the exact queries that are slowing down your application. - -## Requirements - -Before you begin, ensure you have configured [unified service tagging][1]. This is required for all data correlation in Datadog. - -## Setup - -To correlate traces and metrics, you must: - -1. **Instrument database spans**: Add specific OpenTelemetry attributes to your database spans to enable correlation with DBM. - -2. **Configure trace ingestion path**: Enable the correct feature gate on your Collector or Agent to ensure database spans are properly processed for DBM. - -### Step 1: Instrument your database spans - -For DBM correlation to work, your database spans must include the following attributes. - -| Attribute | Description | Example | -|----------------|-----------------------------------------------------------------------------------------------------|------------------------------------| -| `db.system` | **Required.** The database technology, such as `postgres`, `mysql`, or `sqlserver`. | `postgres` | -| `db.statement` | **Required.** The raw SQL query text. This is used for obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` | -| `db.name` | The logical database or schema name being queried. | `user_accounts` | -| `span.type` | **Required (Datadog-specific).** The type of span such as `sql`,`postgres`, `mysql`, or `sql.query` | `sql` | - -#### Example - -The method for adding these attributes depends on your setup. If you are using an OpenTelemetry auto-instrumentation library for your database client, see its documentation for configuration options. If you are manually creating spans with the OpenTelemetry SDK, you can set the attributes directly in your code. For more information, see the [OpenTelemetry documentation][4]. - -The following is a conceptual example of manual instrumentation using Python's OpenTelemetry SDK: - -```python -from opentelemetry import trace - -tracer = trace.get_tracer("my-app.instrumentation") - -# When making a database call, create a span and set attributes -with tracer.start_as_current_span("postgres.query") as span: - # Set attributes required for DBM correlation - span.set_attribute("span.type", "sql") - span.set_attribute("db.system", "postgres") - span.set_attribute("db.statement", "SELECT * FROM users WHERE id = ?") - span.set_attribute("db.name", "user_accounts") - - # Your actual database call would go here - # db_cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,)) -``` - -### Step 2: Configure your ingest path - -Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. - -{{< tabs >}} -{{% tab "Datadog Agent (DDOT Collector)" %}} - - -If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`: - -```yaml -datadog: - otelCollector: - featureGates: datadog.EnableOperationAndResourceNameV2 -``` - -{{% /tab %}} -{{% tab "OTel Collector" %}} - -When starting the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate. This is available in Collector v0.118.0 and later. - -```sh -otelcontribcol --config=config.yaml \ ---feature-gates=datadog.EnableOperationAndResourceNameV2 -``` - -{{% /tab %}} - -{{% tab "Datadog Agent (OTLP Ingest)" %}} - -In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`. - -{{% /tab %}} - -{{< /tabs >}} - -### View correlated data in Datadog - -After your application is sending traces, you can see the correlation in the APM Trace View: - -1. Navigate to [**APM** > **Traces**][3]. -2. Find and click on a trace from your instrumented service. -3. In the trace's flame graph, select a database span (for example, a span with `span.type: sql`) -4. In the details panel, click the **SQL Queries** tab. You should see the host metrics, like CPU and memory utilization, from the host that executed that part of the request. - -## Further reading - -{{< partial name="whats-next/whats-next.html" >}} - -[1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging -[2]: /opentelemetry/integrations/host_metrics -[3]: https://app.datadoghq.com/apm/traces -[4]: https://opentelemetry.io/docs/languages/ \ No newline at end of file From f795a6aa9d0dbe4cfd5551c79b2c5bd9fa499b29 Mon Sep 17 00:00:00 2001 From: Brett Blue <84536271+brett0000FF@users.noreply.github.com> Date: Thu, 7 Aug 2025 16:21:06 -0600 Subject: [PATCH 24/24] Update content/en/opentelemetry/correlate/_index.md Co-authored-by: Heston Hoffman --- content/en/opentelemetry/correlate/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/opentelemetry/correlate/_index.md b/content/en/opentelemetry/correlate/_index.md index c72d82043f7dc..2977c9221fdcc 100644 --- a/content/en/opentelemetry/correlate/_index.md +++ b/content/en/opentelemetry/correlate/_index.md @@ -23,7 +23,7 @@ To ensure your OpenTelemetry data is properly correlated, you must configure you | OpenTelemetry Resource Attribute | Datadog Tag | Notes | |----------------------------------|-------------|---------------------------------------------------------------------------------------------------------| | `deployment.environment.name` | `env` | **Recommended**. Supported in Agent v7.58.0+ and Collector Exporter v0.110.0+. | -| `deployment.environment` | `env` | Use if you are running an Agent version older than v7.58.0 or a Collector Exporter older than v0.110.0. | +| `deployment.environment` | `env` | Use instead of `deployment.environment.name` if you are running an Agent version older than v7.58.0 or a Collector Exporter older than v0.110.0. | | `service.name` | `service` | | | `service.version` | `version` | |