diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 8b0b8bb2bfd33..5697bcbed1d7d 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -750,6 +750,11 @@ menu: identifier: otel_rum parent: otel_explore weight: 703 + - name: DBM and Traces + url: /opentelemetry/correlate/dbm_and_traces/ + identifier: otel_dbm + parent: otel_explore + weight: 704 - name: Integrations url: opentelemetry/integrations/ identifier: otel_integrations diff --git a/content/en/opentelemetry/correlate/_index.md b/content/en/opentelemetry/correlate/_index.md index 2977c9221fdcc..0647809cb3746 100644 --- a/content/en/opentelemetry/correlate/_index.md +++ b/content/en/opentelemetry/correlate/_index.md @@ -89,6 +89,7 @@ After unified service tagging is configured, you can connect your various teleme - [Correlate logs and traces][1] - [Correlate metrics and traces][2] - [Correlate RUM and traces][3] +- [Correlate DBM and traces][4] ## Further reading @@ -96,4 +97,5 @@ After unified service tagging is configured, you can connect your various teleme [1]: /opentelemetry/correlate/logs_and_traces [2]: /opentelemetry/correlate/metrics_and_traces -[3]: /opentelemetry/correlate/rum_and_traces \ No newline at end of file +[3]: /opentelemetry/correlate/rum_and_traces +[4]: /opentelemetry/correlate/dbm_and_traces diff --git a/content/en/opentelemetry/correlate/dbm_and_traces.md b/content/en/opentelemetry/correlate/dbm_and_traces.md new file mode 100644 index 0000000000000..7d3f5e2b43501 --- /dev/null +++ b/content/en/opentelemetry/correlate/dbm_and_traces.md @@ -0,0 +1,147 @@ +--- +title: Correlate OpenTelemetry Traces and DBM +further_reading: +- link: "/opentelemetry/otel_tracing/" + tag: "Documentation" + text: "Send OpenTelemetry Traces to Datadog" +--- + +## Overview + +Correlate backend traces to detailed database performance data in Datadog Database Monitoring (DBM). This allows you to link spans from your OpenTelemetry-instrumented application to related query metrics and execution plans to identify the exact queries that are slowing down your application. + +## Requirements + +Before you begin, ensure you have configured [unified service tagging][1]. This is required for all data correlation in Datadog. + +## Setup + +To correlate traces and metrics, you must: + +1. **Instrument database spans**: Add specific OpenTelemetry attributes to your database spans to enable correlation with DBM. + +2. **Configure trace ingestion path**: Enable the correct feature gate on your Collector or Agent to ensure database spans are properly processed for DBM. + +### Step 1: Instrument your database spans + +For DBM correlation to work, your database spans must include the following attributes. + +| Attribute | Description | Example | +|----------------|-----------------------------------------------------------------------------------------------------|------------------------------------| +| `db.system` | **Required.** The database technology, such as `postgres`, `mysql`, or `sqlserver`. | `postgres` | +| `db.statement` | **Required.** The raw SQL query text. This is used for obfuscation and normalization. | `SELECT * FROM users WHERE id = ?` | +| `db.name` | The logical database or schema name being queried. | `user_accounts` | +| `span.type` | **Required (Datadog-specific).** The type of span such as `sql`,`postgres`, `mysql`, or `sql.query` | `sql` | + +
The span.type attribute is a Datadog-specific convention required for the backend to identify and process database spans. It is not part of the standard OpenTelemetry semantic conventions.
+ +#### Auto instrumentation + +If you are using an OpenTelemetry auto-instrumentation library, you can add required attributes without changing your application code. Most OpenTelemetry auto-instrumentation libraries already add `db.system` and `db.statement`. For DBM correlation, you typically only need to add the Datadog-specific `span.type` attribute. You can do this by using the OpenTelemetry Collector's `attributes` processor to enrich your spans. + +For example, you can add `span.type: sql` to any span that has the `db.system` attribute: + +```yaml +processors: + attributes/add_span_type: + actions: + - key: span.type + value: "sql" + action: insert + # Apply this action only to spans that have the db.system attribute + from_context: span + when: + - span.attributes["db.system"] != nil + +service: + pipelines: + traces: + # Add the processor to your traces pipeline + processors: [..., attributes/add_span_type, ...] +``` + +#### Manual instrumentation + +If you are manually creating spans with the OpenTelemetry SDK, you can set the attributes directly in your code. For more information, see the [OpenTelemetry documentation][4]. + +The following is a conceptual example of manual instrumentation using Python's OpenTelemetry SDK: + +```python +from opentelemetry import trace + +tracer = trace.get_tracer("my-app.instrumentation") + +# When making a database call, create a span and set attributes +with tracer.start_as_current_span("postgres.query") as span: + # Set attributes required for DBM correlation + span.set_attribute("span.type", "sql") + span.set_attribute("db.system", "postgres") + span.set_attribute("db.statement", "SELECT * FROM users WHERE id = ?") + span.set_attribute("db.name", "user_accounts") + + # Your actual database call would go here + # db_cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,)) +``` + +### Step 2: Configure your ingest path + +Depending on how you send traces to Datadog, you may need to enable specific feature gates to ensure database spans are processed correctly. + +{{< tabs >}} +{{% tab "Datadog Agent (DDOT Collector)" %}} + + +If you are using the Datadog Helm chart (v3.107.0 or later), set the feature gate in your `values.yaml`: + +```yaml +datadog: + otelCollector: + featureGates: datadog.EnableOperationAndResourceNameV2 +``` + +{{% /tab %}} +{{% tab "OTel Collector" %}} + +When starting the Collector, enable the `datadog.EnableOperationAndResourceNameV2` feature gate. This is available in Collector v0.118.0 and later. + +```sh +otelcontribcol --config=config.yaml \ +--feature-gates=datadog.EnableOperationAndResourceNameV2 +``` + +{{% /tab %}} + +{{% tab "Datadog Agent (OTLP Ingest)" %}} + +In your Datadog Agent configuration, ensure the `DD_APM_FEATURES` environment variable includes `enable_operation_and_resource_name_logic_v2`. + +{{% /tab %}} + +{{< /tabs >}} + +### View correlated data in Datadog + +After your application is sending traces, you can see the correlation in the APM Trace View: + +1. Navigate to [**APM** > **Traces**][3]. +2. Find and click on a trace from your instrumented service. +3. In the trace's flame graph, select a database span (for example, a span with `span.type: sql`) +4. In the details panel, click the **SQL Queries** tab. You should see performance metrics and execution plans for the query. + +## Troubleshooting + +If you don't see the expected correlation between your APM traces and DBM, it's typically due to a missing or incorrect configuration. Check the following common causes: + +- **All required attributes (`db.system`, `db.statement`, `span.type`) must be present** on the database span. +- **The SQL query may not be parsable**: The correlation relies on Datadog's ability to parse the SQL query from the `db.statement` attribute. If the query uses non-standard or complex syntax, parsing may fail. If you suspect this is the case, [contact Datadog support][5] for assistance. +- **The correct feature gates must be enabled** for your specific trace ingestion path as described in the setup steps. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /opentelemetry/correlate/#prerequisite-unified-service-tagging +[2]: /opentelemetry/integrations/host_metrics +[3]: https://app.datadoghq.com/apm/traces +[4]: https://opentelemetry.io/docs/languages/ +[5]: /help \ No newline at end of file