From b72ae62326c9ba0712e81b10915698e3c0368e63 Mon Sep 17 00:00:00 2001 From: Steve Gordon Date: Tue, 5 Aug 2025 15:36:36 +0100 Subject: [PATCH 1/7] Add troubleshooting for missing transactions --- .../apm-agent-dotnet/apm-net-agent.md | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md index 6a81c8b64e..573e52c540 100644 --- a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md +++ b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md @@ -163,6 +163,37 @@ Agent.Setup(new AgentComponents(logger: new ApmLoggerAdapter())); ``` +## Missing transactions + +If you receive trace data for some, but not all transactions, two main variables control +sampling of transactions which should be checked. + +By default, the agent samples every transaction (e.g. a request to your service). To reduce volume, +the [transaction sample rate](apm-agent-dotnet://reference/config-core.md#config-transaction-sample-rate) configuration +option may have been configured with a reduced sampling rate. Check that this is set to either 1, or +not set at all if you plan to sample all transactions. + +The second variable is context propagation. To facilitate distributed tracing, the W3C specification +defines a [trace context](https://www.w3.org/TR/trace-context/), used to transfer a trace across service boundaries. +For web applications, this occurs through the traceparent header. Of specific importance for this issue is +a flag defined on that context, the [sampled flag](https://www.w3.org/TR/trace-context/#sampled-flag). + +This flag allows a sampling decision made earlier in a distributed trace to continue onto downstream services, +and the .NET APM agent honours this automatically. When a request comes in with the sampling flag +unset (not sampled), the agent will not record a trace in the instrumented service either. This is +by design as incomplete distributed traces are normally not desired. + +When all services within a distributed system are managed and instrumented using Elastic APM agents and communication +is internal, the default behaviour is usually correct. If some requests to instrumented applications may come from +external users, then the `traceparent` header cannot be controlled. In these cases, it's often incorrect to allow +an outside party to control what transactions are recorded through the sampled flag. + +To control the context propagation behaviour, we have a [trace continuation strategy](apm-agent-dotnet://reference/config-http.md#config-trace-continuation-strategy) option. Choose either `restart` or `restart_external`. `restart` will ignore +the sampling flag on all requests and always include a transaction for the trace. `restart_external` is similar, +however, we use an extra header to detect if the calling service was instrumented by an Elastic APM agent. If so, +we honour the sampling flag; if not (i.e. an external request), we force the trace to be sampled. + + ## Following error appears in logs: `The singleton APM agent has already been instantiated and can no longer be configured.` [double-agent-initialization-log] See "[An `InstanceAlreadyCreatedException` exception is thrown](#double-agent-initialization)". From 73bec5b344a7cf64ade79617fbb9227e9808d24a Mon Sep 17 00:00:00 2001 From: Steve Gordon Date: Thu, 7 Aug 2025 15:01:16 +0100 Subject: [PATCH 2/7] Update troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> --- troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md index 573e52c540..ebd699c846 100644 --- a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md +++ b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md @@ -165,8 +165,8 @@ Agent.Setup(new AgentComponents(logger: new ApmLoggerAdapter())); ## Missing transactions -If you receive trace data for some, but not all transactions, two main variables control -sampling of transactions which should be checked. +If you receive trace data for some but not all transactions, you should check the two main variables which control +sampling of transactions. By default, the agent samples every transaction (e.g. a request to your service). To reduce volume, the [transaction sample rate](apm-agent-dotnet://reference/config-core.md#config-transaction-sample-rate) configuration From c16fce4033a07845e711faac9515d8df68617808 Mon Sep 17 00:00:00 2001 From: Steve Gordon Date: Thu, 7 Aug 2025 15:02:05 +0100 Subject: [PATCH 3/7] Update troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> --- troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md index ebd699c846..d0d57dc347 100644 --- a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md +++ b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md @@ -168,10 +168,7 @@ Agent.Setup(new AgentComponents(logger: new ApmLoggerAdapter())); If you receive trace data for some but not all transactions, you should check the two main variables which control sampling of transactions. -By default, the agent samples every transaction (e.g. a request to your service). To reduce volume, -the [transaction sample rate](apm-agent-dotnet://reference/config-core.md#config-transaction-sample-rate) configuration -option may have been configured with a reduced sampling rate. Check that this is set to either 1, or -not set at all if you plan to sample all transactions. +The first variable is the [transaction sample rate](apm-agent-dotnet://reference/config-core.md#config-transaction-sample-rate). To reduce volume, this configuration option may specify a reduced sampling rate. To sample all transactions, ensure that this is either set to `1`, or not set at all. By default, the agent samples every transaction (every request to your service). ``` The second variable is context propagation. To facilitate distributed tracing, the W3C specification defines a [trace context](https://www.w3.org/TR/trace-context/), used to transfer a trace across service boundaries. From e1ace6b7f016846e491bdde3a165eebcd02b70c6 Mon Sep 17 00:00:00 2001 From: Steve Gordon Date: Thu, 7 Aug 2025 15:02:28 +0100 Subject: [PATCH 4/7] Update troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> --- troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md index d0d57dc347..c6dfd99cac 100644 --- a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md +++ b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md @@ -172,7 +172,7 @@ The first variable is the [transaction sample rate](apm-agent-dotnet://reference The second variable is context propagation. To facilitate distributed tracing, the W3C specification defines a [trace context](https://www.w3.org/TR/trace-context/), used to transfer a trace across service boundaries. -For web applications, this occurs through the traceparent header. Of specific importance for this issue is +For web applications, this occurs through the `traceparent` header. Of specific importance for this issue is a flag defined on that context, the [sampled flag](https://www.w3.org/TR/trace-context/#sampled-flag). This flag allows a sampling decision made earlier in a distributed trace to continue onto downstream services, From 531d2cc6465be3c3a7b752067ecd9e6a0d8b8a01 Mon Sep 17 00:00:00 2001 From: Steve Gordon Date: Thu, 7 Aug 2025 15:02:50 +0100 Subject: [PATCH 5/7] Update troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> --- troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md index c6dfd99cac..fc99c9171c 100644 --- a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md +++ b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md @@ -178,7 +178,7 @@ a flag defined on that context, the [sampled flag](https://www.w3.org/TR/trace-c This flag allows a sampling decision made earlier in a distributed trace to continue onto downstream services, and the .NET APM agent honours this automatically. When a request comes in with the sampling flag unset (not sampled), the agent will not record a trace in the instrumented service either. This is -by design as incomplete distributed traces are normally not desired. +by design to avoid collecting incomplete traces. When all services within a distributed system are managed and instrumented using Elastic APM agents and communication is internal, the default behaviour is usually correct. If some requests to instrumented applications may come from From ff1b5e406ee3a126060ce84b13d0b908ea7e7860 Mon Sep 17 00:00:00 2001 From: Steve Gordon Date: Thu, 7 Aug 2025 15:02:58 +0100 Subject: [PATCH 6/7] Update troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> --- troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md index fc99c9171c..2413c492b8 100644 --- a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md +++ b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md @@ -187,7 +187,7 @@ an outside party to control what transactions are recorded through the sampled f To control the context propagation behaviour, we have a [trace continuation strategy](apm-agent-dotnet://reference/config-http.md#config-trace-continuation-strategy) option. Choose either `restart` or `restart_external`. `restart` will ignore the sampling flag on all requests and always include a transaction for the trace. `restart_external` is similar, -however, we use an extra header to detect if the calling service was instrumented by an Elastic APM agent. If so, +however it uses an extra header to detect if the calling service was instrumented by an Elastic APM agent. If so, we honour the sampling flag; if not (i.e. an external request), we force the trace to be sampled. From 77003d774ccda1d7644b63f943f78ddd6db4e5e7 Mon Sep 17 00:00:00 2001 From: Steve Gordon Date: Thu, 7 Aug 2025 15:03:10 +0100 Subject: [PATCH 7/7] Update troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> --- troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md index 2413c492b8..c6e46cf7a3 100644 --- a/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md +++ b/troubleshoot/observability/apm-agent-dotnet/apm-net-agent.md @@ -188,7 +188,7 @@ an outside party to control what transactions are recorded through the sampled f To control the context propagation behaviour, we have a [trace continuation strategy](apm-agent-dotnet://reference/config-http.md#config-trace-continuation-strategy) option. Choose either `restart` or `restart_external`. `restart` will ignore the sampling flag on all requests and always include a transaction for the trace. `restart_external` is similar, however it uses an extra header to detect if the calling service was instrumented by an Elastic APM agent. If so, -we honour the sampling flag; if not (i.e. an external request), we force the trace to be sampled. +it honors the sampling flag; if not (such as for an external request), it forces the trace to be sampled. ## Following error appears in logs: `The singleton APM agent has already been instantiated and can no longer be configured.` [double-agent-initialization-log]