Best Practice on tracing long running tasks

[nicholas-1vbw]

We are using tracing to analyze a long-running task.

In a logging-based approach, we can immediately observe messages from tracing::info/warn within a span, and these logs allow us to trigger monitoring alerts when processing times exceed expectations.

However, in OpenTelemetry, messages (recorded by tracing::info/warn) is only exported when the span finishes. We are unable to implement the monitoring and alerting functionality in this scenario. Even worse, in certain cases, the process might panic or abort, resulting in some trace and log information failing to be uploaded to the collector. (Each message is a single span, most logs should be ok to export before panic)

Some related discussion:
#50
open-telemetry/opentelemetry-specification#3732

[mladedav]

This is how opentelemetry is set up, there's really no way around that if you want to just use tracing.

What we do in production is that we set up two tracing subscribers, one for logs that are exported right when they're generated and another for opentelemetry traces. You can have alerts on logs and then more contextual information on spans. But you still might want to not produce too long spans to be able to leverage the information when you need it and not just hours after the fact.

[VCasecnikovs]

So, the correct answer would be to setup:

1. tracing-opentelemetry to have context in Spans
2. opentelemetry-appender-tracing to add those events as Logs