Skip to content

telemetry.md

Latest commit

 

History

History
198 lines (154 loc) · 7.74 KB

telemetry.md

File metadata and controls

198 lines (154 loc) · 7.74 KB
Raw

OpenTelemetry Telemetry

The 11 BaSyx commands that expose HTTP APIs support optional OpenTelemetry tracing and PostgreSQL connection pool metrics. Telemetry is configured only with standard OpenTelemetry environment variables; it does not add a BaSyx YAML section.

basyxconfigurationservice and historyevidenceverifier do not expose HTTP servers or traced operations, so they remain logging-only commands.

Activation

Tracing is disabled when OTEL_TRACES_EXPORTER is unset, empty, or none. Metrics are independently disabled when OTEL_METRICS_EXPORTER is unset, empty, or none. OTEL_SDK_DISABLED=true disables both signals regardless of their exporters.

Use an OTLP Collector in deployed environments:

OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

The console exporter is available independently for local diagnostics:

OTEL_TRACES_EXPORTER=console
OTEL_METRICS_EXPORTER=console

Unsupported explicit values stop startup with an OTEL-CONFIG-* error. A Collector outage after configuration does not stop the service or fail HTTP requests. Export and shutdown failures are emitted as structured warnings without logging configured endpoints or OTLP headers.

Standard Environment Variables

BaSyx supports these standard settings provided by the OpenTelemetry Go SDK and auto-exporter:

  • OTEL_TRACES_EXPORTER: otlp, console, or none
  • OTEL_METRICS_EXPORTER: otlp, console, or none
  • OTEL_SDK_DISABLED
  • OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
  • OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
  • OTEL_EXPORTER_OTLP_METRICS_PROTOCOL
  • OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TRACES_HEADERS
  • OTEL_EXPORTER_OTLP_METRICS_HEADERS
  • OTEL_EXPORTER_OTLP_COMPRESSION and OTEL_EXPORTER_OTLP_TRACES_COMPRESSION
  • OTEL_EXPORTER_OTLP_METRICS_COMPRESSION
  • OTEL_EXPORTER_OTLP_TIMEOUT and OTEL_EXPORTER_OTLP_TRACES_TIMEOUT
  • OTEL_EXPORTER_OTLP_METRICS_TIMEOUT
  • OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES
  • OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG
  • OTEL_BSP_SCHEDULE_DELAY, OTEL_BSP_EXPORT_TIMEOUT, OTEL_BSP_MAX_QUEUE_SIZE, and OTEL_BSP_MAX_EXPORT_BATCH_SIZE
  • OTEL_METRIC_EXPORT_INTERVAL and OTEL_METRIC_EXPORT_TIMEOUT
  • OTEL_PROPAGATORS

The default telemetry resource service.name is the lowercase command directory name, such as aasenvironmentservice. OTEL_SERVICE_NAME overrides it. BaSyx does not invent a service.version; operators can provide one through OTEL_RESOURCE_ATTRIBUTES.

The default propagators are W3C Trace Context and Baggage. Supported OTEL_PROPAGATORS values are tracecontext, baggage, and none.

See the OpenTelemetry environment variable specification and Go auto-exporter documentation for the detailed value formats.

PostgreSQL Pool Metrics

Each service registers its single shared PostgreSQL writer pool once. The metrics read database/sql.DBStats() during collection and do not execute database queries.

Metric Type DBStats source
db.client.connection.max up/down counter MaxOpenConnections
db.client.connection.count with state=used up/down counter InUse
db.client.connection.count with state=idle up/down counter Idle
basyx.db.client.connection.waits cumulative counter WaitCount
basyx.db.client.connection.wait_time cumulative counter in seconds WaitDuration
basyx.db.client.connection.closed with reason=idle_limit cumulative counter MaxIdleClosed
basyx.db.client.connection.closed with reason=idle_time cumulative counter MaxIdleTimeClosed
basyx.db.client.connection.closed with reason=max_lifetime cumulative counter MaxLifetimeClosed

Open connections are the sum of the used and idle db.client.connection.count points. Every point has db.system.name=postgresql and db.client.connection.pool.name=writer. The service.name resource attribute identifies the BaSyx service. These values are bounded; database names, hosts, users, DSNs, SQL text, AAS identifiers, and request data are not metric attributes.

Use rates for the cumulative wait and closure counters. A rising wait rate while used approaches max indicates that the service pool is constraining concurrency. Low pool utilization with slow requests points elsewhere, such as PostgreSQL query execution, server capacity, or storage latency.

Sampling and Lifecycle

The default sampler is parent-based and always-on. For sustained or high-volume traffic, configure an appropriate sampler instead of tracing every request. For example:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

Completed spans are exported in batches. Metrics use the SDK export interval and timeout. When a service stops, BaSyx flushes and shuts down the enabled trace and metric providers with fresh bounded contexts after HTTP shutdown has completed.

HTTP Spans and Propagation

Every HTTP request creates one SERVER span when tracing is enabled. A valid inbound W3C traceparent and tracestate is extracted; otherwise BaSyx creates a new trace. Matched spans are named <METHOD> <route>, while unmatched routes use <METHOD>.

Server span attributes include:

  • http.request.method
  • url.path
  • http.route when Chi resolves a route
  • http.response.status_code
  • http.response.body.size
  • request.id
  • correlation.id

5xx responses and panics mark the span as an error. Panic spans contain only a generic marker, not the panic value.

Delegated submodel-operation calls create CLIENT spans and inject the active W3C trace context. This instrumentation wraps the existing guarded transport; its host allowlisting, DNS/IP checks, proxy restrictions, timeout, pinned dialing, redirect checks, and authorization behavior remain unchanged. Requests without an active span do not receive fabricated trace headers.

This release does not propagate trace context to OIDC endpoints or arbitrary third-party HTTP clients.

Privacy

Trace spans never capture query strings, request or response bodies, authorization values, arbitrary headers, tokens, user agents, or client IP addresses. Delegated client spans contain only the method, scheme, hostname and port, path, response status, and a generic error marker. PostgreSQL metrics contain only the bounded attributes documented above.

Log Correlation

Every slog event emitted with a valid active span context receives these top-level fields:

  • trace_id
  • span_id
  • trace_flags

The IDs use lowercase hexadecimal values. trace_flags is 01 for sampled contexts and 00 for unsampled contexts. Background logs remain unchanged. The HTTP request completed access event is written before the server span ends, so its identifiers match the span.

Logs still go to stderr. BaSyx does not export logs through OTLP or contain direct clients for Jaeger, Loki, Tempo, or Grafana.

Local Example

The BaSyx observability example builds the BaSyx images from the current checkout and connects:

BaSyx --OTLP/HTTP--> Collector --OTLP/gRPC--> Jaeger
BaSyx --JSON stderr--> Alloy --Loki API--> Loki
Grafana --> Jaeger and Loki

Grafana is provisioned with links from log trace_id values to Jaeger and from Jaeger traces back to Loki.

Scope

This telemetry foundation does not add OTLP log export, database query spans, SQL statement capture, AAS-domain spans, runtime reconfiguration, or custom exporter plugins.