Peer Service Name via Code

[mikelly-psat]

Is there a way we can customize the Peer Service Name via code, or scoped to a particular portion of code?

https://opentelemetry.io/docs/zero-code/java/agent/instrumentation/#peer-service-name

That won't be a good fit for my use case.

We have services that need to communicate to customer servers, so this causes a lot of different services to be inferred in our infra, and we'd rather cut down on that noise, e..g just set the service.peer.name to customer-idp or whatever instead of it being inferred from their hostnames.

Ideally we'd just annotate our code in some way, or just wrap the HTTP client calls in a custom span or "something". So far, main thing I've tried is making a custom client span around the HTTP client using the SDK, but none of the span suppression strategies I've tried seem to have any effect. Roughtly:

        Span span = tracer.spanBuilder("GET Customer Identity")
                .setSpanKind(SpanKind.CLIENT)
                .setAttribute(PeerIncubatingAttributes.PEER_SERVICE, "customer-idp")
                .setAttribute(ServiceIncubatingAttributes.SERVICE_PEER_NAME, "customer-idp")
                .setAttribute(HttpAttributes.HTTP_REQUEST_METHOD, HttpAttributes.HttpRequestMethodValues.GET)
                .startSpan();
        try (Scope ignored = span.makeCurrent()) {
            try (HttpClient httpClient = HttpClient.newHttpClient()) {
                HttpRequest request = HttpRequest.newBuilder()
                        .uri(uri)
                        .GET()
                        .build();
                HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
                span.setAttribute(HttpAttributes.HTTP_RESPONSE_STATUS_CODE, response.statusCode());
                span.setAttribute(NetworkAttributes.NETWORK_PROTOCOL_VERSION, getVersion(response));
                span.setAttribute(UrlAttributes.URL_FULL, response.uri().toString());
                span.setAttribute(ServerAttributes.SERVER_ADDRESS, response.uri().getHost());
                span.setAttribute(ServerAttributes.SERVER_PORT, response.uri().getPort());
            } catch (Exception e) {
                log.error("Failed to fetch customer identity", e);
            }
        } finally {
            span.end();
        }

That still leaves me with 2 spans, the one I made does have the service.peer.name set, but the child one doesn't and it doesn't get suppressed even though I seem to have all the same semconv attributes set other than this additional peer one?

[trask]

not today, but once we get open-telemetry/opentelemetry-specification#4931, this should be possible

[mikelly-psat]

Hm, so we'd have something like this instead?

        Context ctx = Context.current()
                .with(ContextKey.named(PeerIncubatingAttributes.PEER_SERVICE.getKey()), "customer-idp")
                .with(ContextKey.named(ServiceIncubatingAttributes.SERVICE_PEER_NAME.getKey()), "customer-idp");
        try (Scope unused = ctx.makeCurrent()) {
            try (HttpClient httpClient = HttpClient.newHttpClient()) {
                HttpRequest request = HttpRequest.newBuilder()
                        .uri(uri)
                        .GET()
                        .build();
                HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
                // do something with the response
            } catch (Exception e) {
                log.error("Failed to fetch customer identity", e);
            }
        }

[trask]

yeah, details haven't been worked out, but that's the basic idea

[laurit]

@trask wondering whether we should revisit the idea of adding a contextual span processing function from user code #6191 Here we were just lucky that we had a library instrumentation for java http client. A span processing function could allow modifying attributes and also capturing the create span. This could come handy for example with messaging where you could get access to the span created within Message msg = consumer.receive() that you currently can't observe.

[laurit]

but none of the span suppression strategies I've tried seem to have any effect

The span suppression works only with the instrumentation api. If you manage to suppress the built-in instrumenation you will probably have to also handle context propagation (shouldn't be hard). You could try using the manual instrumentation library for the java http client (https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/java-http-client/library). There you could provide a custom attribute extractor

opentelemetry-java-instrumentation/instrumentation/java-http-client/library/src/main/java/io/opentelemetry/instrumentation/javahttpclient/JavaHttpClientTelemetryBuilder.java

Line 43 in 8c4b75e

public JavaHttpClientTelemetryBuilder addAttributesExtractor(

that adds the peer service attribute. The manual instrumentation library should suppress the agent instrumentation.

[mikelly-psat]

Thanks! This seems to "just work" w/o changing how anything else is instrumented!

    private static HttpClient getHttpClient() {
        return JavaHttpClientTelemetry
                .builder(GlobalOpenTelemetry.getOrNoop())
                .addAttributesExtractor(AttributesExtractor.constant(ServiceIncubatingAttributes.SERVICE_PEER_NAME, "customer-idp"))
                .addAttributesExtractor(AttributesExtractor.constant(PeerIncubatingAttributes.PEER_SERVICE, "customer-idp"))
                .build()
                .wrap(HttpClient.newHttpClient());
    }

Just using that specially constructed HttpClient seems to only result in a single Span, that is properly tagged, at least w/ Jaeger.