docs: add infra relationship example for collector

other-examples/collector/internal-telemetry-infra-relationships/README.md

@@ -0,0 +1,144 @@
+# Monitoring Hosts with OpenTelemetry Collector
+
+This example demonstrates configuring the [internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/) of an OTel Collector in way that also provides infrastructure relationships between the collector and its host and container.
+
+## Architecture
+```mermaid
+flowchart LR
+  subgraph Collector
+    direction LR
+    internaltelemetry["Internal Telemetry"]
+    subgraph Pipeline
+      direction TB
+      otlpreceiver["otlpreceiver"]
+      resourcedetectionprocessor["resourcedetectionprocessor <br> (adds host.id)"]
+      otlphttpexporter["otlphttpexporter"]
+    end
+  end
+
+  internaltelemetry -- "OTLP/HTTP" --> otlpreceiver
+  otlpreceiver --> resourcedetectionprocessor
+  resourcedetectionprocessor --> otlphttpexporter
+  otlphttpexporter --> newrelic["New Relic OTLP Endpoint"]
+
+  k8sdownwardapi["k8s Downward API"]
+  k8sdownwardapi -- "namespace, pod" --> collectorconfig
+
+  collectorconfig["Collector Config <br> (adds <br> k8s.cluster.name, <br> k8s.namespace.name, <br> k8s.pod.name, <br> k8s.container.name)"]
+  collectorconfig --> internaltelemetry
+```
+
+## Requirements
+
+- You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. This example was tested on [AWS EKS](https://aws.amazon.com/eks/) with Amazon Linux nodes. The steps for achieving a container relationship should be universal for all k8s clusters - they also work on local clusters like `kind` or `minikube`.
+- Your infrastructure must be instrumented with one of our OTel infrastructure agents. We recommend using the [nr-k8s-otel-collector](https://github.com/newrelic/helm-charts/tree/master/charts/nr-k8s-otel-collector) helm chart for containers and [nrdot-collector-host](https://github.com/newrelic/nrdot-collector-releases/blob/main/distributions/nrdot-collector-host/README.md) for hosts, see instructions below. Please note that we're actively working on `nr-k8s-otel-collector` emitting host entities compatible with relationship synthesis which will eliminate the need for `nrdot-collector-host` but until that is done, there will be some overlap in the host telemetry these solutions scrape. If you are only interested in container relationships, you can follow the instructions below to skip installing it.
+- The host relationship is synthesized based on the `host.id` attribute matching up on the host and collector telemetry. The determination of this attribute heavily depends on your environment and is driven by the `resourcedetectionprocessor` which does not support local clusters out-of-the-box. You might be able to make it work by tweaking the processor configuration, but we won't cover this here as there are too many variables involved.
+- [A New Relic account](https://one.newrelic.com/)
+- [A New Relic license key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key)
+
+### Collector
+We'll use [otelcol-contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib) for the example but if you are using your own collector, here is the what and why regarding components:
+- [otlpreceiver](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver/README.md) to provide a hook for the internal telemetry to get funnelled into a pipeline defined in the collector itself.
+- [resourcedetectionprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor) to add `host.id` to internal telemetry.
+- [otlphttpexporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter) to send telemetry to New Relic.
+- (optional) [memorylimiterprocessor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor) and [batchprocessor](https://github.com/open-telemetry/opentelemetry-collector/tree/v0.102.0/processor/batchprocessor) for best practices.
+
+### Appendix
+- Collector entity definition: [EXT-SERVICE](https://github.com/newrelic/entity-definitions/blob/main/entity-types/ext-service/definition.yml#L72-L94)
+  - requires `service.name` on internal telemetry
+- Collector to container relationship: [INFRA_KUBERNETES_CONTAINER-to-EXT_SERVICE](https://github.com/newrelic/entity-definitions/blob/main/relationships/synthesis/INFRA_KUBERNETES_CONTAINER-to-EXT_SERVICE.yml#L40)
+  - requires `k8s.cluster.name`, `k8s.namespace.name`, `k8s.pod.name`, `k8s.container.name` on internal telemetry that matches
+  equivalent attributes on the container telemetry.
+- Collector to host relationship: [INFRA-HOST-to-EXT-SERVICE](https://github.com/newrelic/entity-definitions/blob/main/relationships/synthesis/INFRA-HOST-to-EXT-SERVICE.yml)
+  - requires `host.id` on internal telemetry that matches the host telemetry.
+
+
+## Running the example
+
+1. Instrument your containers with [nr-k8s-otel-collector](https://github.com/newrelic/helm-charts/tree/master/charts/nr-k8s-otel-collector).
+    ```shell
+      # Cluster name is hard coded as the downward API does not expose it
+      license_key='INSERT_API_KEY'
+      cluster_name='INSERT_CLUSTER_NAME'
+      helm repo add newrelic https://helm-charts.newrelic.com
+      helm upgrade 'nr-k8s-otel-collector-release' newrelic/nr-k8s-otel-collector \
+      --install \
+      --create-namespace --namespace 'newrelic' \
+      --dependency-update \
+      --set "cluster=${cluster_name}" \
+      --set "licenseKey=${license_key}"
+    ```
+1. Update the values in [secrets.yaml](./k8s/secrets.yaml) based on the comments and your setup.
+    * Note, be careful to avoid inadvertent secret sharing when modifying `secrets.yaml`. To ignore changes to this file from git, run `git update-index --skip-worktree k8s/secrets.yaml`.
+
+1. Deploy the collector (see `collector.yaml` - we're using [contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib) as an example) and `nrdot-collector-host` (for host instrumentation) with the following command. If you wish to skip installing `nrdot-collector-host`, you can just delete or comment out the file `k8s/nrdot-host.yaml`
+
+    ```shell
+    kubectl apply -f k8s/
+    ```
+
+1. When finished, cleanup resources with the following command. This is also useful to reset if modifying configuration.
+
+   ```shell
+   kubectl delete -f k8s/
+   helm uninstall 'nr-k8s-otel-collector-release' --namespace 'newrelic'
+   ```
+
+## Viewing your data
+
+### In the UI
+The infrastructure relationships are used to light up our APM UI. Navigate to "New Relic -> All Entities -> Services - OpenTelemetry" and click on the service with name corresponding to value provided in `secrets.yaml` for `COLLECTOR_SERVICE_NAME`. The 'Summary' page shows metrics related to the infrastructure entities related to your collector at the bottom of the page.
+
+
+### From the CLI
+You can also query the relationships through NerdGraph using the [newrelic CLI](https://github.com/newrelic/newrelic-cli/blob/main/docs/GETTING_STARTED.md#environment-setup). Note that the api key in this case is NOT an ingest key (as used above), but instead a user key.
+
+The following script should work if your service name is sufficiently unique as the first part determines the entity guid based on the service name.
+If you have the correct entity guid, you can skip the first part and just query the relationships directly.
+
+```bash
+#!/bin/bash
+export NEW_RELIC_REGION='US'
+export NEW_RELIC_API_KEY='INSERT_USER_KEY'
+SERVICE_NAME='INSERT_SERVICE_NAME'
+
+ENTITY=$(newrelic nerdgraph query "{
+  actor {
+    entitySearch(queryBuilder: {name: \"${SERVICE_NAME}\"}) {
+      results {
+        entities {
+          guid
+          name
+        }
+      }
+    }
+  }
+}")
+SERVICE_ENTITY_GUID=$(jq -r '.actor.entitySearch.results.entities[0].guid' <<< "$ENTITY")
+
+newrelic nerdgraph query "{
+  actor {
+    entity(guid: \"${SERVICE_ENTITY_GUID}\") {
+      relatedEntities(filter: {relationshipTypes: {include: HOSTS}}) {
+        results {
+          source {
+            entity {
+              name
+              guid
+              domain
+              type
+            }
+          }
+          type
+          target {
+            entity {
+              guid
+              name
+            }
+          }
+        }
+      }
+    }
+  }
+}" 
+```

other-examples/collector/internal-telemetry-infra-relationships/k8s/_namespace.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: internal-telemetry-infra-relationship

other-examples/collector/internal-telemetry-infra-relationships/k8s/collector.yaml

@@ -0,0 +1,69 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: collector
+  namespace: internal-telemetry-infra-relationship
+  labels:
+    app.kubernetes.io/name: collector
+spec:
+  selector:
+    matchLabels:
+      name: collector
+  template:
+    metadata:
+      labels:
+        name: collector
+    spec:
+      containers:
+        - name: collector-infra-relationships
+          image: otel/opentelemetry-collector-contrib:0.130.1
+          args:
+            - --config=/config/config.yaml
+            # add k8s metadata as resource attributes
+            - '--config=yaml:service::telemetry::resource::k8s.cluster.name: ${env:CLUSTER_NAME}'
+            - '--config=yaml:service::telemetry::resource::k8s.namespace.name: "${env:NAMESPACE}"'
+            - '--config=yaml:service::telemetry::resource::k8s.pod.name: "${env:POD_NAME}"'
+            # Hardcoded container name - needs to match the container's name above
+            - '--config=yaml:service::telemetry::resource::k8s.container.name: collector-infra-relationships'
+          env:
+            # New Relic OTLP endpoint
+            - name: NEW_RELIC_OTLP_ENDPOINT
+              valueFrom:
+                secretKeyRef:
+                  name: collector-internal-telemetry-secret
+                  key: NEW_RELIC_OTLP_ENDPOINT
+            # The New Relic API key used to authenticate export requests.
+            # Defined in secrets.yaml
+            - name: NEW_RELIC_LICENSE_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: collector-internal-telemetry-secret
+                  key: NEW_RELIC_API_KEY
+            # defines the collector's entity name in New Relic
+            - name: SERVICE_NAME
+              valueFrom:
+                secretKeyRef:
+                  name: collector-internal-telemetry-secret
+                  key: COLLECTOR_SERVICE_NAME
+            - name: CLUSTER_NAME
+              valueFrom:
+                secretKeyRef:
+                  name: collector-internal-telemetry-secret
+                  key: CLUSTER_NAME
+            # k8s.namespace.name from Downward API
+            - name: NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+            # k8s.pod.name from Downward API
+            - name: POD_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+          volumeMounts:
+            - name: config
+              mountPath: /config
+      volumes:
+        - name: config
+          configMap:
+            name: collector-config

other-examples/collector/internal-telemetry-infra-relationships/k8s/configmap.yaml

@@ -0,0 +1,95 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: collector-config
+  namespace: internal-telemetry-infra-relationship
+data:
+  config.yaml: |
+    receivers:
+      otlp/internal:
+        protocols:
+          # listens on localhost:4318 by default, so the collector's internal telemetry otlpexporters can write to this without further configuration
+          http:
+
+    processors:
+      batch:
+      memory_limiter:
+        check_interval: 1s
+        limit_percentage: 75
+        spike_limit_percentage: 15
+
+      # adds host.id as resource attribute
+      # Note: if host entities are monitored (and thus their entity synthesis driven) by a collector C, this processor should mirror C's configuration
+      resourcedetection/internal:
+        # only one detector should technically be necessary, but this list covers the big three cloud providers and their managed Kubernetes services
+        # env detector allows providing the host.id via env var, i.e. OTEL_RESOURCE_ATTRIBUTES=host.id=<host_id> for more static setups and manual testing/debugging
+        detectors: [env, eks, ec2, aks, azure, gcp]
+        timeout: 2s
+
+    exporters:
+      otlphttp/internal:
+        endpoint: "${env:NEW_RELIC_OTLP_ENDPOINT}"
+        headers:
+          api-key: "${env:NEW_RELIC_LICENSE_KEY}"
+
+    service:
+      pipelines:
+        ####
+        #  insert your normal pipelines
+        ####
+        logs/internal:
+          receivers:
+            - otlp/internal
+          processors:
+            - memory_limiter
+            - resourcedetection/internal
+            - batch
+          exporters:
+            - otlphttp/internal
+        metrics/internal:
+          receivers:
+            - otlp/internal
+          processors:
+            - memory_limiter
+            - resourcedetection/internal
+            - batch
+          exporters:
+            - otlphttp/internal
+        traces/internal:
+          receivers:
+            - otlp/internal
+          processors:
+            - memory_limiter
+            - resourcedetection/internal
+            - batch
+          exporters:
+            - otlphttp/internal
+
+      telemetry:
+        resource:
+          service.name: "${env:SERVICE_NAME}"
+        metrics:
+          level: detailed
+          readers:
+            - periodic:
+                exporter:
+                  otlp:
+                    protocol: http/protobuf
+                    endpoint: http://localhost:4318
+        logs:
+          level: INFO
+          disable_stacktrace: true
+          disable_caller: true
+          processors:
+            - batch:
+                exporter:
+                  otlp:
+                    protocol: http/protobuf
+                    endpoint: http://localhost:4318
+        traces:
+          processors:
+            - batch:
+                exporter:
+                  otlp:
+                    protocol: http/protobuf
+                    endpoint: http://localhost:4318

other-examples/collector/internal-telemetry-infra-relationships/k8s/nrdot-host.yaml

@@ -0,0 +1,48 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: nrdot-host
+  namespace: internal-telemetry-infra-relationship
+  labels:
+    app.kubernetes.io/name: nrdot-host
+spec:
+  selector:
+    matchLabels:
+      name: nrdot-host
+  template:
+    metadata:
+      labels:
+        name: nrdot-host
+    spec:
+      containers:
+        - name: nrdot-host
+          image: newrelic/nrdot-collector-host:1.3.0
+          args:
+            - --config=/etc/nrdot-collector-host/config.yaml
+            - "--config=yaml:receivers::hostmetrics::root_path: /hostfs"
+            # remove once nrdot-collector-host does this by default
+            - "--config=yaml:processors::resourcedetection/cloud::ec2::resource_attributes::host.name::enabled: true"
+          env:
+            # The New Relic API key used to authenticate export requests.
+            # Defined in secrets.yaml
+            - name: NEW_RELIC_LICENSE_KEY
+              valueFrom:
+                secretKeyRef:
+                  # reusing the internal telemtry key to ensure host telemetry is colocated in the same account
+                  name: collector-internal-telemetry-secret
+                  key: NEW_RELIC_API_KEY
+                        # New Relic OTLP endpoint
+            - name: OTEL_EXPORTER_OTLP_ENDPOINT
+              valueFrom:
+                secretKeyRef:
+                  name: collector-internal-telemetry-secret
+                  key: NEW_RELIC_OTLP_ENDPOINT
+          volumeMounts:
+            - name: hostfs
+              mountPath: /hostfs
+              readOnly: true
+      volumes:
+        - name: hostfs
+          hostPath:
+            path: /
+            type: Directory

other-examples/collector/internal-telemetry-infra-relationships/k8s/secrets.yaml

@@ -0,0 +1,17 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: collector-internal-telemetry-secret
+  namespace: internal-telemetry-infra-relationship
+stringData:
+  # New Relic API key to authenticate the export requests.
+  # docs: https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key
+  NEW_RELIC_API_KEY: INSERT_API_KEY
+  # The default US endpoint is set here. If your account is based in the EU, use `https://otlp.eu01.nr-data.net` instead.
+  # docs: https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#configure-endpoint-port-protocol
+  NEW_RELIC_OTLP_ENDPOINT: https://otlp.nr-data.net/
+  # The cluster name; will be added as attribute `k8s.cluster.name` to the internal telemetry.
+  CLUSTER_NAME: INSERT_CLUSTER_NAME
+  # Determines the entity name in New Relic; will be added as `service.name` to internal telemetry.
+  COLLECTOR_SERVICE_NAME: infra-relationships-service
+