6473 – Update collector configuration to allow multiple services (#110)

Updates the collector configuration to deal with receiving and exporting metrics from multiple services.

### Important
**When thinking about adding multiple receivers/exporters, we need to take into consideration:**
- unlike most otel-collector's receivers and exporters, we can't add multiple prometheus receivers by aliasing, e.g.: `prometheus/service_1`, `prometheus/service_2`. We can have only one receiver, which can have multiple jobs.
- unlike when sending traces, we need to specify the dataset when exporting metrics to honeycomb via the otlp exporter.

**The way we dealt with this is by using the routing connector:**
- we have a pipeline with the prometheus receiver, and the connector functioning as the exporter
- we have a second pipeline with the connector functioning as the receiver, and otlp set up as the exporter
- inside the connector we have a table where we use the service name to decide into which pipeline it should go
    - the prometheus receiver sets the service name from the job_name

**If we want to send prometheus metrics from another service, we would need to:**
- add the endpoint variable to the otel-collector configuration in the docker-compose file (local development)
- add a new job to prometheus' scrape_config,
- add another item to the routing connector table
- add another exporter `otlp/service_metrics`, with the correct Dataset (considering we are also sending metrics to Honeycomb)
- add another pipeline `metrics/service` with the new exporter

**Notes**
- This does add some duplication, but seems the most straightforward way to deal with this, at least for now
- There is a tool to dynamically allocate targets for the prometheus receiver, but it is specific to kubernetes, so not useful for us right now
- Regarding the connector and debugging: the default_pipelines is where anything that is not matched is sent. We currently only use it when debugging. To debug the connector, you can also use the debug pipeline (commented out), and use the debug exporter.

**References**
- [Routing Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/routingconnector)
- [Sending metrics to honeycomb](https://docs.honeycomb.io/send-data/opentelemetry/collector/#metrics-and-logs-signals)
- [Prometheus Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver)
- [Prometheus Receiver Configuration](https://github.com/prometheus/prometheus/blob/v2.28.1/docs/configuration/configuration.md#scrape_config)
- [Two Prometheus receivers can't be defined](open-telemetry/opentelemetry-operator#3034)
- [Target Allocator](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver#opentelemetry-operator)

Author: vasconsaurus

docker-compose.yml

@@ -169,11 +169,12 @@ services:
   #   volumes:
   #     - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
   #   environment:
+  #     pender_metrics_endpoint: "pender:3200"
   #     x_honeycomb_team: ${X_HONEYCOMB_TEAM} # Set this env var in your shell before running docker-compose
   #     RAILS_ENV: development
   #     SERVER_PORT: 3200
   #   networks:
-  #     - dev 
+  #     - dev
   web:
     build: check-web
     platform: linux/x86_64

otel-collector-config.yaml

@@ -2,10 +2,11 @@ receivers:
   prometheus:
     config:
       scrape_configs:
-        - job_name: "prometheus"
+        - job_name: "pender_metrics"
           scrape_interval: 15s
           static_configs:
-            - targets: ["pender:3200"]
+            - targets: ["${env:pender_metrics_endpoint}"]
+          metrics_path: /metrics
 
 processors:
   memory_limiter:
@@ -17,19 +18,39 @@ processors:
     send_batch_size: 8192
 
 exporters:
-  otlp/metrics:
+  otlp/pender_metrics:
     endpoint: "api.honeycomb.io:443" # US instance
     #endpoint: "api.eu1.honeycomb.io:443" # EU instance
     headers:
       "x-honeycomb-team": ${env:x_honeycomb_team} # Honeycomb API KEY
       "x-honeycomb-dataset": "pender"
+  # for debugging purposes only
+  # debug:
+  #   verbosity: normal
+
+connectors:
+  routing:
+    default_pipelines: []
+    # for debugging purposes only
+    # default_pipelines: [metrics/debug]
+    table:
+      - context: resource
+        condition: attributes["service.name"] == "pender_metrics"
+        pipelines: [metrics/pender_honeycomb]
 
 service:
   # telemetry:
   #   logs:
   #     level: "debug"
   pipelines:
-    metrics:
+    metrics/prometheus:
       receivers: [prometheus]
+      exporters: [routing]
+    metrics/pender_honeycomb:
+      receivers: [routing]
       processors: [memory_limiter, batch]
-      exporters: [otlp/metrics]
+      exporters: [otlp/pender_metrics]
+    # for debugging purposes only
+    # metrics/debug:
+    #   receivers: [routing]
+    #   exporters: [debug]