Add OTLP metrics endpoint

[felixbarny]

Adds an OTLP endpoint for metric ingestion into TSDB.

This simplifies getting data into TSDB as configuration of mappings and dimensions isn't required. It also improves ingestion performance. Partly due to the binary encoding but also because it can re-use partial TsidBuilders for common resource attributes.

My aim was to retain exactly the same semantics and behavior as the OTel collector's Elasticsearch exporter, including data stream routing and mapping hints.

Depends on

• Optimize IP field parsing #132463
• TSDB ingest performance: combine routing and tsdb hashing #132566
• Add metrics OTLP endpoint skeleton #133296

To review, select from the 3rd commit (9cea250) onwards
Note that I might force-push to the branch once the PRs this depends on get merged.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[ruflin]

Did a quick test, and so far seems to be working as expected. For everyone that needs a config to do a quick test run with the otel collector:

receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      cpu:
      disk:
      load:
      filesystem:
      memory:
      network:
      #process:
      paging:

processors:
  batch:


exporters:
  otlphttp/elasticsearch:
    compression: gzip
    encoding: proto
    endpoint: http://localhost:9200/_otlp
    sending_queue:
      enabled: true
      block_on_overflow: true
      queue_size: 10
      num_consumers: 10

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [batch]
      exporters: [otlphttp/elasticsearch] # only use with batch processor

[JonasKunz]

Had a look at the histogram stuff only

[JonasKunz]

According to my reading of the spec it should be legal to have a single count and no bucket boundaries. This implies a single (-INF, +INF) bucket, in which case we should (and I think do) return an empty histogram.
Maybe add that case too?

[felixbarny]

If it's illegal according to the spec we could also return a validation error and drop it.

[rjernst]

We should not be adding any dependencies to server. This needs to be hidden, so as not to collide with any plugins that may try to use (a different version) of this dependency.

[felixbarny]

Good to know. I'm currently looking at the impact of not using this library in favor of the existing murmur implementation.

[felixbarny]

I think I found a good solution to use the existing hasher without regressions: #133226