[countersyncd]: Optimize single-flight OTLP export and benchmark delivery

[50n1c-rnsft]

Rebase and cross-host confirmation

Rebased onto #4884 at 76753cb; current integration commit 01348bc. The PR base remains master until the dependency is merged, so shared IPFIX changes are still included in the diff.

Rebased shared IPFIX pipeline, client on zegan-dev-vm and raw gRPC server on zegan-vm0, both unpinned with all CPUs available, 500 counters/record and 10K counters/export:

Workers	Lanes/worker	Trial 1 (M/s)	Trial 2	Trial 3	Aggregate (M/s)	Payload Gbps
2	4	10.603	10.540	10.586	10.58	11.87
16	2	10.668	10.644	10.647	10.65	11.90
16	4	10.399	10.651	10.654	10.57	11.79

Each trial contains 200M counters; each three-trial group delivered 60,000 complete RPCs. The raw sink validates complete request counts, not per-point contents. Timing includes parsing, routing, encoding and final ACKs. A separate same-direction TCP check reached 11.99/11.90/11.91Gbps with 1/4/8 connections respectively, consistent with a roughly 12Gbps path limit despite the virtual NIC reporting 100Gbps. These cross-host results are network-limited and do not establish the sender's CPU ceiling. Experimental large-frame servers and encode-only diagnostic bypasses are not production changes.

Summary

Optimize OTLP Gauge export with direct protobuf encoding, a configurable ordered worker pool, and shared IPFIX template metadata.

• Encode standard OTLP protobuf directly into precisely sized Tonic buffers, avoiding per-point protobuf object trees and an intermediate whole-request byte copy.
• Retain immutable samples through ACK/retry completion; recover reusable sample storage only after transport references are released.
• Route each (object_name, type_id, stat_id) series to one stable lane. Each lane sends one request at a time, including retries; independent lanes can overlap requests.
• Export canonical SAI metric names and the three point attributes object_name, sai_type, and sai_stat. Preserve individual timestamps, values, duplicate samples, and per-series input order.
• Share IPFIX template identities across sampled records. Cache generation-scoped lane routing plans and encoder field-to-series-slot mappings.
• Drain queued input, tail batches, and ACKs on normal shutdown; propagate worker failures and cancel sibling workers.

This remains a draft. The current measurements do not establish 50M points/s, physical-network capacity, or persistent-storage throughput.

Worker configuration

Option	Default	Meaning
--otel-worker-threads	1	Dedicated sender threads
--otel-in-flight-per-worker	1	Independent ordered single-flight lanes per sender
--otel-worker-cpus	unset	Optional logical CPU ID per sender thread
--otel-worker-queue-capacity	8	Input batches queued per lane
--otel-max-counters-per-export	10000	Point target per lane/request
--otel-flush-timeout-ms	1000	Per-lane flush timeout

The router runs on the daemon runtime in addition to sender threads. Maximum concurrent requests are bounded by workers times lanes, with one request per lane. Defaults retain one worker and one lane; the entire pool is not globally single-flight when configured with multiple lanes. Pool size is fixed at startup.

Benchmark matrix (measured before the API rebase)

The following full worker/lane matrix was measured at b3b0aeb, before rebasing onto #4884. It is retained as measured evidence, not presented as a fresh matrix for the rebased HEAD. Subsequent cross-host measurements of the rebased implementation are summarized below.

Common setup and timing

• Host: zegan-dev-vm, Linux x86-64, Intel Xeon Platinum 8370C.
• Rust 1.90.0, release, native CPU code generation, benchmark-only mimalloc, locked standalone dependencies. Production daemon allocator is unchanged.
• Router pinned to CPU 0; sender threads pinned to CPU 2, CPUs 2/4, or CPUs 2/4/6. Receiver pinned to CPUs 8/10/12/14 in every case.
• Sender-side CPU allocation is therefore worker count plus one router CPU. The receiver has four additional physical cores.
• 500 object/stat time series using 100 known Port stat names; all three canonical-name attributes remain on each point. Values and timestamps vary per record.
• 10,000 points/request, queue capacity 8/lane, two trials of 40 million points per configuration per receiver: 36 timed trials total.
• otel-pool-bench pre-generates owned SAI batches outside timing. Timing includes live routing, hashing, cloning, channels, encoding, worker startup/shutdown, and all final ACKs. It does not include IPFIX parsing or exercise the shared-IPFIX input fast path; that path has a separate integration table below.
• Localhost gRPC, no TLS or compression. Raw and Collector tests run separately.
• Units below are millions of ACKed counter samples per second. Both trials are shown, including slower runs. Two trials are not a statistical confidence interval.

Raw gRPC receive-only sink

The grpc-go sink counts complete requests and payload bytes without decoding protobuf or persisting data. Every configuration delivered 8,000 complete requests across its two trials. Sample semantics are checked separately by decoding delivery tests.

Sender workers	Lanes per worker	Max concurrent RPCs	Trial 1 (M/s)	Trial 2 (M/s)
1	1	1	4.601	4.356
1	2	2	5.873	4.963
1	4	4	5.780	6.103
2	1	2	9.215	8.945
2	2	4	11.602	11.233
2	4	8	9.959	10.326
3	1	3	11.808	11.142
3	2	6	15.835	15.376
3	4	12	14.520	14.509

Actual Go OpenTelemetry Collector

Collector core v0.123.0, OTLP receiver, nop exporter, GOMAXPROCS=4, GOGC=500. This path performs real protobuf decoding. Each configuration accepted exactly 80 million points across its two trials with zero refused points. Accepted counts do not validate every individual value and do not imply durable storage.

Sender workers	Lanes per worker	Max concurrent RPCs	Trial 1 (M/s)	Trial 2 (M/s)
1	1	1	1.318	1.298
1	2	2	2.415	2.554
1	4	4	4.549	2.784
2	1	2	2.586	2.605
2	2	4	4.635	4.634
2	4	8	4.487	5.070
3	1	3	3.510	3.522
3	2	6	5.125	5.058
3	4	12	4.825	4.684

The 3-worker/2-lane configuration delivered the highest aggregate throughput in both matrices: about 15.60M/s to the raw sink and 5.09M/s to the Collector. More lanes did not consistently improve throughput. The default 1-worker/1-lane configuration remains around 1.3M/s against this Collector, so a 5M/s single-flight target is not achieved. Receiver decode/allocation and scheduling costs matter; raw-sink throughput must not be presented as Collector throughput.

IPFIX and shared-input integration benchmarks

The following recorded A/B runs isolate the addition of shared metadata to the already optimized, named, direct-encoding pipeline. Their baseline is the pre-shared implementation inside this PR, not upstream before the entire PR. They are separate from the freshly rerun owned-input worker matrix above.

Actual IPFIX actor: parsing, traversal/checksum, and destruction

Producer, parser, and consumer on CPU 0; same compiler/allocator configuration; 16 pre-generated valid IPFIX payloads; template setup and readiness probe excluded. Three trials per version. Throughput in M counters/s.

Counters/record	Points/trial	Pre-shared trial 1	Trial 2	Trial 3	Shared trial 1	Trial 2	Trial 3
2	40M	5.522	5.435	5.443	5.429	5.489	5.510
500	400M	64.690	64.932	65.012	210.240	210.354	210.186
8,000	400M	65.877	63.930	64.577	251.126	250.685	250.965

Actual IPFIX -> ordered OTel pool -> raw gRPC

Parser/router on CPU 0, three sender workers on CPUs 2/4/6, two lanes each, receiver on CPUs 8/10/12/14; 500 counters/IPFIX record, 10K exports, 100 million points/trial. Parsing and live routing are included; input generation is excluded and final ACKs are included.

Trial	Pre-shared (M counters/s)	Shared (M counters/s)
1	16.316	20.660
2	15.678	20.393
3	15.853	21.155
Aggregate: total points / total time	15.94	20.73

Each trial delivered 10,000 complete requests and 14,062,851,000 protobuf payload bytes. The approximately 30% improvement applies to this shared-input integration comparison. The raw performance sink does not decode values; a separate real IPFIX/shared fan-out/OTel/Tonic test checks decoded fields, timestamps, values, duplicates, and per-series order. An equivalent shared-IPFIX-to-Go-Collector throughput matrix has not been measured.

Correctness, compatibility, and review scope

• Recorded standalone validation for this implementation: 124 tests passed, one manual calibration ignored. Coverage includes protobuf equivalence, per-series order and duplicates, identical retries, partial-success rejection without partial retry, tail/timer flush, worker concurrency ceilings/failure propagation, template generations, cache eviction, mixed batches, and real decoded IPFIX-to-OTLP delivery.
• This branch now uses the unified borrowed SAIStatsBatch::iter() API from [countersyncd]: Share IPFIX output metadata with borrowed batch views #4884. The records() alias and lazy owned projection are removed. Shared names are cached at template construction and reused by the OTel encoder.
• OTel names/attributes change from numeric metric names and sai_type_id/sai_stat_id to canonical SAI names and sai_type/sai_stat. Existing Port, Queue, BufferPool, and IngressPriorityGroup mappings are used; unknown IDs retain explicit ID-bearing fallback names. Downstream queries need migration.
• Series stay in one lane, preserving input order rather than sorting timestamps. Ambiguous transport retries can still duplicate delivery; there is no exactly-once guarantee. Cross-series ordering and downstream processing order are not guaranteed.
• Numeric values still copy during lane gathering and encoder sample append. Direct encoding avoids an intermediate whole-request copy; it is not zero-copy network I/O.
• Routing/slot plan caches are bounded by 64 entries, not total metadata bytes. Per-lane queues and caches, large names/templates, and compatibility projections require memory review. A hot or stalled lane can backpressure the central router.
• After rebasing, Linux SONiC Bookworm checks passed with RUSTFLAGS=-Dwarnings: cargo check --locked -p countersyncd --bins --tests --benches and cargo test --locked -p countersyncd. Results: 187 library tests, 198 binary tests, and 18 integration/helper tests passed; manual encoding calibration ignored in each unit target. The standalone suite passed 127 tests with one calibration ignored. Hardware HFT testing has not been established for this OTel branch.

Reproduction and evidence

Build the current pool benchmark:

CARGO_TARGET_DIR=/tmp/otel-pool-target RUSTFLAGS='-C target-cpu=native' \
  cargo +1.90.0 build --release --locked --features mimalloc \
  --bin otel-pool-bench \
  --manifest-path crates/countersyncd/benches/otel-standalone/Cargo.toml

Run the raw and Collector runners for each sender CPU list 2, 2,4, and 2,4,6, with lanes 1 2 4:

python3 crates/countersyncd/benches/otel-standalone/raw-grpc/run.py \
  --server /path/raw-grpc-sink --client /tmp/otel-pool-target/release/otel-pool-bench \
  --output /tmp/raw-results --pool --named-schema --router-cpu 0 \
  --client-cpus 2,4,6 --server-cpus 8,10,12,14 --procs 4 \
  --inflight 1 2 4 --points 40000000 --repeats 2

python3 crates/countersyncd/benches/otel-standalone/collector_bench.py \
  --collector /path/otelcol --client /tmp/otel-pool-target/release/otel-pool-bench \
  --output /tmp/collector-results --pool --router-cpu 0 \
  --client-cpus 2,4,6 --cpus 8,10,12,14 --gomaxprocs 4 --gogc 500 \
  --inflight 1 2 4 --points 40000000 --repeats 2

Use dedicated ports as configured by the runners. Current matrix JSON, request counts, Collector accepted/refused counters, and process metrics are retained on zegan-dev-vm under /tmp/pr4883-description-head/results/.

Repository references: worker configuration, shared batches and integration A/B, direct encoding, and schema migration. Performance tables in the older documents/comments describe their respective implementation stages; the current-head worker matrices are the ones above.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[50n1c-rnsft]

Follow-up: real Go Collector concurrency and profiling

Pushed f0fbf85 (bounded benchmark-only concurrency/external Collector runner) and d898f0a (timed client CPU, pprof and tuning results).

The single-flight target is still unmet. One client core with eight in-flight requests now exceeds it:

• Go Collector core v0.123.0, production attributes intact, 10K batch, default GC: 5.665 / 5.781M ACKed points/s, two 200M-point trials.
• Tuned GOGC=500, Collector pinned to seven distinct physical cores (GOMAXPROCS=7), same single pinned client runtime thread and eight in-flight requests: 6.908 / 6.993M/s, two more 200M-point trials. Accepted delta=400M, refused=0. Client measured inside timed ACK intervals: 0.916 / 0.912 CPU equivalents.
• GOGC=1000 reached 7.121 / 7.199M/s in shorter 40M-point trials; not the recommended confirmed configuration. N=32 regressed; larger batch was not better.

20-second Go CPU profile at default GC: request protobuf Unmarshal ~68.8% cumulative, mallocgc ~45.1% cumulative, gcDrain ~19.3% cumulative (overlapping call paths, not additive). Allocation profile attributed ~90% of allocated bytes to the metric decode tree. Tuned run still spends substantial CPU on per-point/attribute allocation. It used ~5 Collector CPU equivalents while delivering ~7M/s; final RSS snapshot ~351.5MiB, not a peak.

This is not a protocol theoretical limit. The tuned single-client-core path is approaching saturation: ~130-135ns/point CPU cost gives a conditional current-implementation budget of ~7.4-7.7M/s, not a throughput guarantee. Further large gains require reducing client CPU/copies and receiver allocation, or scaling/sharding. pprof evidence and commands are in crates/countersyncd/benches/otel-standalone/PROFILING.md.

Validation: 21 module/delivery tests passed (manual calibration ignored); concurrent-path tail/timer/partial rejection tests passed with N=8 (single-flight retry test excluded because benchmark concurrent mode intentionally fails fast). Go tests verify all ACKs/count deltas, not per-value checksums. Production remains single-flight without the benchmark feature; concurrency retry/error/ordering semantics remain draft work. No InfluxDB persistence or physical NIC throughput claimed.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

_{---Powered by SONiC BuildBot}

[mssonicbld]

/azp run

[50n1c-rnsft]

Configurable ordered worker pool — 582be6d

Added startup configuration to the daemon:

• --otel-worker-threads (default 1)
• --otel-in-flight-per-worker (default 1)
• --otel-worker-cpus (optional comma-separated logical CPU IDs)
• --otel-worker-queue-capacity (default 8)

This is an ordered pool, not the prior unordered benchmark experiment. Concurrency is threads × independent series lanes; each lane uses the real actor's single-flight export/retry path. Every series hashes to one lane. Startup validation bounds threads/lanes/queues and validates CPU lists. Normal close drains tails/ACKs and joins worker threads; worker error/panic is detected even with idle upstream, and sibling workers are cancelled. Defaults remain one worker / one lane. No automatic live resizing: it would require a drain/ownership-migration protocol.

Added otel-pool-bench with live routing/hash/stat-cloning included in timing. This should not be compared directly with the older pre-sharded POC. On zegan-dev-vm: three sending workers plus one router core, two lanes per worker, 10K batch, two 200M-point trials against the raw gRPC byte sink achieved 16.827 / 17.042M/s, with 40,000 complete requests. Same pool against actual Go Collector with four receiver cores/GOGC=500 achieved 5.423 / 5.406M/s (two 40M trials; accepted 80M/refused 0). No InfluxDB/50M claim.

Validation: 26 standalone Rust tests passed (one manual calibration ignored). New pool tests check exact decoded fields/duplicates/per-series order, retry ordering, per-lane/global concurrent request ceilings, overlapping lanes, bounds/CPU validation, empty shutdown, and idle-input errors. The four pool tests also passed on Linux with the unordered benchmark environment override set to 8, proving the ordered pool still uses single-flight lanes. Root cargo metadata --locked resolved successfully on Linux. Daemon CLI parser tests were added but the full native SONiC daemon build remains untested here.

Details and commands: crates/countersyncd/benches/otel-standalone/WORKER_CONFIGURATION.md. Also includes the raw gRPC sink and ordered scaling experiments previously run locally. PR remains draft pending full target integration and review of cancellation/retry/downstream ordering semantics.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[50n1c-rnsft]

Canonical SAI names + refreshed matrix (7a28d92)

Known type/stat pairs now export real SAI names: Metric.name=SAI_PORT_STAT_IF_IN_OCTETS, attributes object_name, sai_type=SAI_OBJECT_TYPE_PORT, sai_stat=SAI_PORT_STAT_IF_IN_OCTETS. The old _id attribute keys are replaced. Existing SAI tables cover Port, Queue, BufferPool and IngressPriorityGroup stats; unknown pairs retain explicit collision-free ID-bearing fallback names rather than dropping samples. Resolve/cache once per series, not per sample. Internal routing identity remains numeric, and timestamps/values/order/retry behavior are unchanged.

52 standalone tests passed (one manual calibration ignored); Linux tests also passed before adding the final workload-schema assertion. Tests cover all four name families, unknown/extensions, fast encoder vs message conversion, and real RPC attribute/order/retry behavior.

All benchmarks include real routing/clone/channel cost; router has one additional core. Refreshed input has 100 real port stat names across 500 object/stat series, not 500 distinct metric types. Fixed receiver four physical cores, 10K batch, native Rust/mimalloc, localhost/no TLS/no compression.

Raw gRPC receive-only, two 40M-point trials, Mpoints/s ranges:

Sender threads	Lanes/worker=1	=2	=4
1	4.728-4.868	5.756-5.892	4.298-4.394
2	7.976-8.318	7.736-9.219	7.241-7.726
3	10.109-10.178	10.458-11.032	12.543-12.969

Long confirmation (200M/trial, two trials): 3x4 13.445/13.026M/s. Same-input old numeric baseline 17.139/16.710M/s, ~22% reduction. Payload ~91 -> ~141 bytes/point (~55% increase). Do not attribute this solely to name lookup; it is cached. Raw server verifies complete requests/bytes, not decoded values.

Actual Go Collector core v0.123.0, four cores/GOGC=500, 40M/trial confirmation: 3x2 4.755/4.765M/s, 3x4 4.673/4.728M/s. Each pair accepted 80M points, zero refused. Full 9-combination Collector short-test matrix and reproduction are in crates/countersyncd/benches/otel-standalone/NAMED_METRICS.md. No InfluxDB persistence or 50M/s claim.

Schema migration requires downstream measurement/label query updates. Historical performance docs are explicitly marked as old numeric schema.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[50n1c-rnsft]

Shared metadata + unified iterator integration (b3b0aeb)

Implemented real IPFIX output as shared immutable template metadata plus numeric values. SAIStatsBatch.records() exposes an allocation-free borrowed view for owned/shared records; the old iter() slice API remains as a lazy OnceLock projection for compatibility. Hot CounterDB, StatsReporter and OTel consumers use the new view. OTel caches generation-scoped routing plans and field-to-local-slot mappings, with old metadata Arcs pinned and slot plans invalidated on eviction. All named OTLP fields/order/duplicates remain unchanged.

Actual IPFIX performance did not regress materially in measured workloads. Same-input old/new runs on zegan-dev-vm, Rust 1.90/native/mimalloc, one parser/consumer core:

Counters/record	Old Mpoints/s	Shared Mpoints/s
2 (40M points/trial)	5.435-5.522	5.429-5.510
500 (400M/trial)	64.690-65.012	210.186-210.354
8000 (400M/trial)	63.930-65.877	250.685-251.126

These measure the actual actor with parsing, full output iteration/checksum and destruction, using 16 pre-generated IPFIX payloads. They are not wire ingest or persistence rates. Small-record differences are within observed noise, not a formal no-regression proof.

Actual IPFIX -> OTel -> raw gRPC integration: old 15.678/15.853/16.316M/s, new 20.393/20.660/21.155M/s (~30% aggregate gain), three 100M-point trials. Parser/router CPU0, three sender CPUs2/4/6, two lanes/worker, receiver CPUs8/10/12/14. Every run: 10,000 RPCs and 14,062,851,000 payload bytes. A decoded end-to-end test verifies real IPFIX/shared fan-out/OTel fields, values and order; raw performance sink itself only counts complete requests/bytes.

Validation: 124 standalone tests pass, one manual calibration ignored, including actual IPFIX/StatsReporter tests, mixed-view projection/splitting, old-generation retention, slot eviction, real decoded fan-out delivery and retry/order tests. CounterDB iteration changed and its shared-input assertion added, but native SONiC/Redis tests were not run. Full native build remains a draft integration requirement.

See SHARED_BATCHES.md for exact scope, baseline reproduction, failed early benchmark runs excluded due to premature template-channel close, and cache/memory review points. Numeric samples still copy during lane gathering and sample append; this is not the isolated C prototype's fully exclusive Vec transfer. No 50M/s, physical NIC or InfluxDB claim.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).