go-proxy: honest shaped delivery rate via tcpi_delivery_rate (delivery_rate_mbps) — reconcile WIP + plumb to dashboard

[jonathaneoliver]

Context

The proxy's network-log throughput figure (bytes_out / transfer_ms) over-reports ~1000× on small/sub-buffer transfers: transfer_ms times only the proxy's write+flush, which returns once the kernel accepts the bytes into the socket send buffer — NOT when they reach the client. tc HTB shaping drains the qdisc below the socket, so a sub-buffer segment (~50–140 KB) is absorbed instantly and reads as 1000s of Mbps while the wire is actually capped near the video bitrate.

This is the server-side gap behind the whole iOS ABR over-read investigation — the init-segment "6–17 Mbps" garbage and the cold-start over-selection wedge (see the avplayer-cold-start-wedge finding). We've been sidestepping it with mbps_transfer_rate; this is the honest fix.

WIP (already started, preserved on a branch)

A delivery_rate feature samples the kernel's tcpi_delivery_rate (getsockopt(TCP_INFO)) on the client socket at end-of-transfer and records delivery_rate_mbps on each network-log entry — the kernel's own estimate of the actual drained-onto-the-wire rate. Connection-level (under HTTP/2 it reflects the whole socket, not one stream); Linux-only with a macOS no-op stub so the dev build compiles.

Branch: fix/network-log-delivery-rate @ 40f5f4a2 (pushed). NOTE: committed on a stale base (the #823 point), so it needs reconciling onto current dev.

Files:

• go-proxy/cmd/server/delivery_rate.go — stampDeliveryRate(conn, entry)
• go-proxy/cmd/server/delivery_rate_linux.go (//go:build linux) — readDeliveryRateBps via getsockopt(TCP_INFO)
• go-proxy/cmd/server/delivery_rate_other.go (//go:build !linux) — stub
• go-proxy/cmd/server/main.go — NetworkLogEntry.DeliveryRateMbps field + 2 call sites in handleProxy
• go-proxy/cmd/server/v2_adapter.go — surfaces delivery_rate_mbps in networkEntryToMap

To revisit

1. Reconcile the WIP commit onto current dev (cherry-pick/rebase; the main.go/v2_adapter.go anchors may have moved).
2. Verify the linux build — confirm the tcpConnFromContext(r.Context()) helper exists (or is part of the WIP) and the TCP conn is reachable from the request context on both write paths.
3. Plumb delivery_rate_mbps end-to-end (the field-plumbing checklist: NetworkLogEntry → v2_adapter → forwarder → ClickHouse schema → dashboard) so it's queryable and chartable alongside the existing throughput series.
4. Open a focused PR (like feat(proxy): single-owner group-pattern fan-out + driven-slave shaping UI #849).

Why it matters

This gives the honest server-side shaped-rate signal the ABR/startup work kept lacking — directly useful for confirming throttle accuracy and diagnosing the init/small-file over-read without trusting bytes_out/transfer_ms.

🤖 Generated with Claude Code

[jonathaneoliver]

Points: 8 (Fibonacci)
Effort: ~2–3 days
Priority: P3 — someday/queue; sidestep (mbps_transfer_rate) covers active needs and the ABR investigation it supports is currently parked
Value: medium — honest server-side shaped-rate signal fixes a recurring over-reporting confound in throttle/ABR work, but has no direct user-facing impact
Rationale: The WIP kernel (tcpi_delivery_rate sampling + build-tag split) is already written and pushed on fix/network-log-delivery-rate. The remaining work is dominated by the 8-layer field-plumbing chore (NetworkLogEntry → v2_adapter → forwarder → ClickHouse schema + migration → dashboard charting) plus rebasing off the stale #823 base and verifying the tcpConnFromContext helper compiles cleanly on linux. Each of those layers is individually small but the chain is long — hence 8 rather than 5. No priority/value mismatch: P3 + medium is coherent for genuine-but-non-urgent tooling debt.

[jonathaneoliver]

Reconciled onto current dev (e38a12a): rebased the WIP commit clean — no conflicts, the additive edits slotted straight into dev's main.go/v2_adapter.go. Builds on both macOS (stub) and linux (real getsockopt(TCP_INFO) path); tcpConnFromContext already exists in dev (cmd/server/rtt.go:49). Branch fix/network-log-delivery-rate force-pushed to the new base (b2d10bf). Remaining work is just the delivery_rate_mbps plumbing (forwarder -> ClickHouse -> dashboard) + the PR.