Research: streaming QoE 'health metric' models — implementation, justification, and fitness for our LL-live rig

[jonathaneoliver]

Context

While exploring whether we could run an autonomous characterization loop over the sweep rig (issue #772) — tuning test-config parameters rather than code, in the spirit of karpathy/autoresearch — the question arose: should we reduce each streaming session to a single "health" scalar we could then optimize?

This issue captures a deep-research pass (24 sources, 108 claims extracted, 25 adversarially verified: 24 confirmed / 1 refuted) into how existing streaming-QoE "health metrics" are actually implemented and justified, and what that means for our low-latency-live rig, which today emits a severity-tagged label vector rather than a scalar.

Bottom line: the evidence backs deferring the scalar for characterization and keeping the label vector. A scalar remains a defensible opt-in optimization target later — but with transparent, swappable weights, validated against our own dispositioned plays.

---

Two model families, distinguished by how they're justified

Family A — perceptual, subjective-MOS-validated (ITU-T P.1203 / P.1204)

Modular: per-1s video Pv (P.1203.1) + per-1s audio Pa (P.1203.2) + integration Pq (P.1203.3) → 5-point MOS. Stalling is a pathway inside Pq (not a separate module).

Pv video-quality decomposition — D = max(min(Dq + Du + Dt, 100), 0) on a 0–100 impairment scale. Coefficients (Raake et al., QoMEX 2017):

degradation	formula	coefficients
compression Dq	q̂ = q1 + q2·exp(q3·quant)	q1=4.66, q2=−0.07, q3=4.06
upscaling Du	Du = u1·log10(u2·(scaleFactor−1)+1)	u1=72.61, u2=0.32
temporal Dt	jerkiness; none perceived above 24 fps	t1=30.98, t2=1.29, t3=64.65

Input modes: 0 (metadata only) → 1 (+frame types/sizes, DRM-safe) → 2 (+~2% QP) → 3 (full QP). Only quant is mode-specific; a higher mode was standardized only if statistically-significantly better.

Validation corpus: >600 subjects, ~5,000 sequences, 26 databases (13 train / 13 validation), 5-point ACR.

Fitness caveats (the load-bearing ones for us):

• P.1203 is scoped to 1–5 min sessions; P.1204.3 to 5–10s bitstreams (optimal ~8s). Sub-second LL-live is outside the validation envelope.
• P.1203's short-term per-second video model was not trained on short-term ground truth — it was reverse-engineered from long-session integral MOS.
• The full model is a paid ITU standard; the open-source itu-p1203 implementation covers Mode 0 only.

Family B — additive engineering scalars (the ABR-research lineage)

All share the skeleton QoE = Σ quality − λ·Σ|switches| − µ·Σ rebuffer − µ_s·startup. Critical finding: the weights are free author-preference parameters — NOT derived from subjective MOS or engagement regression.

model	quality term	weights	how weights chosen
Yin/MPC 2015	q(R) identity, ladder {350,600,1000,2000,3000} kbps	λ=1, µ=µ_s=3000 (1s rebuffer ≡ −3000 kbps)	author preference; 3 profiles (Balanced / Avoid-Instability λ=3 / Avoid-Rebuffer µ=6000)
Pensieve 2017 · QoE_lin	q(R)=R (Mbps)	µ=4.3	hand-set (= max bitrate)
Pensieve · QoE_log	q(R)=log(R/Rmin)	µ=2.66	hand-set (= ln(4.3/0.3))
Pensieve · QoE_hd	HD-favoring map (0.3→1 … 4.3→20)	µ=8	hand-set
SODA 2024 (Prime Video)	mean utility v̄	β=10 (rebuffer), γ=1 (switch)	"to reflect the high importance of minimizing rebuffering"
LoL+ / Bentaleb 2022 (LL-DASH/HLS)	explicit 5-term additive incl. latency	—	closest domain match to our rig

RobustMPC self-reported +15% / +10% / +5% normalized-QoE (broadband / cellular / synthetic); prior non-MPC algorithms reached only 60–70% of offline-optimal.

---

The killer scalarization result (SODA, SIGCOMM 2024)

Optimizing RobustMPC's linear objective is itself the cause of pathology: it produced 29 rebuffer events in 200s because the scalar deemed tolerating rebuffers (to avoid bitrate switches) "optimal." And even a 20× rebuffer penalty only shortened the rebuffers — never eliminated them.

A single weighted scalar can make a pathological trade-off optimal, and weight-tuning does not fix it.

This is the strongest single reason to keep the multi-dimensional label vector for characterization — collapsing to one number destroys exactly the signal we need to tell distinct failure modes apart. (An adjacent claim that SODA "explicitly abandons scalarization" was refuted 0-3 — SODA still reports a QoE scalar for evaluation.)

---

Recommendation for our rig

1. Characterization loop → keep the severity-tagged label vector. Do not force a scalar; it preserves the multi-modal signal (the exact thing SODA's pathology demonstrates a scalar loses).
2. Do not adopt P.1203/P.1204 — wrong regime (validated on 1–5 min / 5–10s, not sub-second LL-live).
3. If/when we want an optimization loop, make the health scalar an opt-in target with transparent, swappable weights, and validate it against our already-dispositioned plays (refuted false-positives must score healthy; a real stall_frozen must score unhealthy). Domain-matched starting point is LoL+'s 5-term latency-aware additive, not the standards.
4. Same prerequisite either way: the harness confounds (worker cold-start, sim wedge, ABR jitter — see the sweep-queue gaps finding) would poison a health-optimizer worse than the label loop. Fix those first.

---

Honest coverage gaps (fetched but NOT verified — needs a follow-up pass)

The adversarial filter's top-25 budget dropped three sub-topics; sources were fetched but no claims survived verification, so treat these as unresearched:

• Engagement regression (Dobrian 2011, Balachandran 2013, YouSlow) — the "rebuffer-ratio-drives-abandonment" coefficients and the empirical watch-time link.
• Puffer/Fugu's SSIM-dB + stall-time specifics and what their randomized-experiment data said matters most. (Fugu's per-chunk objective QoE = Q(Kᵢ) − λ·|Q(Kᵢ)−Q(Kᵢ₋₁)| − µ·max{T−B, 0}, Q in SSIM-dB, surfaced in search but not in the verified set.)
• Formal Pareto / multi-objective QoE — represented only obliquely via SODA.
• Specific PCC/RMSE-vs-MOS accuracy numbers for P.1203/P.1204 (the ">600 subjects" figure is corpus size, not a reported accuracy).

---

Sources (verified, all 3-0)

ITU-T standards: itu-p1203 reference impl · Raake et al. QoMEX 2017 (P.1203.1 video model) · ACM MMSys P.1204 paper · bitstream-based models project page

Additive ABR scalars: Yin et al. SIGCOMM 2015 (MPC) · Pensieve SIGCOMM 2017 · Pensieve tech report · SODA SIGCOMM 2024

LL-live / engagement (fetched, unverified): Bentaleb et al. LoL+ ACM MM 2022 · Puffer/Fugu NSDI 2020 · Dobrian et al. SIGCOMM 2011 · Balachandran et al. SIGCOMM 2013

---

Relates to #772 (sweep/characterization loop). Not a bug — a design/reference note to inform whether we ever adopt a QoE health scalar.

🤖 Generated with Claude Code

[jonathaneoliver]

Points: 3 (Fibonacci)
Effort: ~half a day (4-6 hrs)
Priority: P3 — someday. This is a captured reference note; its core recommendation (defer the QoE scalar, keep the label vector) is already settled and actionable with zero further work — nothing here blocks or gates current characterization work (#772).
Value: low — the note itself is valuable as a decision record, but the only open work is the follow-up research pass on the 3 flagged coverage gaps (engagement regression, Puffer/Fugu SSIM-dB specifics, formal Pareto/multi-objective), which only matters if/when a health scalar becomes an actual roadmap item. Speculative/completeness value, not near-term impact.
Rationale: Sizing the remaining actionable work (the follow-up research pass), not the already-written note — re-running an adversarial-verification research pass on 3 narrower sub-topics is smaller than the original 24-source/108-claim pass but still real desk research (search, read, extract, cross-check), landing at 3 points / half a day rather than trivial. The optional opt-in-scalar implementation mentioned in the note is explicitly deferred and has a hard prerequisite (fix the sweep-queue harness confounds first per #772) — it isn't sized here; it'd warrant its own issue + estimate once actually scheduled.