[Discussion] WekaTraceReplay: runs must complete every session (can take 9+ hours on real traces) and the drain-down tail skews throughput metrics

[yangligt2]

Summary

Two related observations about the weka_trace_replay load type, opening this for discussion:

1. A stage only finishes when all loaded sessions run to completion - there is no request-count or duration-based exit. On real traces this means runs measured in many hours, and without clear documentation users may conclude the benchmark is hung.
2. Because the session pool is a fixed set with no backfill at the end of the corpus, concurrency decays during the drain-down phase, and throughput is averaged over the full wall-clock span with no steady-state windowing - so the reported numbers understate what the system can do at target concurrency.

1. Exit condition and expected run duration

The session-stage loop exits only when every dispatched session has completed (or failed): run_session_stage in inference_perf/loadgen/load_generator.py (the "All N sessions completed" check). num_sessions defaults to all loaded sessions.

Some napkin math on a real trace file we profiled:

Stat	Value
traces (sessions)	393
main turns	28,444
subagent inner requests	39,822
total model requests	68,266
total input tokens	~6.89B
total output tokens	~58.7M

Even on a very powerful serving system - e.g. 32 GB200 chips sustaining ~200K tokens/sec with high KV-cache hit rates - the pure token-throughput floor is:

(6,891,228,864 + 58,728,807) tokens / 200,000 tok/s ≈ 34,750 s ≈ 9.7 hours

And that is a floor, not an estimate: turns within a session are strictly sequential (~72 main turns per session on average here), and the replay preserves recorded inter-turn delays via wait_ms (idle gaps capped at trace_idle_gap_cap_seconds, default 60s, which still adds up across tens of thousands of turns).

Nothing in the current docs/weka_trace_replay.md warns about this. A user pointing this datagen at a real trace with default config gets a run that looks stuck for hours.

2. Drain-down tail skews throughput results

should_start_next_session() only backfills the pool while pending sessions remain. Once the corpus is exhausted, active sessions monotonically decrease (e.g. 80 -> 20 -> 1), and since main turns are sequential, a tail session often contributes only ~1 in-flight request. Meanwhile the report computes throughput over the full span from first request start to last request end (summarize_requests in inference_perf/reportgen/base.py):

total_time = max(x.end_time for x in metrics) - min(x.start_time for x in metrics)
output_tokens_per_sec = sum(output_tokens) / total_time

There is no steady-state windowing or warmup/drain-down trimming, so one long tail session (session durations in agentic traces are heavy-tailed) dilutes *_tokens_per_sec and requests_per_sec for the whole stage. Latency percentiles are less affected - if anything the tail sees an unloaded system and improves them.

Related interaction: if num_sessions <= concurrent_sessions, the concurrency cap is inert and the entire run is effectively ramp-down.

Ideas for discussion

• Docs (cheap, high value): add a "Run duration expectations" section to docs/weka_trace_replay.md with the math above as a worked example, and document the existing bounding knobs: num_sessions, stage timeout (noting the stage reports FAILED but metrics are kept), trace_idle_gap_cap_seconds, num_dataset_entries.
• Steady-state windowing: optionally compute throughput over a trimmed window (e.g. exclude ramp-up/drain-down, or timestamp-based post-filtering guidance using per_request_lifecycle_metrics).
• Better progress signal: periodic log line with completed/active sessions and a rough ETA so long runs do not look hung.
• Session recycling / backfill option: keep concurrency saturated by reusing corpus sessions (with unique-session ID injection) until a duration or request budget is hit.

Happy to contribute the doc update and discuss the rest.

/kind documentation
/kind feature

[yangligt2]

cc: @lenadankin

[lenadankin]

One thought on the framing: I think most of the observations here are really about session replay rather than specifically Weka trace replay.

Weka traces are one source of replay sessions, but the execution model (session lifecycle, completion when all sessions finish, concurrency behavior, throughput calculation, etc.) comes from the shared replay infrastructure. OTEL replay uses the same session replay machinery, so many of the discussion points here would apply there as well, even if today's OTEL traces tend to be smaller.

Framing this as a discussion about session replay would also make any improvements (documentation, progress reporting, steady-state windowing, alternative exit conditions, etc.) naturally benefit both Weka and OTEL replay.

On "Exit condition and expected run duration": I think it's always a good idea to improve the documentation if there are gaps. The more interesting discussion to me is the semantics of a timeout.

Right now, session replay is designed around replaying a set of sessions to completion, and the main knobs (num_sessions, etc.) define what gets replayed rather than how long the stage should run. In that model, hitting stage_timeout is naturally treated as an incomplete replay.

If we'd like to support a different benchmarking mode—where the goal is to occupy the system for a target duration (e.g., 30 minutes or 2 hours) and consider that a successful run—that feels like a separate design discussion. It would likely require additional semantics beyond the current timeout, for example replaying sessions until the time budget is exhausted (potentially looping over the corpus), or other duration-oriented stopping criteria.

On "Drain-down tail skews throughput results", "Steady-state windowing", and "Session recycling": I think these are closely related to the point above. The current design is session-oriented: the workload is defined by a set of sessions, so it's expected that concurrency decreases as the finite corpus is exhausted, and the reported throughput reflects replaying that corpus end-to-end.

If we decide to support a time-oriented (or request-oriented) benchmark instead, then keeping concurrency saturated by recycling sessions would naturally become part of that execution model. In that case, steady-state throughput would be the more relevant metric. To me, this is less about how throughput is computed today and more about whether we want to introduce an additional benchmarking mode with different semantics.

On "Better progress signal": I agree that better progress reporting would be valuable, especially for long-running session replays. One minor thought is that I'd be careful about reporting an ETA, since replay session durations can be highly variable and estimates may be misleading. Reporting objective progress (completed/total sessions, active sessions, elapsed time, etc.) might be more robust.

Overall, I don't necessarily see these as alternatives to the current design. The current session-oriented replay semantics are useful in their own right, and there may be value in supporting an additional duration-oriented replay mode alongside them, depending on the use cases we want to optimize for.

[lenadankin]

adding to my previous comment: WekaTraceReplayConfig introduces trace_idle_gap_cap_seconds param while SessionReplayConfig (which is the super class) has max_wait_ms. To my understanding, these two are quite similar so this is an example of why we should aim to distinguish between features that are specific to the source of the traces, and these that are relevant to the general session replay.

[shaigrt]

I plan to improve documentation and reporting of the existing timeout mechanism. In addition I will work on a windowing computation.

[lenadankin]

@achandrasekar