Running notes for performance work on the HIP runtime. Keep entries factual: what changed, how it was measured, and what we learned.
Added a first Base-model voice-clone path that loads precomputed x-vector-only
prompt JSON artifacts exported from the Python reference. This reuses the existing
streaming-style text prefill and replaces the CustomVoice speaker-token embedding
with the exported ref_spk_embedding. ICL/ref-code prompting and Rust-side
reference-audio encoding are still future work.
Prompt export used a local 10-second reference clip against the cached 1.7B Base snapshot.
Rust smoke command used cargo run --profile timing --bin hip-custom-voice-generate
with the 1.7B Base model, max_frames=80, Qwen-default sampling, and
text_lookahead_tokens=8.
frames=70
ended_by_eos=true
audio_seconds=5.600000
generation_seconds=2.952218
decode_seconds=0.327853
inference_seconds=3.280071
generation_rtf=0.527182
decode_rtf=0.058545
inference_rtf=0.585727
output_wav=/tmp/opencode/voice-clone-xvector-rust.wav
Discovery: Base model loading works with the existing 1.7B model assumptions, and
the x-vector-only prompt path generates valid audio with EOS termination. For exact
reference parity, use text_lookahead_tokens=1; the default 8 keeps the current
streaming-quality tradeoff.
Added python-reference/qwen3_tts_reference.py voice-clone to export Base-model
voice-clone codes from a precomputed x-vector prompt JSON, plus
scripts/qwen3-hip-voice-clone-parity.sh to rerun the Python export and compare
Rust generation with sampling disabled.
Measured with cached 1.7B Base, max_new_tokens=12, streaming-style Python input,
greedy semantic and acoustic generation, repetition penalty 1.0, and Rust
text_lookahead_tokens=1.
python_frames=11
rust_frames=11
codes_match=true
Discovery: the Rust x-vector-only prefill path matches Python exactly for the tested deterministic Base-model rollout.
Measured with cargo run --release --bin hip-e2e-bench using 39 generated frames,
3 measured iterations, and 1 warmup iteration.
| Model | Generation RTF | Decode RTF | E2E RTF |
|---|---|---|---|
| 0.6B | 0.330165 |
0.102253 |
0.432418 |
| 1.7B | 0.414669 |
0.102953 |
0.517622 |
Added timing output to hip-custom-voice-generate so one-shot generation reports:
- model load time
- code generation time
- optional audio decode time
- optional WAV write time
- audio duration
- generation/decode/inference RTF when audio is decoded
This made quick command-line checks easier, but it is not a benchmark harness because single runs include capture/warmup effects.
Tried graph replay in the production generation path. The first acoustic token stays eager; the repeated remaining acoustic-group loop is captured once and replayed for later frames.
Measured with cargo run --release --bin hip-e2e-bench.
| Model | Before E2E RTF | After E2E RTF | Change |
|---|---|---|---|
| 0.6B | 0.432418 |
0.421308 |
~2.6% faster |
| 1.7B | 0.517622 |
0.505415 |
~2.4% faster |
0.6B after graph replay:
generation_rtf=0.319113, decode_rtf=0.102195, e2e_rtf=0.421308
Discovery: HIP Graph replay helps, but only modestly. Launch overhead is not the main bottleneck in the current generation path.
Tried to use rocprofv3, but the installed ROCm image does not include the profiler
CLI in /opt/rocm/bin or /opt/rocm-7.2.4/bin. The install currently has profiler
registration libraries only.
Because of that, added a synchronized stage profile to hip-rollout-bench for coarse
breakdown without external profiler tooling.
Measured 0.6B, 39 frames, release build, fixture-backed rollout.
prefill_seconds=0.009874
prepare_prefix_seconds=0.001644
code_predictor_seconds=0.712256
build_step_input_seconds=0.001654
talker_decode_seconds=0.269924
total_seconds=0.995351
| Stage | Share |
|---|---|
| CodePredictor | 71.56% |
| Talker decode | 27.12% |
| Prefill | 0.99% |
| Prepare prefix | 0.17% |
| Build step input | 0.17% |
Discovery: generation is dominated by HipCodePredictor, not talker decode and not
prefix/step-input glue.
Measured graph replay on the lower-level DecodeStepStack path.
5-layer CodePredictor-sized stack: graph_speedup=1.067
28-layer talker-sized stack: graph_speedup=1.056
Discovery: even at the transformer-stack level, graph replay only gives about 5-7%. This matches the small end-to-end gain and points away from launch overhead as the main bottleneck.
The next serious optimization target should be CodePredictor math efficiency, especially the many single-token/small-matrix operations used while predicting the 15 acoustic groups per frame.
Likely next experiments:
- replace small
m=1rocBLAS SGEMMs with custom GEMV kernels - fuse GEMV-adjacent operations where practical, such as bias/norm/argmax pieces
- profile CodePredictor sub-stages more deeply if ROCm profiler tooling becomes available
- evaluate fp16/bf16 paths later, after exact f32 path remains stable
Added a diagnostic profile path for HipCodePredictor and called it from
hip-rollout-bench. Measured 0.6B, 39 frames, release build.
code_predictor_seconds=0.813764
prefix_projection_seconds=0.000007
stack_prefill_seconds=0.057901
first_logits_seconds=0.003347
first_token_seconds=0.001633
remaining_projection_seconds=0.000094
remaining_stack_seconds=0.679855
remaining_logits_seconds=0.046197
remaining_token_seconds=0.022536
output_copy_seconds=0.001217
Discovery: CodePredictor time is dominated by the transformer stack, especially the remaining acoustic-group decode steps. LM-head logits and token/embedding glue are secondary.
Added gemv-bench to compare rocBLAS m=1 SGEMM against a simple custom one-block
per-output-column GEMV kernel.
Shape (n, k) |
rocBLAS Mean | Custom GEMV Mean | Result |
|---|---|---|---|
(1024, 1024) |
27.833 us |
94.379 us |
custom slower |
(2048, 1024) |
32.386 us |
158.965 us |
custom slower |
(6144, 1024) |
164.140 us |
323.879 us |
custom slower |
Discovery: a naive f32 GEMV replacement is not competitive with rocBLAS. If we replace rocBLAS, it needs a more specialized/tiled kernel, not the simple reduction kernel.
Added a diagnostic DecodeStepStack::decode_step_profiled path and printed it from
decode-step-graph-bench.
Before removing decode identity permutes, CodePredictor-sized 5-layer stack:
total_seconds=0.002605
input_norm=0.000147
qkv_gemm=0.000350
qk_layout_cache=0.000268
attention=0.000205
output_gemm_residual=0.000399
post_norm=0.000139
gate_up_gemm=0.000392
swiglu=0.000128
down_gemm_residual=0.000548
final_copy=0.000029
Discovery: decode-step cost is spread across GEMMs, with the MLP down projection and attention output projection among the larger buckets. The non-GEMM q/k layout/cache bucket was also worth checking.
In decode-step paths, permute_bshd_to_bhsd is an identity when steps=1. Removed
the q/k/v permute launches from eager, stream/graph, and profiled decode-step paths;
prefill paths still keep the real layout transform.
Measured with cargo run --release --bin hip-e2e-bench.
| Model | Before E2E RTF | After E2E RTF | Generation RTF After |
|---|---|---|---|
| 0.6B | 0.421308 |
0.412695 |
0.309310 |
| 1.7B | 0.505415 |
0.497384 |
0.393780 |
Discovery: removing the no-op permutes produced another small but real win, roughly
2% e2e. Combined with CodePredictor graph replay, 0.6B moved from about 0.432 to
about 0.413 e2e RTF.
Tested removing the conservative one-token D2H copy before CodePredictor graph replay.
Quick parity passed, but 0.6B e2e did not improve (0.413102 RTF in the test run), so
the explicit sync copy was kept for safer stream ordering.
Compared Python WAV output against HIP output for:
She said she would be here by noon.
speaker=Ryan
language=English
The HIP path is not sample-rate slow: both Python and HIP write 24000 Hz WAVs.
The apparent differences came from generation settings, not WAV sample rate.
| Output | Frames | Duration | Notes |
|---|---|---|---|
| Python default high-level | 48 |
3.84s |
non-streaming, sampled subtalker, repetition penalty |
| Python non-streaming argmax | 45 |
3.60s |
non-streaming, deterministic subtalker |
| Python streaming argmax | 39 |
3.12s |
streaming mode, repetition penalty 1.05 |
Python streaming argmax, repetition penalty 1.0 |
39 |
3.12s |
matches HIP default codes |
| HIP default | 39 |
3.12s |
streaming-style, argmax, repetition penalty 1.0 |
Discovery: the project can keep streaming mode as the primary behavior. Python parity
for this mode requires making semantic repetition penalty configurable. Added
GenerateOptions::repetition_penalty, initially defaulting to 1.0 so existing
streaming argmax behavior and parity stayed unchanged. The public default was later
changed to 1.05 after matching Python streaming argmax with repetition penalty;
parity scripts explicitly pass 1.0 for the original no-penalty fixtures.
Validation:
HIP repetition_penalty=1.05 matched Python streaming argmax codes exactly.
python_stream_1p05 duration=3.12s rms=0.050849 peak=0.503632
hip_stream_1p05 duration=3.12s rms=0.050840 peak=0.503632
GenerateOptions::default() now follows the Qwen TTS wrapper generation defaults while
keeping this project's streaming path:
do_sample=true
top_k=50
top_p=1.0
temperature=0.9
repetition_penalty=1.05
subtalker_dosample=true
subtalker_top_k=50
subtalker_top_p=1.0
subtalker_temperature=0.9
seed=0
The parity script explicitly passes do_sample=false, subtalker_dosample=false, and
repetition_penalty=1.0 because the stored fixtures test the deterministic greedy path.
Measured hot e2e RTF for the streaming/Qwen-default path with 3 measured iterations and 1 warmup iteration, excluding model load:
| Runtime | Frames | Audio Seconds | Generation RTF | Decode RTF | E2E RTF |
|---|---|---|---|---|---|
HIP HipTtsEngine |
34 |
2.72 |
0.440904 |
0.097584 |
0.538488 |
| Python reference | 46 |
3.68 |
1.755115 |
0.150108 |
1.905224 |
These use the same streaming/Qwen-default generation settings, but sampling means the exact output lengths can differ unless the sampling RNGs are matched exactly.
Added optional ROCTx ranges behind QWEN3_HIP_ROCTX=1 and profiled with
rocprofv3 --marker-trace --kernel-trace --memory-copy-trace --kernel-rename.
The installed rocprofv3 needed libhsa-amd-aqlprofile64.so.1; a Torch wheel had
the same SONAME under an unversioned filename, so a temporary symlink under
/tmp/opencode/rocm-prof-libs was used for profiling.
Engine profile, 0.6B Qwen-default sampling, 1 warmup and 1 measured iteration:
| Renamed kernel range | Kernel time share |
|---|---|
engine.rollout.code_predictor |
54.75% |
engine.rollout.talker_decode |
19.95% |
codec.decoder.causal_conv |
12.38% |
codec.decoder.transconv |
11.01% |
engine.rollout.talker_prefill |
1.04% |
Codec-focused profile, codes generated once, 1 warmup decode and 3 measured decodes:
| Renamed kernel range | Kernel time share |
|---|---|
engine.rollout.code_predictor |
31.65% |
codec.decoder.causal_conv |
28.87% |
codec.decoder.transconv |
25.62% |
engine.rollout.talker_decode |
11.52% |
The codec benchmark still includes one code-generation pass, but repeated decode makes
the codec decoder kernels clear. codec.decoder.transconv is only 16 calls in the
codec-focused profile and averages 28.84 ms per dispatch, while
codec.decoder.causal_conv averages 5.00 ms over 104 dispatches. The next codec
optimization target should be codec_transconv1d_channels_f32, followed by
codec_causal_conv1d_dilated_f32.
Tried optional tiled/reduction kernels for decoder transposed convolution and causal
convolution. They parallelized each output across 16 reduction lanes and 16 output
channels per block, and are available only when QWEN3_HIP_TILED_CODEC=1.
Correctness passed full waveform parity:
max_abs=0.000002561, mean_abs=0.000000056
Performance regressed, so the original kernels remain the default:
| Path | Decode Mean | Decode RTF | E2E RTF |
|---|---|---|---|
| Original default | 0.284988s |
0.104775 |
0.542482 |
| Tiled experiment | 0.466068s |
0.171349 |
0.607684 |
Discovery: a simple block-level reduction adds too much scheduling/reduction overhead. The next transposed-convolution attempt should use a GEMM-style formulation with prepacked per-phase weights, not one block per small output tile.
Reformulated decoder transposed convolution by phase. For output phase p, the
kernel has two contributing taps, p and p + stride:
output[t * stride + p] = input[t] @ W[p] + input[t - 1] @ W[p + stride] + bias
Weights are prepacked per phase at load time. Runtime decode transposes the input to time-major form, builds a one-frame-shifted copy, runs two rocBLAS GEMMs per phase, then scatters the summed phase output back to channel-major waveform-decoder layout.
Correctness passed waveform parity:
max_abs=0.000002727, mean_abs=0.000000064
Release benchmark, Qwen-default sampling, 0.6B, 34 frames:
| Path | Decode Mean | Decode RTF | E2E RTF |
|---|---|---|---|
| Direct transconv default before | 0.285582s |
0.104993 |
0.542482 |
| GEMM transconv | 0.184353s |
0.067777 |
0.503268 |
GEMM transconv is now the default. Set QWEN3_HIP_DIRECT_TRANSCONV=1 to use the old
direct kernel path for comparison.
Measured with rocprofv3 --marker-trace --kernel-trace --memory-copy-trace --kernel-rename
using a delayed hot-window collection after model load. The benchmark used 0.6B,
Qwen-default sampling, 1 warmup and 1 measured iteration.
Benchmark wall time:
generation_mean=1.388315s
decode_mean=0.183308s
e2e_mean=1.571623s
Profiler kernel time approximation:
| Stage | Wall Time | GPU Kernel Time | Estimated Host Time |
|---|---|---|---|
| Generation | 1.388s |
0.965s |
0.423s |
| Decode | 0.183s |
0.144s |
0.039s |
| E2E | 1.572s |
1.109s |
0.463s |
Approximate e2e split:
GPU kernels: 70.6%
CPU/host overhead: 29.4%
Largest host-side contributors from marker wall time minus renamed kernel time:
| Stage | Approx Host Time |
|---|---|
engine.prepare_text |
0.241s |
engine.rollout.code_predictor |
0.310s |
engine.rollout.talker_decode |
0.095s |
engine.decode_codes |
0.040s |
The CodePredictor and talker-decode host time is likely dominated by CPU-side sampling and GPU/CPU synchronization/copies. This reinforces GPU-side sampling as the next primary optimization target.
Added a device-side sampler for the Qwen-default sampling shape:
do_sample=true
top_k in 1..=256
top_p=1.0
Unsupported sampling settings still use the CPU sampler. Set QWEN3_HIP_CPU_SAMPLING=1
to force the old CPU path for comparison.
The first prototype used one GPU thread to do top-k selection and regressed badly:
e2e_rtf=0.796936
Replacing that with a shared-memory bitonic sort over the vocab made the sampled path faster than the CPU sampler:
| Path | Generation RTF | Decode RTF | E2E RTF |
|---|---|---|---|
| CPU sampling fallback | 0.437339 |
0.066594 |
0.503933 |
| GPU bitonic top-k sampling | 0.425911 |
0.066317 |
0.492228 |
This is a modest win, roughly 2.3% e2e, because the sampler removes large logits D2H/H2D traffic but adds many sampling kernel launches and still copies selected semantic tokens to host for loop control/EOS. The remaining large host-side items are still CodePredictor/talker orchestration and text preparation.
Extended CodePredictor graph replay to the sampled path when sampling can stay on GPU.
Group 0 remains eager; acoustic groups 1..14 are captured with projection, decode step,
logits, GPU top-k sampling, token store, embedding lookup, and residual add. Random
values are written into a stable device buffer before graph replay so graph kernel
arguments remain reusable. The captured graph is keyed by SamplingConfig, so changing
top-k/temperature recaptures instead of reusing stale scalar params.
Measured with hip-engine-bench, 0.6B, Qwen-default sampling, 5 measured iterations and
1 warmup iteration:
| Path | Generation RTF | Decode RTF | E2E RTF |
|---|---|---|---|
| GPU sampling, eager sampled CodePredictor | 0.425326 |
0.058447 |
0.483773 |
| GPU sampling, sampled CodePredictor graph replay | 0.419242 |
0.058644 |
0.477886 |
Discovery: sampled graph replay is correct and helps, but only modestly (~1.2% e2e). This matches earlier greedy graph replay results: launch overhead is not the dominant cost, but graph replay is still worth keeping on the default sampled path.