Skip to content

Commit 80b518b

Browse files
committed
audio: tensorflow: tune: add single-speaker Piper keyword generator
The existing sof_tflm_generate_keyword_dataset.sh is hard-wired to the English multi-speaker piper-sample-generator (LibriTTS-R). It cannot train a positive class in any other language because the phonemizer inside that checkpoint only knows English graphemes. Add a companion script that drives the regular piper-tts package against any single-speaker voice from rhasspy/piper-voices (Finnish fi_FI-harri, Swedish sv_SE-nst, etc.). To compensate for the missing speaker-embedding axis it (a) randomizes Piper noise-scale / noise-w per utterance while cycling several length-scale values and (b) fans each synthesized clip out into sox pitch-cents plus pitch-preserving tempo copies. Output layout, IR augmentation hook, and gain jitter match the English generator so sof_tflm_train_pipeline.sh consumes it unchanged. Update the tune README with a language-agnostic pipeline table entry, a non-English setup section. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
1 parent ec69c89 commit 80b518b

2 files changed

Lines changed: 652 additions & 2 deletions

File tree

src/audio/tensorflow/README.md

Lines changed: 79 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -301,8 +301,9 @@ chains four steps into one command:
301301

302302
| Step | Script | What it does |
303303
|------|--------|-------------|
304-
| 0 | [sof_tflm_generate_keyword_dataset.sh](./tune/sof_tflm_generate_keyword_dataset.sh) | Synthesize `<label>/*.wav` for one keyword with Piper-TTS + augmentation (rerun once per keyword) |
305-
| 1 | [sof_tflm_prepare_silence_unknown.sh](./tune/sof_tflm_prepare_silence_unknown.sh) | Slice `silence/` + sample `unknown/` from Speech Commands v2 |
304+
| 0a | [sof_tflm_generate_keyword_dataset.sh](./tune/sof_tflm_generate_keyword_dataset.sh) | Synthesize `<label>/*.wav` for one English keyword with `piper-sample-generator` (multi-speaker LibriTTS-R) + augmentation |
305+
| 0b | [sof_tflm_generate_keyword_dataset_piper_tts.sh](./tune/sof_tflm_generate_keyword_dataset_piper_tts.sh) | Same output layout, but for any single-speaker Piper voice (Finnish, Swedish, Hungarian, German, ...). Recovers speaker diversity via per-utterance prosody randomization + sox pitch/tempo perturbation |
306+
| 1 | [sof_tflm_prepare_silence_unknown.sh](./tune/sof_tflm_prepare_silence_unknown.sh) | Slice `silence/` + sample `unknown/` from Speech Commands v2 (English is fine as a negative-class source even for non-English keywords) |
306307
| 2 | [sof_mfcc_extract_features.sh](./tune/sof_mfcc_extract_features.sh) | Emit SOF mel40 features via `sof-testbench4` |
307308
| 3 | [sof_tflm_train.py](./tune/sof_tflm_train.py) | Train `tiny_conv`, int8-quantize, write `.tflite` + C array |
308309

@@ -400,6 +401,82 @@ PIPER_REPO=~/git/piper-sample-generator \
400401
`PIPER_REPO` is required because upstream ships the `piper_train`
401402
package only in the git tree, not in the PyPI wheel.
402403

404+
### 1b. Non-English keywords: single-speaker Piper voices
405+
406+
`piper-sample-generator` only ships the English `en_US-libritts_r-medium`
407+
multi-speaker checkpoint. For a keyword in another language use
408+
[sof_tflm_generate_keyword_dataset_piper_tts.sh](./tune/sof_tflm_generate_keyword_dataset_piper_tts.sh),
409+
which drives the regular `piper-tts` package against any voice from the
410+
[rhasspy/piper-voices](https://huggingface.co/rhasspy/piper-voices) set
411+
(Finnish `fi_FI-harri`, Swedish `sv_SE-nst`, Hungarian `hu_HU-anna`,
412+
German `de_DE-thorsten`, etc.). Because those voices are single-speaker,
413+
this variant recovers diversity by (a) randomizing Piper's
414+
`--noise-scale` / `--noise-w` per utterance while cycling several
415+
`--length-scale` values, and (b) fanning each synthesized clip out into
416+
`PERTURB_PER_UTT` sox copies with a random pitch shift in cents and a
417+
pitch-preserving `tempo -s` factor.
418+
Set up the venv once:
419+
420+
```bash
421+
python3 -m venv ~/venvs/piper-tts
422+
source ~/venvs/piper-tts/bin/activate
423+
pip install --upgrade pip
424+
pip install piper-tts
425+
```
426+
427+
Fetch one voice per language (example: Finnish, Swedish, Hungarian):
428+
429+
```bash
430+
BASE=https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0
431+
mkdir -p ~/git/piper-voices/fi_FI ~/git/piper-voices/sv_SE ~/git/piper-voices/hu_HU
432+
cd ~/git/piper-voices/fi_FI
433+
wget "$BASE/fi/fi_FI/harri/medium/fi_FI-harri-medium.onnx"
434+
wget "$BASE/fi/fi_FI/harri/medium/fi_FI-harri-medium.onnx.json"
435+
cd ~/git/piper-voices/sv_SE
436+
wget "$BASE/sv/sv_SE/nst/medium/sv_SE-nst-medium.onnx"
437+
wget "$BASE/sv/sv_SE/nst/medium/sv_SE-nst-medium.onnx.json"
438+
cd ~/git/piper-voices/hu_HU
439+
wget "$BASE/hu/hu_HU/anna/medium/hu_HU-anna-medium.onnx"
440+
wget "$BASE/hu/hu_HU/anna/medium/hu_HU-anna-medium.onnx.json"
441+
```
442+
443+
Generate the Finnish "piparkakku" dataset:
444+
445+
```bash
446+
source ~/venvs/piper-tts/bin/activate
447+
MAX_SAMPLES=1000 \
448+
PIPER_VOICE=~/git/piper-voices/fi_FI/fi_FI-harri-medium.onnx \
449+
PIPER_REPO=~/git/piper-sample-generator \
450+
./sof_tflm_generate_keyword_dataset_piper_tts.sh \
451+
--keyword piparkakku ~/wov/wavs
452+
deactivate
453+
```
454+
455+
`PIPER_REPO` is optional here: if set, the same IR-convolution +
456+
volume-jitter augment stage the English pipeline uses runs after
457+
synthesis; if unset, the pipeline falls back to a plain 16 kHz resample
458+
plus the built-in gain jitter. `PIPER_TTS_VENV` is an alternative to
459+
sourcing `activate` yourself — the script only auto-activates when the
460+
venv path actually contains `bin/activate`.
461+
462+
Mixed-language multi-keyword models work by pairing `--keyword` with
463+
`--voice` positionally:
464+
465+
```bash
466+
source ~/venvs/piper-tts/bin/activate
467+
MAX_SAMPLES=1000 \
468+
./sof_tflm_generate_keyword_dataset_piper_tts.sh \
469+
--keyword piparkakku --voice ~/git/piper-voices/fi_FI/fi_FI-harri-medium.onnx \
470+
--keyword pepparkaka --voice ~/git/piper-voices/sv_SE/sv_SE-nst-medium.onnx \
471+
--keyword mézeskalács --voice ~/git/piper-voices/hu_HU/hu_HU-anna-medium.onnx \
472+
~/wov/wavs
473+
deactivate
474+
```
475+
476+
Everything downstream (silence/unknown from Speech Commands v2, feature
477+
extraction, training) is identical to the English recipe — Speech
478+
Commands v2 English clips remain a valid negative source.
479+
403480
### 2. End-to-end pipeline: silence/unknown → features → train → quantize
404481

405482
The remaining three steps run in one command. From

0 commit comments

Comments
 (0)