Vietnamese Translation

[thinhlpg]

🎯 Goal

• To have a realtime Vietnamese - English transaltion demo
• Current demo state: v0.2

Backlog

• ASR Quality and Speed (do some hacking like @new5558 did with existing checkpoint or finetune new model with better data?)
• Small bench voice for ASR for Vietnamese, Thai (low resource language), as suggested by @Yip-Jia-Qi in real life environment (coffeeshop, home, record by laptop mic, webcam mic, phone mic,...)
• VAD @thinhlpg or streaming @new5558 (I'm following the VAD path, the problem is that VAD sometime suck and missing some speech at the start of the speech)
• Serving model for optimial latency, currently took 6s from speech start to TTS (the demo wasn't optimized at all, lot of room for improvement). For context, Google speech translate demo 4 samples and all took exact 2 secs as I inspect manually https://youtu.be/hyXqcsWOONo?feature=shared)
• Traning experiemnts for translation: better data, hyperparameters?

How good is the commercial baseline for ASR?

Know your enemy - we want to beat the commercial ones, not just opensource

• Test Zalo Kiki (the ASR part).
• Test other popular options
  => Their reponse time, their output quality?

[thinhlpg]

250612

The original pipeline v0.1 is quite straightforward, but the lang detect in the prompt didn't work, which made the translation from Vietnamese to English fail.

I designed V0.2 pipeline, which seperate the language detection parts. It also train two Loras for two directions, which in my assumption will make the result better since each LoRA focus on only one direction.

I have a barebone demo for v0.2 right now (the speech part is not yet included), which already fixed the Vietnamese -> English translation issue, but the quality is still low (needs finetuning).

I will use Unsloth + LoRA. My plan is to have two adapters: one for English -> Vietnamese, one for Vietnamese -> English.

The two LoRAs can then be loaded with vllm (I tested the latency already—almost no additional latency compared to just a normal checkpoint, but the quality is better).

For the language detection module, I chose lingua-py (10/10 correct small testset) https://github.com/pemistahl/lingua-py. I also tested fast-langdetect, but the accuracy is only about 6/10.

To scale to multiple languages, we just need to finetune more LoRAs for each language pair. The language detect can be handled easily and added to the prompt.

Status now:

• Research and checked Vietnamese Resources ✅ (Things kinda stuck at 2022, nothing news after that)
• Check two best dataset avaiable for Vietnamese: PhoMT and mTet. Chose PhoMT because they have language label. ✅
• Tried all small baselines -> choose qwen3 1.7B (/no_think) or qwen2.5 1.5B (same quality, same latency) ✅

• Small test set to validate things ✅
• Two well crafted prompt for each direction ✅
• 10k data done ✅
• Training script done (still double-checking to ensure I didn't make any stupid mistakes) 🙏
• Simple demo with langdetect and lora ✅
• Tested lantency and performance for each part in v0.2 pipeline ✅ (text - to - text lantency is about 0.3s)

Today I will:

• Start the finetuning with 10k data from the PhoMT dataset (the least amount of data to have performance improvement according to this paper https://arxiv.org/abs/2409.03454).

• Eval script. Metrics include automated metrics in https://arxiv.org/abs/2409.03454 (COMET as main metric), human in the loop validation
• Eval results

🎯 Goal: have two LoRAs; when loaded, the translation quality should increase (both auto and manual check).
And most importantly, the translation from English to Vietnamese should be fluid and shouldn't contain any weird Chinese characters.

🙏 I currently place my code here https://github.com/[menloresearch/vietnamese-translation-resources-overview](https://github.com/menloresearch/vietnamese-translation-resources-overview) as an internal repo since my prefer way of working is to commit all local working state for quick and accuracte reproducibility in other environment such as Runpod. I will add the clean code to the main repository when it's ready.

[thinhlpg]

250613

• Finished training Lora, 4 checkpoints, all training eval curve goes up indicated good training progress
• Did some bug fixed in evaluation and training code
• Made quick text to text demo app, demo translation quality improvement
  
• Spent rest of the time with Jan-nano

[thinhlpg]

250616

• Convert trained models from full weight to Lora for Norapat to make demo
  • https://huggingface.co/janhq/vi-translation_v0.1_20250612-230103_Qwen3-1.7B_checkpoint-500_en-vi-v2
  • https://huggingface.co/janhq/vi-translation_v0.1_20250612-230103_Qwen3-1.7B_checkpoint-500_vi-en-v2
• Spent rest of the time with Jan-nano

[thinhlpg]

250617

• pair with @new5558 for Vietnamese -> English translation demo
• debug Vietnamese ASR quality issue

[thinhlpg]

250618

• debug Vietnamese ASR quality issue
  • Tested baselines: whisper-large-v3-turbo ✅, large-v3, phowhisper-large, NhutP/ViWhisper-medium
• demo Vietnamese -> English app
  • VAD+ASR -> Translation -> TTS (kokoro)
  • UI with Gradio
• demo video

Issues

• ASR speed
• ASR quality for Vietnamese in real world environment. Current best: whisper-large-v3

Bench reponse time - GroqAPI vs Local, averages of 5

Two demos - two appoarches

• VAD -> seperate speech segment -> offline ASR -> Text @thinhlpg

• Streaming ASR @new5558

Discussions

• ASR: One big all in one model (whisper large v3) or specialized small models for each languages?

[thinhlpg]

250619

• 🎯P0: Pay tech debt 😭: cleanup, refactor demo code -> Easy and seamlessly plug-and-play whenever new checkpoint training is done or swap new ASR, TTS
• improve demo latency. Time from source speech start to first TTS word should be under 4s (current: 6s)
• improve translation quality by including context

[thinhlpg]

250620 ⛵

• Wrapup, release and pivot to next project