Asking about the best practice to fine-tuning?

[Crestina2001]

Goal

mimic a specific timbre

dataset preparation

Question 1: length of all the audio segments combined shall be at least?
Question 2: recommended length range of each segment?
Question 3: Is it beneficial to trim the leading and ending silence? What about leading and ending breath(I have methods to trim it, so I just wonder whether we shall trim it)?
Question 4: Do we need to normalize volume?

hyper-params and success metrics

Question 5: How to tune batch size and maximum steps? Is there an empirical formula for the size of the dataset vs maximum steps?
Question 6: I have noticed a phenomenon during fine-tuning: the loss on validation datasets has risen after a few epochs, but the timbre similarity improves. So do we need to watch the loss on val datasets at all? Is the training loss the only thing we need to watch?
Question 7: what loss/diff is considered to be 'good', which seems to drop unsteadily.

other confusions

Question 8: Without reference audios and only given the lora-fted model, the timbre may vary a lot: the varying can be like emotional changes, volume changes and pitch changes. It is unrealistic that the training clips are all very similar in emotions, tones etc. And we do want some emotional changes when it reads long texts. But the changes are too abrupt and often unintended. What is the best way to increase the steadiness of the trained model?

Many thanks to any helpful suggestions!

[a710128]

Q1: The more data, the better. However, I've successfully mimicked a timbre with just 15 minutes of audio. If your dataset is small, it's highly recommended to also use prompt_audio and prompt_text during inference.

Q2: I've previously used segments around 30 seconds long, but I believe longer ones should work as well.

Q3: Yes, trimming leading and ending silence is beneficial. It's best to keep any remaining silent segments at the start or end under 1 second.

Q4: No, that's not necessary.

Q5: This depends on the size of your dataset. It usually takes some experimentation to find the optimal learning rate and number of steps.

Q6: It's best to monitor the validation loss. It's a good indicator of when to stop training and helps prevent overfitting, which might be what you're seeing.

Q7: The loss/diff decreases very slowly. I'd recommend judging the model's quality by periodically generating test audio rather than relying solely on the loss value.

Q8: To increase the stability of a LoRA-finetuned model, the most effective method is to provide a reference audio during inference. This has been my consistent experience after a lot of testing.