Some suggestions on training and inference

[ylacombe]

Hey there, congrats on releasing this great effort!

There are two immediate suggestions that could help you:

• For inference: The current W2V-bert implementation doesn't use Flash Attention 2 and SDPA. Integrating it to transformers should be relatively easy, building upon the Wav2vec implementation
• For training, if you get rid of this part, you can get easy training capability improvements by adding an adapter (add_adapter=True)

Hope that helps!
Congrats on the release again

[kwindla]

That's super helpful. Will look at both things. Thank you.

[marcus-daily]

In Smart Turn v2 we've moved to wav2vec2 instead of wav2vec2-BERT:

https://www.daily.co/blog/smart-turn-v2-faster-inference-and-13-new-languages-for-voice-ai/

Using Flash Attention for the model is something we still need to investigate/benchmark. As part of the next cycle of development we'll be looking at a variety of performance improvements, including quantization.