Audio and Video Backing Tracks for alphaTab

[Danielku15]

The last few weeks

With alphaTab 1.5 released I started to work on one of the next big things for alphaTab in 1.6 🚀🎸. The possibility to synchronize your music sheet with an a real audio track or video has been quite a highlight feature in various music notation software and platforms. alphaTab was lacking behind there as I focused on other features.

Even though I had other plans for 1.6 first, I wanted to tinker a bit on a proof-of-concept and it turned out to be easier than expected. #1961 covers the key aspects around this topic to synchronize alphaTab with external media, ensuring correct cursor placement.

chrome_AIg2O0gMxP.mp4

Having a synthesizer is nice, but to make it sound really good you need HUGE soundfonts which is not feasable for the web. A separate MP3, OGG or video results in way more impressive sound and experience (at the cost of loading a separate file for every song instead of having the soundfont once in the browser cache).

Most functional parts are done (including Guitar Pro 8 audio track support) and its now just a matter of making the feature easy to use 🥳. You can already check out some first demos here:

https://demo.alphatab.net/audio-track/ (using a Guitar Pro 8 file which I synced with an OGG, alphaTab handles the audio playback)
https://demo.alphatab.net/youtube/ (using the Youtube iFrame Player API, just to demo how to manually integrate the cursor with any external media)

I didn't put much effort into synchronizing the song perfectly but for testing and demo purposes its enough 😅 Locally I tested with some more free demo files included in Guitar Pro 8. On the way I made a variety of improvements to the cursor placement like a smoother placement and less compute intense updates.

So what's next

Now I'm working on a new "playground" area in alphaTab to make this and other features more accessible for fiddling around. Beside most standard options to customize alphaTab, I plan to add a "sync point editor" allowing users to synchronize files with external audio tracks.

Track Selection	Settings

Beside the manual syncing by setting sync point markers, I am also diving a bit into the world of Machine Learning and AI. My wish would be a feature where a mini AI which analyzes the audio and aligns the music sheet and song for you. Overall there will be 3 modes:

1. Manual setting and editing of sync points.
2. Semi-Automatic placement by using the audio start+end and relative tempo changes.
3. AI assisted placement of sync points (if I succeed in my AI/ML journey).

After that its mainly a matter of finalizing the public API to use it and write the documentation.

All this done, I plan to directly release it as 1.6 so people can start using this great addition.

Stay tuned for more updates. 😎🤘

[h16416]

Where to find the Media Sync Editor, I search the repo but not found

[Danielku15]

It is available on the alphatab website and not part of alphatab itself.

https://alphatab.net/docs/playground/ Open it on the bottom

More details here:
https://alphatab.net/docs/guides/media-sync-editor

[h16416]

Thanks. I have read the document. Great job done!
As you mentioned, automatic synchronization of audio and video remains challenging, especially when the media content includes explanatory narration. In such cases, modifying the score by inserting blank sections is not a good solution.
The functionality you currently provide is extra player synchronization in the direction ​from score to media. Could you additionally offer a playback synchronization mechanism in the direction ​from media to score​?

[Danielku15]

You can implement an own external media control: https://alphatab.net/docs/guides/audio-video-sync#custom-external-media-player

Thia way you can control the time tag of the song. For explanatory sections you might also opt for pausing the playback within alphaTab but keep yout external media running. Then provide time tags accordingly.