[RFC] Add subrandr SRV3 subtitle renderer

[afishhh]

This PR adds support for subrandr, a subtitle rendering library I've been working on for, uh, the past 7 months.

The whole point is to render non-ASS subtitle formats correctly, without conversion, because conversion is most of the time lossy. Currently the supported formats are SRV3 which is YouTube's subtitle format and WebVTT.

Results

I have collected a few videos that use more complex SRV3 subtitles while working on subrandr, so I spent some time making three funny dwm four way comparisons between:

• Top left: mpv with subrandr
• Top right: conversion to ass via my ffmpeg decoder. Alternatively one may use YTSubConverter which I believe supports ruby text via manual layout with font metrics at conversion time (I don't know whether this is actually the case?), this approach is obviously fragile with font fallback in the mix so personally I don't consider it a real solution.
• Bottom left: status quo when playing a video from a URL now, which is subtitles in WebVTT format converted on YouTube's side, converted to ASS by ffmpeg and then played using libass
• Bottom right: YouTube web player, ground truth

Comparisons of example videos

【original anime MV】幽霊船戦【hololive/宝鐘マリン】
Hololive music videos often have ruby text and as such are decent testing material. subrandr should handle SRV3 ruby text correctly although it's not implemented for WebVTT yet.

Worst Teambuilding Exercise Ever
This video contains a lot of positioned subtitles with different types of text shadow, and at this particular moment also exercises line-wrapping, which for SRV3 should be greedy.

sodapoppin checks out Northernlion's stream
This one is not that special and the ASS conversion is decently close, but the positioning is of course incorrect because it is fully in the video frame and the font size is similarly just slightly off.

For the sake of completeness, the process I used to create the comparisons

Each quadrant is either an mpv or firefox window, ran under X11 with the dwm window manager in master layout with nmasters = 2, with the windows being constructed as follows:
Top left: Use mpv compiled with subrandr and play a downloaded copy of the video with an accompanying srv3 file. (--sub-format srv3 in ytdl)
Top right: Convert srv3 file to ass file via ffmpeg (ffmpeg -i <in>.srv3 <out>.ass) then play the video with mpv and switch to the ASS track. (requires this fork of ffmpeg)
Bottom left: Play a downloaded copy of the video with an accompanying vtt file downloaded from YouTube.
Bottom right:

1. Disable browser decorations via userChrome.css.
2. Open the video on YouTube.
3. Seek to the appropriate time, put the player in theater mode.
4. Run below snippet in the console

(() => {  
function full(el, h) {
  el.style.position = "fixed"
  el.style.zIndex = 10000000;
  if(h == "auto") {
    el.style.top = "50%"
    el.style.left = "50%"
    el.style.transform = "translate(-50%, -50%)";
  } else {
    el.style.top = "0"
    el.style.left = "0"
  }
  el.style.backgroundColor = "black"
}
let player = document.querySelector("#movie_player");
full(player, "100vh");
full(player.querySelector("video"), "auto");
player.querySelector(".ytp-chrome-bottom").style.display = "none";
})()

5. Make sure that if the subtitles move if the video is paused (don't remember what this depends on) that the video is playing when the screenshot is taken

I could've probably made separate screenshots in fullscreen mode and then stitched them together inside a markdown table... whatever, this is the first thing I thought of and it probably looks better.

Limitations

Since the library is still in "early" stages there's a lot of things that are not done correctly yet, this is a non-exhaustive list of the most important such things:

• No DirectWrite font provider. (no fonts will be found on Windows)
• No CoreText font provider. (no fonts will be found on MacOS)
• Line-breaking is very naive and does unnecessary reshaping, lines are only broken on whitespace instead of following the Unicode line breaking algorithm. Fixed.
• Unicode bidirectional algorithm is not used. Implemented.
• Any form of vertical text is unsupported.
• Font selection is not compliant with the CSS font matching algorithm, this has been mostly implemented in a branch but is not yet finished. I have since learned that chromium does something pretty close to what I do, so it's staying.
• Subpixel glyph rendering is not implemented so positions are rounded to integer values, this looks wrong on non-HiDPI displays in lower window sizes. I plan to fix this soon.
• The rasterizer is not optimized to libass levels, everything is portable (although unsafe) Rust code. The design allows for some GPU acceleration implemented in an optional wgpu rasterizer although mostly only blitting and blurring with actual path rendering being too complicated, and because mpv is designed with software-only OSD rendering in mind it can't be integrated easily. (correct me if I'm wrong)
• There are like two places where it can panic because of unimplemented things but that's easily fixable. Fixed.

The lack of a font provider means that the library will immediately return an error from sbr_renderer_render as soon as it tries to render text, so it's not currently usable on MacOS without fontconfig.

mpv integration unresolved issues

• I changed ytdl-hook to request srv3 subtitles, however this is not gated behind conditional compilation or any sort of runtime check so it breaks in builds without subrandr. Maybe a runtime property could be added that Lua can read? Solved with subrandr-version property.
• subrandr is always passed a dpi of 72, this shouldn't have impact on subtitle layout with the currently supported formats, but it does impact debug UI when enabled via SBR_DEBUG=draw_version,draw_perf,draw_layout and may break in the future if support for CSS in WebVTT is added since one could do ::cue { text-size: 20px; } which must be scaled by the device pixel ratio.
  I have no idea how to get dpi information in get_bitmaps without digging into mpv_global which contains a warning specifically telling you not to do that.
• Oh and I almost forgot, currently I forcefully un-align the stride of the resulting mp_image which could probably cause issues down the line on some platforms, so that should probably be changed. Fixed.

Building

So if you got this far and are on Linux with FreeType, HarfBuzz (with FreeType support), and Fontconfig libraries installed, here's how you build and install the library:

git clone https://github.com/afishhh/subrandr
cd subrandr
cargo xtask install --prefix /<path>/<to>/<prefix>

You also need Rust installed, the latest stable toolchain should work.
The prefix should be set to a writable path where the library should be installed, it will create the following filesystem structure there:

include/
  subrandr/
    <headers>
lib/
  pkgconfig/
    subrandr.pc
  libsubrandr.a
  libsubrandr.so

Then you need to make sure <prefix>/lib/pkgconfig is on your pkg-config path when running meson setup (for example via the --pkg-config-path meson arg).

After building the library itself, you should be able to build mpv as usual, by passing -D subrandr=enabled to meson setup you can ensure the library is correctly detected or you will get a build error. The library itself is linked statically, with mpv inheriting the dynamic library dependencies. Static linking is never a good idea somehow.

Alternatives

Is this all necessary? Other possible approaches could be:

• Adding extensions to libass for things like ruby text, doesn't account for format idiosyncrasies like positioning which is especially important with WebVTT where the positioning algorithm is very different (step 10 onward).
• Conversion in ffmpeg, see my patch, it does a decent job but has no chance of supporting stuff like ruby text because loading fonts in a decoder would be very strange and probably never get merged into mainline ffmpeg. Did I mention it's hacky and fragile yet?
• Do nothing, the sad truth is that these formats, even though they're quite powerful, are seldom used to even a fraction of their full potential. YouTube themselves don't support some of SRV3's features on non-Web players (crazy idea would be to add subrandr to revanced, food for thought).

Naturally, after spending months working on this, I am slightly biased and believe a separate renderer is worth because it allows iterating on other subtitle formats without having to worry about the unhinged format known as Advanced Substation Alpha. At first I was developing ASS support in parallel to SRV3 in subrandr, but then realized how much horrible complexity ASS adds and purged it from the code base, this in my eyes confirmed that it is significantly simpler to have other formats handled separately.

Thanks for reading, hope you like my work :)

[llyyr]

Excellent job! This works almost perfectly on the few videos I tried. Besides the few limitations you've listed, one other thing I noticed was that alignment is a bit off in the video I tested with at 00:59. Scratch that, subrandr matches Chromium output, it's actually Firefox that's dorky here. Probably sensible to treat Chromium as the ground truth here

You should set PREFIX to a path where $PREFIX/lib/pkgconfig will be on PKG_CONFIG_PATH, $PREFIX/lib will be on the linker library path, and $PREFIX/include will be on the include path. (I believe /usr/local/ works on "usual" Linux distributions but can't test)

This is actually not necessary, you can set PREFIX to any directory then point meson to the pkgconfig file. For example I installed it to PREFIX=/opt/subrandr and configured meson with meson setup build --pkg-config-path /opt/subrandr/lib/pkgconfig. This avoids littering your /usr/local files or needing to change environment variables.

[github-actions]

Download the artifacts for this pull request:

Windows

• mpv-aarch64-pc-windows-msvc
• mpv-i686-w64-mingw32
• mpv-x86_64-pc-windows-msvc
• mpv-x86_64-w64-mingw32

macOS

• mpv-macos-13-intel
• mpv-macos-14-arm
• mpv-macos-15-arm

[afishhh]

Alternatives -> Nothing

Didn't FFMPEG finally add WebVTT support just 2 days ago?

By "Nothing" I meant that we could "do nothing" not that there's no alternatives, there wouldn't be an alternatives section with alternatives if there weren't alternatives. Looking at ffmpeg-devel I don't see any patches for improving styling support, ffmpeg supported simple webvtt throwing out all most of the styles for a long time already.

Also I fixed the stride hack so now the resulting mp_image has proper aligned stride.

[hooke007]

The lack of a font provider means that the library will immediately return an error from sbr_renderer_render as soon as it tries to render text, so it's not currently usable on platforms other than "unix with fontconfig".

Hi I wonder why windows is blocked by this reason. Fontconfig is available on windows too.

[afishhh]

The lack of a font provider means that the library will immediately return an error from sbr_renderer_render as soon as it tries to render text, so it's not currently usable on platforms other than "unix with fontconfig".

Hi I wonder why windows is blocked by this reason. Fontconfig is available on windows too.

Well if that's the case that would make things easier. After downloading many dlls from msys packages I even got it to compile, run, and find the config file in wine but it doesn't find any fonts. I'm hoping that this actually does work on Windows but someone on Windows would have to actually check that. In particular I have no idea what encoding fontconfig returns in FC_FILE on Windows and am currently wishfully assuming it's UTF-8.

[afishhh]

Implemented WebVTT snap-to-lines = false layout and Unicode line breaking. This means I'm now slightly less afraid of breaking people's WebVTT rendering.

Is there anything to do on the MPV side before this is ready for review? I was thinking that it may be confusing if people using very customized subtitle options have their customization ignored in WebVTT (because sd_sbr doesn't implement them). Maybe initially we could just use subrandr for SRV3 to not cause regressions with WebVTT if subrandr is enabled, though there is still ytdl_hook that will now start ignoring your stuff by preferring SRV3 but that's less difficult to have configurable at runtime I guess.

[kasper93]

Not sure it makes much sense, but mpv generally allow to configure such parameters.

[afishhh]

I guess it could be done but there's not really a point since the probing should identify the subtitle from the first line alone.

[kasper93]

up to you. we tend to avoid hardcoded params where possible though

[afishhh]

Okay I've made this configurable now. Wondering whether the demuxer should also accept an option to disable forcing itself on you and let you fallback to lavf+libass, not sure how to call it though.

[kasper93]

Wondering whether the demuxer should also accept an option to disable forcing itself

For main demuxer we have --demuxer, so if it's something you want to support, you could add --sub-demuxer.

[afishhh]

--sub-demuxer already exists so I guess that does mostly address that although you won't be able to use other demuxers if you set it to lavf-only. Now that I think about it the best way to address my concern would probably be a more general --{,sub-,audio-}demuxer-blacklist option. I'll leave that as future work since --sub-demuxer does indeed address most of the use case.

One thing I noticed is that if --sub-demuxer=lavf is set then playing a video with ytdl_hook will unconditionally fail to load subs, even if --ytdl-raw-options-add=sub-format=vtt is set. Note that this is the case even on mpv v0.40.0 (without subrandr).

[afishhh]

Bumping this, I've since improved subrandr's line-height handling and ruby positioning, also am in the process of implementing a mini web engine to match browser styling more precisely (along with improving the layout subsystem on master in the process).
I think this PR is pretty much ready, there's some things that could be done in the future but I don't think there's anything blocking (my main worry is still incompatibility with user ASS styles but that's mostly fixed by --sub-demuxer=lavf existing).

[kasper93]

Sorry for the delay, I might have some time this weekend to look at this. On thing that stands out is, how we streamline building and shipping new library. I was thinking of defining dummy meson.build which would consist of custom target that runs cargo build and declare dependency with built library location, so we can use it in mpv meson as subproject. My point is that we should enable it in our CI builds and this probably would be easy to connect to existing pipeline.

[afishhh]

On thing that stands out is, how we streamline building and shipping new library. I was thinking of defining dummy meson.build which would consist of custom target that runs cargo build and declare dependency with built library location, so we can use it in mpv meson as subproject. My point is that we should enable it in our CI builds and this probably would be easy to connect to existing pipeline.

I looked into doing the meson.build stuff, the issue with using declare_dependency() is that it duplicates all the information in the preexisting pkg-config file and the logic in generating it (also such a meson file must re-implement a crude meson target_machine -> rustc target triple translation for cross compilation).

It just seems like a decent amount of work and it would have to be updated whenever something linking-related is changed. I can try but I just don't know whether I want to maintain all that honestly.
It would be better if meson could just read that pkg-config file post-build but that's practically the same thing as just building the library as part of the CI job so I wonder whether that would be simpler overall due to not duplicating internal logic.

[kasper93]

Except few minor nitpicks, LGTM. I have not tested it as I wouldn't know what to test with VTT, but as far as integration in mpv is concerned it is fine.

As discussed on IRC, remaining thing is to make this somehow accessible. As likely mpv will be a main user of this at least at the beginning, building it on CI would be good to make it available in our test builds.

[afishhh]

I added subrandr to the CI builds for Linux, x86_64 mingw64, and x86_64 msys2. Currently i686 and aarch64 Windows targets don't work which I track on my side in afishhh/subrandr#31.

[afishhh]

Fixed ARM64 implibs in implib-rs upstream, so aarch64 msys2 can now be tested.

[afishhh]

Right makes sense the weird mutual exclusivity of these options could cause issues, although they do seem pretty niche in this case. I was meaning to add pkg-config support at some point but it is currently pretty low on my priority list.
If you care about this enough you can create an issue or a PR with your changes and we can discuss this further there.

[BergmannAtmet]

If you care about this enough

Not really. This is not even my main account ;)

[afishhh]

Along the lines of #16271 (comment), I'm thinking of maybe removing WebVTT from this PR. That would mean it no longer fixes #16456 but we could come back to that later with an opt-in solution. Sounds like the safest approach, thoughts?

[sfan5]

I think it's fine as is. We can figure out what to do about that later.
Possibly:

• Runtime option to disable subrandr internally
• Warn people that subrandr is experimental and you shouldn't compile normal builds with it
• 🤷

[na-na-hi]

I think it's fine as is. We can figure out what to do about that later.

I consider unable to style subtitles a regression.

• Runtime option to disable subrandr internally

A generic option to costomize demuxer or sd probe list like for vo/gpu-api/gpu-context will solve this while also being more flexible.

• Warn people that subrandr is experimental and you shouldn't compile normal builds with it

If you really agree with this, then it should be disabled by default in meson.options.

[sfan5]

I consider unable to style subtitles a regression.

Yes and we can figure that out later, before the next release.

A generic option to costomize demuxer or sd probe list like for vo/gpu-api/gpu-context will solve this while also being more flexible.

Well that won't get rid of subrandr-version, which ytdl_hook uses for detection. So I was thinking specifically of a "pretend subrandr doesn't exist" option.

If you really agree with this, then it should be disabled by default in meson.options.

Sure why not.

[afishhh]

I think it would be better to remove WebVTT for now since I truly don't care enough about it and na-na-hi is right.
subrandr provides a good amount of value for SRV3 and making enabling it be a tradeoff instead of an upgrade would probably mean most builds would be compiled without it (even if it's just "temporarily"). Instead we can think of a way to make this not a regression and then re-enable it, it's a tiny change in the demuxer anyway.

[Akemi]

just on a side note for people who want to build it on macOS too. i had to manually set the lib paths for freetype, fontconfig and harfbuzz. there might be a better way to path those -L flags. retrived those paths with pkg-config --libs fontconfig|harfbuzz|freetyp.

RUSTFLAGS="-L/opt/homebrew/opt/freetype/lib -L/opt/homebrew/Cellar/harfbuzz/11.4.3/lib -L/opt/homebrew/Cellar/fontconfig/2.16.0/lib" cargo xtask install --prefix=$(brew --prefix)

Does this work for you?

RUSTFLAGS shouldn't be relied on as its value is not guaranteed to be appended to rustc calls.

sry it took so long to test. it can't find freetype, but harfbuzz and fontconfig are okay with that commit. i think i should have been more specific with my first comment, eg pkg-config --libs freetype doesn't find anything but pkg-config --libs freetype2 does.

error: linking with `cc` failed: exit status: 1
  |
  = note:  "cc" "-Wl,-exported_symbols_list" "-Wl,/var/folders/p1/k5kgl3bj6zqg23mwx_5b0lrh0000gn/T/rustcEmdcbl/list" "/var/folders/p1/k5kgl3bj6zqg23mwx_5b0lrh0000gn/T/rustcEmdcbl/symbols.o" "<17 object files omitted>" "-lfontconfig" "/Users/Akemi/Projects/subrandr/target/aarch64-apple-darwin/release/deps/{libunicode_bidi-6be253d180c7c1c1.rlib,libonce_cell-c788aa764e892aa3.rlib,libtext_sys-b5f574629bd1ef7c.rlib,libicu_segmenter-32fed029e07cf6be.rlib,libicu_locale-ffdf4954356e079e.rlib,libicu_locale_data-61154579b9d51fd2.rlib,libicu_segmenter_data-6e220c6d6d6e74dd.rlib,libutf8_iter-05b64ee6d4bf8296.rlib,libcore_maths-7d8a5f49c8da0474.rlib,liblibm-234ee7f1c7d31be5.rlib,libicu_collections-11219d8930850f39.rlib,libpotential_utf-28efcd7bd2b4ab4e.rlib,libicu_provider-f69e24ac9addbb4a.rlib,libicu_locale_core-9416f850a04f357c.rlib,libtinystr-da1e120ffcd995f0.rlib,liblitemap-ff35df871368c3ec.rlib,libwriteable-0f49d70b651d77e3.rlib,libzerovec-51254eb51d75a969.rlib,libyoke-cddd712bbf10ba05.rlib,libstable_deref_trait-20d54840d2a0f5a7.rlib,libzerofrom-17d326580a8882cc.rlib,libzerotrie-a894adb247fc5df6.rlib,libthiserror-17559b00179b4cb0.rlib,libquick_xml-fe4dd31c33c33753.rlib,libmemchr-17ba13cbea6c520d.rlib,libsbr_rasterize-6582e96dc6cda95b.rlib,libsbr_util-0eeb8ca60a80b41b.rlib}.rlib" "<sysroot>/lib/rustlib/aarch64-apple-darwin/lib/{libstd-*,libpanic_abort-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-lfreetype" "-lharfbuzz" "-lSystem" "-lc" "-lm" "-arch" "arm64" "-mmacosx-version-min=11.0.0" "-L" "/opt/homebrew/Cellar/fontconfig/2.17.1/lib" "-L" "/opt/homebrew/Cellar/fontconfig/2.17.1/lib" "-L" "/opt/homebrew/Cellar/harfbuzz/11.4.5/lib" "-L" "/opt/homebrew/Cellar/harfbuzz/11.4.5/lib" "-o" "/Users/Akemi/Projects/subrandr/target/aarch64-apple-darwin/release/deps/libsubrandr.dylib" "-Wl,-dead_strip" "-dynamiclib" "-nodefaultlibs" "-Wl,-compatibility_version,0" "-Wl,-current_version,0.2.0"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: ld: library 'freetype' not found
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: could not compile `subrandr` (lib) due to 1 previous error
Error: `cargo build` failed: exit status: 101

[afishhh]

I removed the sub-format changes from this PR since I don't think it's worth blocking this on getting consensus for that.
Will open a PR with those changes where they can be discussed separately if that's fine (Personally my use-case doesn't involve playing srv3 via the ytdl_hook but I'd still be nice to make that option less obscure via documentation or an option).

[afishhh]

Realized that DirectWrite can, in fact, return errors from some font family functions while working on something else. This means an error from sbr_renderer_create is not actually that unlikely so I made sd_sbr gracefully log errors from all subrandr functions instead of mp_require'ing some.

Also just realized there are merge conflicts, I'll wait with rebasing for now since this is blocked on a release anyway.