av1: decode frames smaller than the sequence maximum at their real size

[yevhen-sychov]

av1: decode frames smaller than the sequence maximum at their real size

What you see

With some AV1 streams, hardware decode through this driver starts out fine and
then falls apart: a few seconds after each keyframe, moving parts of the picture
turn into broken blocks, and the damage keeps spreading the more things move
until the next keyframe arrives. Static content can look almost normal, which
makes it easy to mistake for a network or encoder problem.

We hit it in Chromium WebRTC loopback and screen sharing with a hardware AV1
encoder, but it is not Chromium-specific: ffmpeg -hwaccel vaapi shows exactly
the same corruption on the same files, while libdav1d and NVIDIA's own Vulkan
Video decoder (same GPU) decode them perfectly.

Which streams are affected

AV1 lets a stream declare a maximum frame size in its sequence header and then
code individual frames smaller than that (frame_size_override_flag). Hardware
encoders do this all the time. NVENC, for example, works in 16-pixel-aligned
blocks, so for 1080p content it declares a 1920x1088 maximum and codes every
frame at 1920x1080 (and 3840x2176 vs 3840x2160 for 4K).

Streams whose frames are coded at the full maximum size (for example most
software-encoded content) are not affected, which is probably why this has
gone unnoticed.

Why it happens

Players correctly create the VA context and surfaces at the sequence maximum.
The driver then assumed every frame was that size as well:

1. copyAV1PicParam told NVDEC the current frame was the context size
   (1920x1088), and gave every reference frame the size of its surface
   (also 1920x1088), instead of the sizes the frames were really coded at
   (1920x1080). The first frames after a keyframe still come out right, but
   prediction runs against the wrong frame geometry, so small errors appear
   wherever there is motion and compound frame after frame.

2. Once the right sizes are passed, a second issue shows up: NVDEC does not
   crop a smaller frame out of the decoder output, it stretches it to fill
   the display area the decoder was created with. A 1080-line frame would come
   out scaled to 1088 lines.

The first point was found by recording the exact CUVIDAV1PICPARAMS this
driver and ffmpeg's own NVDEC AV1 hwaccel send for the same stream: the only
fields that differed were the frame and reference sizes.

What the patch changes

• Real sizes in the picture parameters. The current frame gets its coded
  size; that size is remembered on the surface the frame is decoded into, and
  each reference frame is described with its remembered size instead of its
  surface size. The picture parameters now match ffmpeg's NVDEC hwaccel.

• Display area follows the frame size. When a frame's size differs from the
  decoder's display area, the display area is changed with
  cuvidReconfigureDecoder. The frame is placed unscaled in the top-left of the
  unchanged, surface-sized target, so the existing copy into the surface keeps
  working as before.

  Two details worth knowing when reviewing:

  • NVDEC refuses a reconfigure before the decoder has decoded anything, so the
    first change is applied right after the first picture is decoded and before
    that picture is handed to the resolve thread (the display area takes effect
    when a frame is mapped, not when it is decoded).
  • If the size changes again later in a stream, frames already queued for
    output are allowed to finish before the display area changes, so they are
    not cropped with the new size.

Streams whose frames are all coded at the maximum size never request a display
area change and take exactly the same path as before.

How it was tested

RTX GPU, NVIDIA driver 610.57.04, direct backend. Each stream was decoded
through this driver with ffmpeg -hwaccel vaapi and compared frame by frame
against libdav1d:

Stream	Before	After
1920x1080 frames, 1920x1088 sequence maximum	59 / 150 frames bit-exact	150 / 150
3840x2160 frames, 3840x2176 sequence maximum	30 / 90	90 / 90
AV1 frames equal to the sequence maximum (av1_nvenc)	150 / 150	150 / 150
H.264 decode	—	byte-identical to before

No reconfigure failures or CUDA errors were logged. Also confirmed by eye in a
Chromium WebRTC AV1 loopback with heavy motion: before, the picture degraded
into blocks within seconds; after, it stays clean.

Not covered: a stream whose frame size changes in the middle of playback (no
such sample was available). The code path for it is described above.

Reproducing it yourself

You need a stream whose frames are smaller than the sequence maximum. The
simplest way to check an existing file:

ffmpeg -i input.ivf -c:v copy -bsf:v trace_headers -f null - 2>&1 \
  | grep -E "max_frame_height_minus_1|frame_size_override_flag|frame_height_minus_1" | head

frame_size_override_flag = 1 with frame_height_minus_1 smaller than
max_frame_height_minus_1 means the stream is affected.

Then compare the hardware decode with libdav1d (1080p example):

ffmpeg -c:v libdav1d -i input.ivf -f rawvideo -pix_fmt yuv420p dav1d.yuv
LIBVA_DRIVER_NAME=nvidia ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi -i input.ivf \
  -vf 'hwdownload,format=nv12' -f rawvideo -pix_fmt yuv420p vaapi.yuv
ffmpeg -f rawvideo -pix_fmt yuv420p -s 1920x1080 -i vaapi.yuv \
       -f rawvideo -pix_fmt yuv420p -s 1920x1080 -i dav1d.yuv \
       -lavfi psnr=stats_file=psnr.txt -f null -
grep -c 'psnr_y:inf' psnr.txt   # frames that match exactly

Without the patch the count stops matching roughly 30 frames after every
keyframe; with it, every frame matches.

🤖 Generated with Claude Code

[donbernhardo]

Thanks for this PR. After a review I propose two small changes you might consider merging:

I tested a change on top of this PR that fixes a use-after-free risk in nvDestroySurfaces. A decoded surface can outlive its VA context. After vaDestroyContext frees the context, the new lastQueuedSurface check would dereference the surface's stale context pointer. This commit checks that the context is still in the driver's object table before reading it.

Commit: donbernhardo/nvidia-vaapi-driver@b65b70d

Feel free to cherry-pick it:

git cherry-pick b65b70d

Another patch fixes an inconsistent CUVID picture parameter block. The AV1-specific width and height now describe the current frame, but PicWidthInMbs and FrameHeightInMbs still describe the VA context maximum. The CUVID header calls these coded-frame dimensions, and FFmpeg's NVDEC AV1 implementation derives them from its current frame. This commit makes all four fields refer to the same frame.

Commit: donbernhardo/nvidia-vaapi-driver@4e8bf7a

Feel free to cherry-pick it independently of the first commit:

git cherry-pick 4e8bf7a

[yevhen-sychov]

Thanks for reviewing this so carefully — both points were worth making, and the branch is now rebased onto current master with them addressed.

4e8bf7a (macroblock dimensions): cherry-picked as-is, with your authorship intact. You are right that the block was self-contradictory: the AV1 fields described the coded frame while PicWidthInMbs/FrameHeightInMbs still described the context maximum. I checked FFmpeg's nvdec_av1.c and it derives both from the same cur_frame, so this matches the reference implementation. As on your 3070, it changes no output here — the affected streams already decoded bit-exactly — so I have kept your commit message's honest framing that this aligns the API parameters rather than fixing visible corruption.

b65b70d (use-after-free): the finding was correct, but the fix could not be taken literally. Rebasing onto master pulled in your surface-lifetime work, which replaced surface->context with surface->contextId, so the pointer your patch guarded no longer exists. Instead of looking the context up by pointer, I clear lastQueuedSurface inside the context loop that nvDestroySurfaces already runs under objectCreationMutex to retire VideoProc render targets — so the stale pointer is gone by construction, without an extra object-table lookup. The waitSurfaceResolved() call from my original hunk is also gone, since detachBackingImageFromSurface() now waits for resolution itself. Credit for spotting the lifetime hole is yours; please do check whether the replacement convinces you.

Re-verified after the rebase, decoding through the driver with ffmpeg -hwaccel vaapi and comparing every frame against libdav1d:

Stream	master (2ed4160)	this branch
1920x1080 frames in a 1920x1088 sequence	29 / 150 frames bit-exact	150 / 150
3840x2160 frames in a 3840x2176 sequence	29 / 90	90 / 90
frames equal to the sequence maximum (av1_nvenc)	150 / 150	150 / 150

The driver log shows a single Decoder display area set to 1920x1080 (surfaces 1920x1088) per stream, with no reconfigure failures or CUDA errors.

[yevhen-sychov]

Added a regression test, since the repro for this only existed in the PR description.

tests/test_av1_frame_size_override.sh decodes a stream whose frames are coded smaller than the sequence maximum through the driver and through libdav1d, and requires them to agree — a correct AV1 hardware decode is bit-exact with a software one. It also decodes a control stream whose frames match the maximum.

Against the unpatched decoder the two cases separate cleanly, which is what makes the test worth having:

  AV1 320x180 frames in 320x192 sequence     FAIL (PSNR=13.10 dB < 40.0 dB)
  AV1 frames equal to the sequence maximum   PASS (bit-exact)

With the fix both are bit-exact.

A few notes on how it is built, in case you would rather it looked different:

• The fixture is generated, not checked in, to match samples/ being .gitignore'd. SVT-AV1's forced-max-frame-width/height is the only widely available encoder knob I found that emits this layout — libaom has no equivalent option exposed through FFmpeg, and av1_nvenc always sets the maximum equal to the frame size, so it cannot reproduce it.
• The test proves its own fixture. It parses the generated stream with trace_headers and only runs the comparison if frame_size_override_flag is really set with a maximum larger than the coded frames; otherwise it skips. Without that it could quietly pass while covering nothing, should a future SVT-AV1 ignore the parameter.
• It skips rather than fails when ffmpeg has no libsvtav1 encoder or no libdav1d decoder, following test_vp8_decode.sh.
• This adds the first test() to meson.build, guarded by find_program('ffmpeg', required: false). If you would prefer the PR to stay purely a driver fix, I am happy to drop the meson wiring and leave the script standalone, or to split the test into its own PR.

[elFarto]

Thanks for the patch!