vp8: reconstruct frame headers from VA-API parameters

[donbernhardo]

vp8: reconstruct frame headers from VA-API parameters

VP8 decoding can return a frozen image while reporting success. The driver attempts to recover the frame header by reading before the client's slice-data pointer and also extends slice reads beyond the copied data.

This change reconstructs the VP8 frame tag and key-frame header from the VA-API picture and slice parameters, then appends the submitted partition data. It removes the alignment-based read before the client buffer and checks slice ranges before copying.

Reproduce and verify

Requires:

• FFmpeg with libvpx
• An NVIDIA GPU with VP8 decode support
• /dev/dri/renderD128

Build the driver and run the following commands from the repository root. Repeat the test in the unpatched and patched checkouts, each using its own build directory.

meson setup build-release -Dbuildtype=release
ninja -C build-release

# Generate three seconds of moving VP8 video.
ffmpeg -y -f lavfi -i 'testsrc2=size=640x360:rate=30:duration=3' \
  -c:v libvpx -deadline realtime -pix_fmt yuv420p vp8-test.webm

# Decode a software reference.
ffmpeg -y -i vp8-test.webm -pix_fmt nv12 -f rawvideo software.nv12

# Decode with the driver built in this checkout.
LIBVA_DRIVER_NAME=nvidia LIBVA_DRIVERS_PATH="$PWD/build-release" \
NVD_BACKEND=direct NVD_LOG_VERBOSE=0 \
timeout -k 1s 15s ffmpeg -y \
  -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi -i vp8-test.webm \
  -vf 'hwdownload,format=nv12' -f rawvideo hardware.nv12

# Compare decoded pixels.
cmp software.nv12 hardware.nv12

Verified on an RTX 3070:

• Unpatched master (f77ef1c): hardware output differs from the software reference.
• With this fix: cmp produces no output and exits with status 0.
• Both runs decode all 90 frames, producing 31,104,000 bytes each. Frame count alone does not detect the bug.

Additional validation

The Meson/Ninja build succeeds.

A separate 90-frame VP8 clip and a 90-frame H.264 control also match software decode byte-for-byte.

[elFarto]

Thanks for the patch, I was never happy with that hack but VP8 just isn't used that much now.

[donbernhardo]

You"re welcome! I think as long as we have NVDEC hardware fixed-function pipelines in our GPUs for these codecs, we might as well have a driver that makes them accessible ;)