Memory bugs in video/audio pipeline: AVFrame leak, double-free, integer truncation

[grisutheguru]

Summary

Multiple memory safety issues in the video and audio pipeline.

Details

1. AVFrame and AVPacket leak in legacy FFmpeg path (src/video/camerasource.cpp:511-527)

In the #if LIBAVCODEC_VERSION_INT < AV_VERSION_INT(57, 48, 101) path:

AVFrame* frame = av_frame_alloc();
if (!frame) {
    return;  // packet not unref'd
}
if (packet.stream_index == videoStreamIndex) {
    int frameFinished;
    avcodec_decode_video2(cctx, frame, &frameFinished, &packet);
    if (!frameFinished) {
        return;  // frame leaked, packet not unref'd
    }
    // ...
}
// falls through without freeing frame when stream_index != videoStreamIndex

av_frame_free(&frame) is missing on all early return paths. av_packet_unref(&packet) at line 544 is also bypassed by these returns.

Fix: Use RAII wrappers or add cleanup before all returns:

struct AVFrameDeleter { void operator()(AVFrame* f) { av_frame_free(&f); } };
std::unique_ptr<AVFrame, AVFrameDeleter> frame(av_frame_alloc());

2. inputBuffer dangling pointer after delete (audio/src/backend/openal.cpp:576-590)

void OpenAL::cleanupInput()
{
    // ...
    delete[] inputBuffer;
    // inputBuffer NOT set to nullptr -> double-free risk
}

If cleanupInput() is called twice (e.g., explicit call + destructor), or if doInput() fires between cleanup and object destruction, the dangling pointer is used.

Fix: Add inputBuffer = nullptr; after delete[].

3. Integer truncation in ToxYUVFrame dimensions (src/video/videoframe.cpp:326-327)

return ToxYUVFrame{
    static_cast<std::uint16_t>(frameSize.width()),   // int -> uint16_t, wraps at 65536
    static_cast<std::uint16_t>(frameSize.height()),

No bounds check before the cast. A malformed video frame with dimensions > 65535 causes silent wraparound, leading to incorrect buffer size calculations.

Fix: Add bounds check:

if (frameSize.width() > std::numeric_limits<uint16_t>::max() ||
    frameSize.height() > std::numeric_limits<uint16_t>::max()) {
    return {ToxYUVFrame{}, ReadWriteLocker()};
}

4. Missing null check for avcodec_alloc_context3 (src/video/camerasource.cpp:436)

cctx = avcodec_alloc_context3(codec);
if (avcodec_parameters_to_context(cctx, cparams) < 0) {  // cctx could be null

Fix: Check cctx != nullptr before use.

Impact

Memory leaks accumulate over time causing OOM. Double-free and integer truncation can cause crashes. All are reachable during normal video/audio usage.