VideoCapture from io.Reader

[agorman]

Would it be possible to open video capture from an io.Reader? Currently it looks like it can either open a url or device. It would be nice if there was a way to just pass it a io.Reader directly.

[swdee]

The VideoCapture functions are just wrappers to the underlying C++ code of OpenCV which in turn uses backends like FFMpeg and GStreamer that handle the video source. So there isn't any way of using io.Reader.

What use case are you trying to solve? Often non standard ones are solved with a pipeline using the Gstreamer backend.

[agorman]

Thanks for the reply @swdee.

I have a video processing pipeline where the first step uses ffmpeg to encode the video into a specific format. I then take the output file and run it through gocv to process the frames for various tasks. It would save time if I could combine that into one step where the output of my ffmpeg command could pipe directly into gocv's video capture.

I'm currently just punting to ffmpeg using exec.Command and I was thinking it would be really nice to just pipe the output directly to the gocv.VideoCapture using an io.Reader. I understand the underlying library doesn't support it so if you have any thoughts on good alternatives I'd appreciate it!

[swdee]

I assume your using ffmpeg as a pipe via exec.Command. How about using the io.Reader to write directly into a Mat, bypassing the need of a VideoCapture? Something like this;

package main

import (
	"bufio"
	"fmt"
	"io"
	"os"
	"os/exec"

	"gocv.io/x/gocv"
)

func main() {
	// Example source:
	//   - file: "input.mp4"
	//   - RTSP: "rtsp://user:pass@host:554/stream"
	//   - device on Linux: "/dev/video0" (use -f v4l2 in ffmpeg args)
	if len(os.Args) < 2 {
		fmt.Println("Usage: ffpipe <input-url-or-file>")
		os.Exit(1)
	}
	src := os.Args[1]

	// You MUST know (or decide) the output dimensions because rawvideo has no headers.
	// Pick something that matches your model / UI needs.
	width := 640
	height := 480
	frameBytes := width * height * 3 // bgr24

	// ffmpeg command:
	// - decodes the input
	// - scales to width/height
	// - outputs raw bgr24 frames to stdout
	//
	// Notes:
	// - "-an" disables audio
	// - "-fflags nobuffer" and "-flags low_delay" are sometimes helpful for low-latency streams
	// - "-vf scale=..." forces a consistent frame size
	args := []string{
		"-hide_banner",
		"-loglevel", "error",
		// Uncomment for lower latency on some live sources:
		// "-fflags", "nobuffer",
		// "-flags", "low_delay",
		"-i", src,
		"-an", // no audio
		"-vf", fmt.Sprintf("scale=%d:%d", width, height),
		"-pix_fmt", "bgr24",
		"-f", "rawvideo",
		"pipe:1",
	}

	cmd := exec.Command("ffmpeg", args...)

	stdout, err := cmd.StdoutPipe()
	if err != nil {
		fmt.Printf("StdoutPipe error: %v\n", err)
		os.Exit(1)
	}

	// Capture stderr so you get useful errors if ffmpeg fails.
	stderr, err := cmd.StderrPipe()
	if err != nil {
		fmt.Printf("StderrPipe error: %v\n", err)
		os.Exit(1)
	}

	if err := cmd.Start(); err != nil {
		fmt.Printf("ffmpeg start error: %v\n", err)
		os.Exit(1)
	}

	// If ffmpeg prints an error and exits, read it.
	go func() {
		b, _ := io.ReadAll(stderr)
		if len(b) > 0 {
			fmt.Printf("ffmpeg stderr:\n%s\n", string(b))
		}
	}()

	img := gocv.NewMat()
	defer img.Close()

	reader := bufio.NewReaderSize(stdout, frameBytes*4)

	buf := make([]byte, frameBytes)

	for {
		// Read exactly one frame worth of bytes.
		_, err := io.ReadFull(reader, buf)
		if err != nil {
			if err == io.EOF || err == io.ErrUnexpectedEOF {
				fmt.Println("End of stream.")
			} else {
				fmt.Printf("ReadFull error: %v\n", err)
			}
			break
		}

		// Wrap bytes in a Mat. NewMatFromBytes COPIES the data into Mat-owned memory.
		// This is safe: buf can be reused each loop.
		m, err := gocv.NewMatFromBytes(height, width, gocv.MatTypeCV8UC3, buf)
		if err != nil {
			fmt.Printf("NewMatFromBytes error: %v\n", err)
			break
		}

		// Move m into img (avoid allocations over time).
		// Alternatively: img.Close(); img = m (but be careful with defer)
		m.CopyTo(&img)
		m.Close()

                // now img contains your frame
	}

	// Stop ffmpeg
	_ = cmd.Process.Kill()
	_ = cmd.Wait()
}

Another way is to use Gstreamer which could use the VideoCapture.

package main

import (
	"fmt"
	"net/url"
	"os"
	"path/filepath"

	"gocv.io/x/gocv"
)

func fileURI(path string) (string, error) {
	abs, err := filepath.Abs(path)
	if err != nil {
		return "", err
	}
	u := url.URL{Scheme: "file", Path: abs}
	return u.String(), nil
}

func main() {
	if len(os.Args) < 2 {
		fmt.Println("Usage: gstcap <input-video-file>")
		os.Exit(1)
	}
	inPath := os.Args[1]

	uri, err := fileURI(inPath)
	if err != nil {
		fmt.Println("fileURI error:", err)
		os.Exit(1)
	}

	// Choose what "different format" means for OpenCV:
	// - Convert to raw BGR (OpenCV-friendly)
	// - Optionally scale to fixed size
	width := 640
	height := 480

	// This pipeline:
	// - uridecodebin: reads file + demux + decode automatically
	// - videoconvert: colorspace conversion
	// - videoscale + caps: force size + BGR raw frames
	// - appsink: where OpenCV pulls frames from
	//
	// Notes:
	// - drop=true/max-buffers=1 helps keep latency down (useful for live; harmless for files)
	// - sync=false means "don’t sync to clock" (often smoother when pulling frames as fast as you can)
	pipeline := fmt.Sprintf(
		`uridecodebin uri=%s ! videoconvert ! videoscale ! video/x-raw,format=BGR,width=%d,height=%d ! appsink drop=true max-buffers=1 sync=false`,
		uri, width, height,
	)

	// Prefer forcing GStreamer backend when available.
	// If your gocv version doesn't have OpenVideoCaptureWithAPI / VideoCaptureGStreamer,
	// use gocv.VideoCaptureFile(pipeline) directly.
	cap, err := gocv.OpenVideoCaptureWithAPI(pipeline, gocv.VideoCaptureGStreamer)
	if err != nil {
		// Fallback attempt
		cap, err = gocv.VideoCaptureFile(pipeline)
	}
	if err != nil {
		fmt.Println("OpenVideoCapture error:", err)
		fmt.Println("Pipeline was:", pipeline)
		os.Exit(1)
	}
	defer cap.Close()

	frame := gocv.NewMat()
	defer frame.Close()

	for {
		if ok := cap.Read(&frame); !ok || frame.Empty() {
			break
		}

		// frame now has your frame/image
	}
}

[agorman]

Wow, I had no idea I could read into mats like that. I'll give that a try!

[agorman]

Thank you I used the ffmpeg pipe method and it worked link a charm!