Optimize storage_subprocess.py

[mhagnumdw]

I'm using Viseron with docker on a very modest piece of hardware, an Orange Pi Zero 3 with 1.5 GB of RAM, only recording (no transcoding), with go2rtc configured.

I have 9 cameras and I record the low-resolution stream. It works well.

I've set the recording configuration for each ffmpeg.camera, I don't use tiers, like this:

ffmpeg:
  camera:
    cam_portao:
      name: cam_portao
      host: 127.0.0.1
      port: 8554
      path: /cam_portao
      rtsp_transport: tcp
      audio_codec: null
      record_only: true
      recorder:
        audio_codec: null
        continuous_recording: true
        continuous:
          max_size:
            gb: 20

Since it's a fairly modest hardware setup, I've been observing the processes, and there's one called viseron_storage_subprocess_578 that's by far the biggest CPU and memory consumer. Right now it's constantly consuming 15% of CPU and 25% of memory. These numbers fluctuate, but it always consumes significantly more resources than the other processes.

Is there any significant optimization that can be done?

Tks for the project!

[roflcoopter]

The subprocess is responsible for figuring out what files to discard, which is quite complex.
It has already been optimized to some degree, but there might be more ways to reduce the resource usage further.

The RAM usage is a little missleading, since its a forked process it keeps references from the main process but should be garbage collected when needed.

There are a few ways you can control the process tho, limiting how fast it runs and how much CPU it is allowed to use.
Namely the options tier_check_cpu_limit, tier_check_batch_size and tier_check_sleep_between_batches.
You should be able to keep your existing configuration and just add this at the root level:

ffmpeg:
  camera:
    cam_portao:
      name: cam_portao
      host: 127.0.0.1
      port: 8554
      path: /cam_portao
      rtsp_transport: tcp
      audio_codec: null
      record_only: true
      recorder:
        audio_codec: null
        continuous_recording: true
        continuous:
          max_size:
            gb: 20

storage:
  tier_check_cpu_limit: 1
  tier_check_batch_size: 1
  tier_check_sleep_between_batches: 1

You can play around with the numbers and see if it gets any better. RAM usage probably wont be reduced tho.
Here is the docs that explains these config options:
https://viseron.netlify.app/components-explorer/components/storage

[mhagnumdw]

Tks @roflcoopter , I'll try your configuration and adjust if necessary.

It has already been optimized to some degree, but there might be more ways to reduce the resource usage further.

Since I'm not familiar with Python or Viseron logic, I decided to run an AI test to optimize storage_subprocess.py. I built the Docker image and memory consumption was significantly reduced. Apparently, the features I use were working fine.

If you're interested in seeing the code, I can publish it. The change was made from the main branch.

[roflcoopter]

Nice, yes please do, doesn't hurt to look at least

[mhagnumdw]

@roflcoopter , follow master...mhagnumdw:viseron:storage-memory

I've also included the generated documentation artifacts. Part of it is in pt-BR, so you can convert it to en-US.

I hope this helps in some way. Tks!