Intel GPU Monitoring question

[brikim]

I have an intel gpu and passed the /dev/dri:/dev/dri in my docker container and do not see the GPU show up. Is there anything else I have to do to get this to work. I pass the same thing into my emby and plex containers for hw transcoding and they can use the device. I didn't see anything else in the example docker compose I am supposed to do.

Thx for any help.

[Profex]

I face the same issue for a arcane-headless installation running inside LXC / Proxmox. RenderD128 device is visible inside arcane-headless container (see below output from arcane-headless shell), but GPU monitoring box does not show up in dashboard view. On the same Docker / LXC / Proxmox installation I also have a frigate container running with GPU configured, and frigate can access and use GPU without issues. So basic LXC / Proxmox configuration is ok.

/ # ls -la /dev/dri
total 0
drwxr-xr-x    2 root     root            60 Nov 18 05:44 .
drwxr-xr-x    6 root     root           360 Nov 18 05:44 ..
crw-rw----    1 root     104       226, 128 Nov 18 05:44 renderD128

And just as a side remark: on another system with NVidia RTX 4070 TI GPU this works just fine (GPU usage box is showing on arcane dashboard view, displaying correct GPU model and GPU usage).

[brikim]

@kmendell @mrtimothyduong Does the GPU monitoring only work with nvidia gpu's? Thx

[kmendell]

I don't have a way to test anything other than nvidia, the docs website has examples of other GPU's though that we're given from the PR

[Profex]

As per PR example the only thing you need to configure for Intel is below

    devices:
      # Intel GPU device passthrough
      - /dev/dri:/dev/dri

This is what I configured, and once you do this you will see the /dev/dri/renderD128 device on arcane side. As outlined above this is the case for me and I can see it, but still the GPU info box is not displayed inside arcane. So me feeling is the Intel GPU monitoring has still an issue (while NVidia GPU monitoring works perfectly fine using the configuration outlined in the Nvidia GPU example).

Some more details on my GPU (Intel i915 / HD Graphics 620), captured from inside the container and showing the Intel render device is accessible

# apk add pciutils
(1/3) Installing hwdata-pci (0.395-r0)
(2/3) Installing pciutils-libs (3.13.0-r1)
(3/3) Installing pciutils (3.13.0-r1)
Executing busybox-1.37.0-r19.trigger
OK: 16 MiB in 39 packages
/app # lspci -nnk -s 00:02.0
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 620 [8086:5916] (rev 02)
        DeviceName: Onboard - Video
        Subsystem: Intel Corporation Device [8086:2068]
        Kernel driver in use: i915

# apk add lshw
(1/3) Installing libgcc (14.2.0-r6)
(2/3) Installing libstdc++ (14.2.0-r6)
(3/3) Installing lshw (02.20-r1)
Executing busybox-1.37.0-r19.trigger
OK: 22 MiB in 42 packages
/app # lshw -c video
  *-display                 
       description: VGA compatible controller
       product: HD Graphics 620
       vendor: Intel Corporation
       physical id: 2
       bus info: pci@0000:00:02.0
       logical name: fb0
       version: 02
       width: 64 bits
       clock: 33MHz
       capabilities: vga_controller bus_master cap_list rom fb
       configuration: depth=32 driver=i915 latency=0 resolution=3840,2160
       resources: irq:142 memory:db000000-dbffffff memory:90000000-9fffffff ioport:f000(size=64) memory:c0000-dffff

[brikim]

I searched the logs and don't see anything print out that has GPU in the name (not sure if there are any). I don't think the intel one is working though it doesn't show up on my local machine or my headless one both have intel GPU's.

[brikim]

Not sure if you want me to but I can write a bug to track it.

[brikim]

@kmendell Is the intel-gpu-tools package installed in the container? I tried to run intel_gpu_top in the container and it says not found it looks like the code might be looking for this

[kmendell]

I dont recall so, if you add it into the container does it work?

[mrtimothyduong]

I'll need to get an Intel dGPU / iGPU to properly test, I'll have some mental bandwidth next week to hopefully look into this. I have a old Windows laptop with an iGPU somewhere.

[Profex]

Looking at your code in system_handler.go the main issue for Intel GPU seems to be that you rely on intel_gpu_top to detect if an Intel GPU is present.

	// Check for Intel GPU
	if path, err := exec.LookPath("intel_gpu_top"); err == nil {
		h.gpuDetectionCache.Lock()
		h.gpuDetectionCache.detected = true
		h.gpuDetectionCache.gpuType = "intel"
		h.gpuDetectionCache.toolPath = path
		h.gpuDetectionCache.timestamp = time.Now()
		h.gpuDetectionCache.Unlock()
		h.detectionDone = true
		slog.InfoContext(ctx, "Intel GPU detected", slog.String("tool", "intel_gpu_top"), slog.String("path", path))
		return nil
	}

The problem: intel_gpu_top is not available in Arcane, so this test will always fail.

/app # which intel_gpu_top
/app #

From what I have found intel_gpu_top for alpine can be installed via igt-gpu-tools package, however this package is only available in alpine:edge repository in testing branch, so we can't install it via apk add igt-gpu-tools in Alpine 3.22 with main and community repositories configured.

[Profex]

Maybe this helps to further troubleshoot

I now did a test upgrade to arcane:edge on my Arcane container using below.

Run arcane container in privileged mode using by adding below to compose for arcane (required for intel_gpu_top)
privileged: true

Once container is up and running in privileged mode configure arcane edge repositories including testing repositiry in /etc/apk/repositories

https://dl-cdn.alpinelinux.org/alpine/edge/main
https://dl-cdn.alpinelinux.org/alpine/edge/community
@testing https://dl-cdn.alpinelinux.org/alpine/edge/testing

Save file, and then execute

apk update
apk upgrade
reboot

After container is restarted install igt-gpu-tools with below command
apk add igt-gpu-tools@testing

Once installation is complete I am able to execute intel_gpu_top in container shell, and it shows valid results

intel-gpu-top: Intel Kabylake (Gen9) @ /dev/dri/card1 -   58/  57 MHz;  90% RC6
    0.20/ 3.31 W;      132 irqs/s

      IMC reads:     1302 MiB/s
     IMC writes:      440 MiB/s

         ENGINES     BUSY                                        MI_SEMA MI_WAIT
       Render/3D    2.30% |▉                                   |      0%      0%
         Blitter    0.00% |                                    |      0%      0%
           Video    1.25% |▌                                   |      0%      0%
    VideoEnhance    0.00% |                                    |      0%      0%

PID     Render/3D         Blitter           Video        VideoEnhance    NAME   

Unfortunately even then GPU Infobox is still not displayed on Arcane Dashboard, and In arcane log I see the following:

STDOUT      Nov 20 20:36:41.787 INF Intel GPU detected tool=intel_gpu_top path=/usr/bin/intel_gpu_top
STDOUT      Nov 20 20:36:44.787 WRN Failed to execute intel_gpu_top error="signal: killed"
STDOUT      Nov 20 20:36:47.790 WRN Failed to execute intel_gpu_top error="signal: killed"

[kmendell]

igt-gpu-tools only has a amd64 version no arm64 version, so this may be tricky in order to add it as one image wouldnt be able to use it... Honestly a easier option is maybe to just switch from alpine to debain slim...

[mrtimothyduong]

i was thinking about that (changing image base) originally when coding for nvidia, but stuck to alpine.

I'm currently reviewing this using an ancient laptop with an igpu intel HD Graphics 520 and seeing what I can do without igt etc.

[kmendell]

I got it showing up now with this image: ghcr.io/getarcaneapp/arcane:debian-test

Still a WIP as parsing the stats for this is a PIA. but at least it shows up now..

[brikim]

@kmendell Don't know if you saw Profex reply.

[kmendell]

Oh i never build the agent version of it. However based on my findings, intel_gpu_top doesnt give vram stats or anything, as it shares the system memory, it should be able to show the usage though. I rebased the debian chnages off into main and put the other changes on a another branch, ill try to look more this weekend.

[Profex]

Oh i never build the agent version of it.

Was expecting that, thanks for confirmation🙂Will be happy to test again once there is a test agent build that supports Intel GPU monitoring.

[Profex]

I now tested again on the v1.10.0 agent version based on Debian Trixie.

There is no GPU monitoring in Arcane dashboard, and in Arcane agent logs I can see this:

arcane-agent STDOUT Nov 30 07:09:56.868 INF Intel GPU detected tool=intel_gpu_top path=/usr/bin/intel_gpu_top
arcane-agent STDOUT Nov 30 07:09:59.870 WRN Failed to execute intel_gpu_top error="signal: killed"

Executing intel_gpu_top in arcane agent shell works fine, and gives valid results:

intel-gpu-top: Intel Kabylake (Gen9) @ /dev/dri/card1 -   60/  61 MHz;  89% RC6
    0.21/ 3.48 W;      132 irqs/s

      IMC reads:     1365 MiB/s
     IMC writes:      434 MiB/s

         ENGINES     BUSY                                        MI_SEMA MI_WAIT
       Render/3D    2.24% |▉                                   |      0%      0%
         Blitter    0.00% |                                    |      0%      0%
           Video    1.39% |▌                                   |      0%      0%
    VideoEnhance    0.00% |                                    |      0%      0%

PID     Render/3D         Blitter           Video        VideoEnhance    NAME   

[mrtimothyduong]

@kmendell , wanted to continue the conversation somewhere else outside of the PR 1034.

RE: Intel-based dgpu vs igpu & shared-vram: I'm going to review https://github.com/b3rs3rk/gpustat-unraid and see how they approach this.

Specifically line 172 onwards https://github.com/b3rs3rk/gpustat-unraid/blob/dev/src/gpustat/usr/local/emhttp/plugins/gpustat/lib/Intel.php

[mrtimothyduong]

After doing some review, i think we should update the GPU Monitoring doc page and remove Intel / AMD since it's still WIP.

My bad on that, i didn't have the resources to test it originally. I'll do a PR in the meantime with arcane-docs

[mrtimothyduong]

PR Raised for this: getarcaneapp/website#128

[mrtimothyduong]

FYI Intel+amd based gpu monitoring has been removed from the docs and will be reviewing how to implement!

[Impact123]

This gets spammed in the log every few seconds

INF Intel GPU detected tool=intel_gpu_top path=/usr/bin/intel_gpu_top
WRN Failed to execute intel_gpu_top error="exit status 1"
WRN Failed to execute intel_gpu_top error="exit status 1"
WRN Failed to execute intel_gpu_top error="exit status 1"
WRN Failed to execute intel_gpu_top error="exit status 1"

I do not have a intel CPU and do not pass any GPU to the container either

# docker exec arcane intel_gpu_top
No device filter specified and no discrete/integrated i915 devices found

# docker exec arcane grep "model name" /proc/cpuinfo | uniq
model name      : AMD Ryzen 7 5700G with Radeon Graphics

It seems like the code simply checks for the existance of the intel_gpu_top binary.

[kinghat]

ya im getting a bunch of this in the logs but it looks like it found it after tossing everything at it:

STDOUT Dec 01 03:46:16.070 WRN Failed to execute intel_gpu_top error="exit status 1"
STDOUT Dec 01 03:46:18.069 WRN Failed to execute intel_gpu_top error="exit status 1"
STDOUT Dec 01 03:46:20.070 WRN Failed to execute intel_gpu_top error="exit status 1"
STDOUT Dec 01 03:46:22.063 INF Intel GPU detected tool=intel_gpu_top path=/usr/bin/intel_gpu_top
STDOUT Dec 01 03:46:22.070 WRN Failed to execute intel_gpu_top error="exit status 1"
STDOUT Dec 01 03:46:24.070 WRN Failed to execute intel_gpu_top error="exit status 1"
STDOUT Dec 01 03:46:26.070 WRN Failed to execute intel_gpu_top error="exit status 1"

how jellyfin suggests passing the igpu: https://jellyfin.org/docs/general/post-install/transcoding/hardware-acceleration/intel#configure-with-linux-virtualization

    group_add:
      - 107      
    devices:
      # - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri:/dev/dri

arcane v1.10.0

also, why does arcane need my gpu passed?

[mrtimothyduong]

I'll do a PR and create a environment boolen for arcane and arcane-headless (agent), e.g. gpu-monitoring, default is false

[mrtimothyduong]

Testing required from others:

1. Please update your existing Docker Compose image: with either: (Note, hosted on docker-hub, not gh)

• mrtimothyduong/arcane-headless:latest
• mrtimothyduong/arcane:latest

2. Add the following env var to your Docker Compose .yaml

• GPU_MONITORING_ENABLED=true
• GPU_TYPE=auto (alternative entries are nvidia, note: amd and intel are accepted but needs to be implemented as per other work with debian re-base)

Testing completed:

• Able to deploy test arcane-headless image to x86 host without env-var flags (no gpu stats)
• Able to deploy test arcane-headless image to x86 host without env-var flags true + auto
• Able to deploy test arcane-headless image to x86 host without env-var flags true + nvidia

Let me know how testing goes and I'll raise PR. PR Raised via #1034

[Profex]

I tested the mrtimothyduong/arcane-headless:latest image on two of my headless agent systems which have a GPU available.
Status before tests:

• System 1: Intel GPU - GPU monitoring not working, and log spammed with "Failed to execute intel_gpu_top error="signal: killed""
• System 2: NVidia GPU - GPU monitoring working and no errors in log

System 1: Intel GPU

GPU_MONITORING_ENABLED=false ==> no GPU usage displayed on dashboard, no "Failed to execute intel_gpu_top error="signal: killed" message flood in arcane log

GPU_MONITORING_ENABLED=true // GPU_TYPE=auto ==> no GPU usage displayed on dashboard, log shows

arcane-agent  | Dec 02 17:28:00.619 INF Intel GPU detected tool=intel_gpu_top path=/usr/bin/intel_gpu_top
arcane-agent  | Dec 02 17:28:03.620 WRN Failed to execute intel_gpu_top error="signal: killed"

GPU_MONITORING_ENABLED=true // GPU_TYPE=intel ==> no GPU usage displayed on dashboard, log shows

arcane-agent  | Dec 02 17:36:08.175 INF Using configured GPU type type=intel
arcane-agent  | Dec 02 17:36:11.176 WRN Failed to execute intel_gpu_top error="signal: killed"

System 2: Nvidia GPU

GPU_MONITORING_ENABLED=false ==> no GPU usage displayed on dashboard, no error messages in arcane log

GPU_MONITORING_ENABLED=true // GPU_TYPE=auto ==> GPU usage displayed on dashboard, log shows

arcane-agent | Dec 02 17:32:59.616 INF NVIDIA GPU detected tool=nvidia-smi path=/usr/bin/nvidia-smi

GPU_MONITORING_ENABLED=true // GPU_TYPE=nvidia ==> GPU usage displayed on dashboard, log shows

arcane-agent | Dec 02 17:34:00.002 INF Using configured GPU type type=nvidia

[mrtimothyduong]

Hi Profex, confirming that is as intended as per my comment, but apologies and I was not clear it is still not functioning:
"note: amd and intel are accepted but needs to be implemented as per other work with debian re-base"

RE: Intel and/or AMD monitoring will be worked in future versions, this PR is to address:

• unneccessary CPU cycles / gpu monitoring must be explicitly be enabled/toggled via env var

[Profex]

Yes, I knew that Intel still wouldn't work. I saw your comment about it. I just wanted to share the results I received during the tests. 🙂