[HW Accel Support]: iGPU Proxmox LXC Passthrough

[tdashmike]

Describe the problem you are having

Frigate is not able to use the iGPU on a Debian LXC with docker. I think card1 is supposed to be used with render D128, usually it's card0 but I have another NVIDIA GPU installed which I plan on using after I can get the iGPU to work. I have also added root to the render and video groups on both the host and guest. I added configs for fb0 and tty7 but I don't know if they are necessary.

Proxmox Kernel Version

Linux 6.8.8-2-pve

Version

0.13.2-6476F8A

Frigate config file

Works fine with the hardware acceleration turned off so I'm only posting the hwaccel argument.

ffmpeg:
  hwaccel_args: preset-intel-qsv-h264
#hwaccel_args: preset-vaapi (Gives the same errors)

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "256mb" # update for your cameras based on calculation above
    devices:
#      - /dev/bus/usb:/dev/bus/usb  # Passes the USB Coral, needs to be modified for other versions
#      - /dev/apex_0:/dev/apex_0    # Passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
#      - /dev/video11:/dev/video11  # For Raspberry Pi 4B
      - /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
      - /dev/dri/card1:/dev/dri/card1
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /nas/Security/Frigate/config:/config
      - /nas/Security/Frigate/media:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "xxxx"

Relevant log output

2024-07-07 19:42:07.895073908  [2024-07-07 19:42:07] watchdog.T-FRONTDOOR           ERROR   : Ffmpeg process crashed unexpectedly for T-FRONTDOOR.
2024-07-07 19:42:07.895153004  [2024-07-07 19:42:07] watchdog.T-FRONTDOOR           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-07-07 19:42:07.895218948  [2024-07-07 19:42:07] ffmpeg.T-FRONTDOOR.detect      ERROR   : [AVHWDeviceContext @ 0x5e40aa405b00] No VA display found for device -hwaccel_output_format.
2024-07-07 19:42:07.895293742  [2024-07-07 19:42:07] ffmpeg.T-FRONTDOOR.detect      ERROR   : Device creation failed: -22.
2024-07-07 19:42:07.895358212  [2024-07-07 19:42:07] ffmpeg.T-FRONTDOOR.detect      ERROR   : Failed to set value '-hwaccel_output_format' for option 'qsv_device': Invalid argument
2024-07-07 19:42:07.895418324  [2024-07-07 19:42:07] ffmpeg.T-FRONTDOOR.detect      ERROR   : Error parsing global options: Invalid argument

FFprobe output from your camera

FFPROBE just spins.

Operating system

Proxmox

Install method

Docker Compose

Network connection

Wired

Camera make and model

Wyze Cam v3

Any other information that may be helpful

Proxmox Host:

ls -l /dev/dri/*
crw-rw---- 1 root video  226,   0 Jul  6 23:04 /dev/dri/card0
crw-rw---- 1 root video  226,   1 Jul  6 23:04 /dev/dri/card1
crw-rw---- 1 root render 226, 128 Jul  6 23:04 /dev/dri/renderD128
crw-rw---- 1 root render 226, 129 Jul  6 23:04 /dev/dri/renderD129

/dev/dri/by-path:
total 0
lrwxrwxrwx 1 root root  8 Jul  6 23:04 pci-0000:00:02.0-card -> ../card1
lrwxrwxrwx 1 root root 13 Jul  6 23:04 pci-0000:00:02.0-render -> ../renderD128
lrwxrwxrwx 1 root root  8 Jul  6 23:04 pci-0000:02:00.0-card -> ../card0
lrwxrwxrwx 1 root root 13 Jul  6 23:04 pci-0000:02:00.0-render -> ../renderD129

Subgid:

root:104:1
root:44:1
root:5:1

LXC Conf:

arch: amd64
features: nesting=1
memory: 4096
ostype: debian
...
unprivileged: 1
lxc.idmap: g 0 100000 5
lxc.idmap: g 5 5 1
lxc.idmap: g 6 100006 38
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 61
lxc.idmap: g 106 104 1
lxc.idmap: g 107 100107 893
lxc.idmap: u 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: g 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 1001 101001 64535
lxc.cgroup2.devices.allow: c 226:1 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 4:7 rwm
lxc.cgroup2.devices.allow: c 29:0 rwm
lxc.mount.entry: /dev/dri/card1 dev/dri/card1 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/tty7 dev/tty7 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file 0, 0

Groups:

getent group video | cut -d : -f3
44
getent group render | cut -d : -f3
104

Guest:

ls -l /dev/dri/*
crw-rw---- 1 nobody video  226,   1 Jul  7 03:04 /dev/dri/card1
crw-rw---- 1 nobody render 226, 128 Jul  7 03:04 /dev/dri/renderD128

Groups:

getent group video | cut -d : -f3
44
getent group render | cut -d : -f3
106

VAINFO:

error: can't connect to X server!
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.1 ()
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD

Frigate:

GPUS:

error-gpu
There was an error getting usage stats. This does not mean hardware acceleration is not working. Either your GPU does not support this or Frigate does not have proper access to get statistics. This is expected for the Home Assistant addon.

VAINFO:

VAINFO OUTPUT
Return Code: 1

Process Error:

error: XDG_RUNTIME_DIR is invalid or not set in the environment.
error: can't connect to X server!
error: failed to initialize display

[hawkeye217]

Does it work if you use preset-vaapi? That's generally recommended over qsv.

[tdashmike]

Yes, it throws the same errors though.

[hawkeye217]

In case you haven't seen this thread, there may be some helpful things: #5773

[tdashmike]

Yeah, I tried adding "lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir" but the guest is now saying "ls: cannot access '/dev/dri/*': Not a directory.

He mentioned he could only get it to work with:

On the Proxmox host console type chmod 666 /dev/dri/renderD128

This will give rw access to the renderD128 device for ALL users

but in the comment section, @Redsandro says:

The idmap is recommended because when your GPU is readable to all, it means everyone, even guests at any level inside or outside any container on the system, including unwanted guests from over the world, can potentially spy on your screen without having to do some kind of privilege escalation exploit. ⚠️

so I have not tried that yet.

[jubjubrsx]

maybe try the native port (not using docker) from ttek script? just to test.

[tdashmike]

It looks like the tteck script is changing the permission of the devices as well.

msg_info "Setting Up Hardware Acceleration"
$STD apt-get -y install {va-driver-all,ocl-icd-libopencl1,intel-opencl-icd,vainfo,intel-gpu-tools}
if [[ "$CTTYPE" == "0" ]]; then
  chgrp video /dev/dri
  chmod 755 /dev/dri
  chmod 660 /dev/dri/*
fi