How to use GPU for Blender rendering?

[Wuziyi616]

Hi, thanks for the great work! I want to use GPU to accelerate rendering. According to this line, I try to set the environment by:

os.environ["KUBRIC_USE_GPU"] = "1"

However, it doesn't seem to detect any GPUs as it prints:

INFO:kubric.renderer.blender:Using the following GPU Device(s): []

I bash into the docker image and copy-paste the GPU detection code into python shell, it only detects my CPU:

>>> bpy.context.preferences.addons["cycles"].preferences.get_devices()
([<bpy_struct, CyclesDeviceSettings("Intel Xeon Silver 4110 CPU @ 2.10GHz") at 0x8f6eb88>], [<bpy_struct, CyclesDeviceSettings("Intel Xeon Silver 4110 CPU @ 2.10GHz") at 0x8f6eb88>])

However, when I do nvidia-smi there is indeed output

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:06:00.0 Off |                    0 |
| N/A   43C    P0    27W /  70W |      0MiB / 15109MiB |      6%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Any thought on this? I tried on multiple machines (desktop, cluster) but they all failed

[HAWLYQ]

Hi, @Wuziyi616, I also ran into this problem, have you solved it?

[Qwlouse]

I have not used a GPU from within the docker, so I cannot comment on that. But it works nicely on my linux machine where I have built and installed the Blender python module natively. That way of installing can be a bit finicky, but on Ubuntu, you should be able to follow the dockerfile to make it work.

[Wuziyi616]

@HAWLYQ unfortunately I didn't solve it... @Qwlouse Thanks for the reply. It would be very tricky to build it on HPC but I'll try it.

[ganow]

I encountered the same issue, and I solved it by changing the build script of the Docker image. Let me share my workaround below.

The current kubricdockerhub/kubruntu does not support NVIDIA Container Toolkit because it uses ubuntu:20.04 as its base image. This may be the reason why bpy, on which kubric internally depends, does not recognize the GPU devices even if we run the Docker container with --gpus all option.

The above problem was solved by

• changing the base image in the docker/Blender.dockerfile from ubuntu:20.04 to nvidia/cuda:11.4.1-devel-ubuntu20.04 and
• building Blender Python API with CUDA support (add the option set(WITH_CYCLES_CUDA_BINARIES ON CACHE BOOL "" FORCE) to the cmake config file).

The following commit contains the actual changes: ganow@206eecd

The execution log is as follows:

$ docker run --rm \
             --interactive \
             --gpus all \
             --env KUBRIC_USE_GPU=1 \
             --volume "$(pwd):/kubric" \
             kubricdockerhub/kubruntu \
             /usr/bin/python3 examples/helloworld.py
INFO:kubric.renderer.blender:Using the following GPU Device(s): ['NVIDIA GeForce RTX 2080 Ti', 'NVIDIA GeForce RTX 2080 Ti']
INFO:kubric.renderer.blender:Saving 'output/helloworld.blend'
INFO:kubric.renderer.blender:Using scratch rendering folder: '/tmp/tmpdwy3_o_l'
INFO:kubric.renderer.blender:Rendered frame '/tmp/tmpdwy3_o_l/images/frame_0001.png'
INFO:root:Writing to 'output/helloworld.png'
INFO:root:Writing to 'output/helloworld_segmentation.png'
INFO:root:Writing to 'output/helloworld_depth.png'
INFO:root:Depth scale: {'min': 3.359675884246826, 'max': 11.91185188293457}
Error: Not freed memory blocks: 1, total unfreed memory 0.001259 MB

[Wuziyi616]

Hi @ganow thank you so much for your reply, the solution seems very reasonable.

However, since I'm new to docker, can you kindly tell me how to build the docker image after applying your modification? Say I've git clone your repo, then how should I do? I guess I cannot do docker pull kubricdockerhub/kubruntu because we don't want to pull from the official kubric repo. Maybe something similar to docker build -t kubric-gpu -f docker/Kubruntu.Dockerfile .? I tried that but it didn't work, and I guess that's because I also need to build the dockerfile of Blender. But I'm not sure how to build several docker images.

Thanks in advance!

[ganow]

Hi @Wuziyi616.

can you kindly tell me how to build the docker image after applying your modification?

Would you try the following commands?

$ git clone https://github.com/ganow/kubric.git
$ cd kubric
$ docker build -f docker/Blender.Dockerfile -t kubricdockerhub/blender:latest .  # build a blender image first
$ docker build -f docker/Kubruntu.Dockerfile -t kubricdockerhub/kubruntu:latest .  # then build a kubric image of which base image is the blender image above

The above command overwrites the name of the existing Docker image locally, so perhaps you should give it a different name for the GPU-enabled image like blender-gpu and kubric-gpu instead of kubricdockerhub/blender:latest and kubricdockerhub/kubruntu:latest. You can change the image name by changing the string after the -t option to anything you like. Note that, in that case, please change the name of the base image specified in the FROM statement in docker/Kubruntu.Dockerfile line 10 to the corresponding one.

kubric/docker/Kubruntu.Dockerfile

Line 10 in a3d5eaa

FROM kubricdockerhub/blender:latest

BTW, thanks to this issue I was able to find the option to enable GPU in kubric. Thank you very much!

[Wuziyi616]

@ganow Ah yes I forgot to change the name of this image haha. I tried your solution and it works! Thank you so much for your kind reply. Glad this issue help :)

[Wuziyi616]

A few problems I've encountered and how I solve them:

• Running docker build -f docker/Kubruntu.Dockerfile -t kubricdockerhub/kubruntu:latest . reports error:

Step 11/13 : COPY dist/kubric*.whl .
COPY failed: no source files were specified

See #88, you need to build the kubric wheel packages first by running python setup.py bdist_wheel

• When running the command with --gpus all --env KUBRIC_USE_GPU=1 it says:

docker: Error response from daemon: could not select device driver "" with capabilities: [[GPU]].

See this issue, this is likely because you didn't install the nvidia-container-toolkit. I follow the instructions here and successfully run the kubric with GPU.

Hope this helps!

[Wuziyi616]

@ganow just out of curiosity, do you benchmark the performance improvement after using GPU for rendering? I tried to render some 1024x1024 images with a 3090 GPU (the program indeed prints INFO: kubric.renderer.blender: Using the following GPU Device(s): ['NVIDIA GeForce RTX 3090']), but didn't see much speed improvement compared to purely using CPUs.

[ganow]

@Wuziyi616 To be honest, I have confirmed no difference in rendering performance between CPU and GPU (NVIDIA GeForce RTX 2080 Ti), even in my environment. However, in my case, the target I wanted to render was 128px and not very high resolution, so the computing resources may not necessarily be the bottleneck, and I cannot determine if it is a configuration error.

If there is no performance difference even when rendering 1024px images, I thought that maybe I have the wrong settings somewhere. If you can use a Blender GUI, you may load the .blend file generated by kubric with the GUI App and measure how long it takes to render it.

[ganow]

@Wuziyi616

A few problems I've encountered and how I solve them:

According to these problems,

you need to build the kubric wheel packages first by running python setup.py bdist_wheel

Sorry, I forgot to mention this procedure. Thank you for providing a full description.

When running the command with --gpus all --env KUBRIC_USE_GPU=1 it says: ...

Yes, we need to set up NVIDIA docker first.

[Quasimondo]

Thanks for the pointers and help how to enable GPU rendering! Now that I managed to make it run I am wondering how I could find out whether Blender actually is using the GPU at all since as you already noted, it does not feel like there is any speedup.

What I find strange is that whilst my log shows
INFO:kubric.renderer.blender:Using the following GPU Device(s): ['NVIDIA GeForce RTX 3090', 'Quadro RTX 6000']
I do not see any processes actually using CUDA when calling nvidia-smi during the blender rendering path, neither from inside the docker image nor from the outside.

I am wondering if there are some additional steps required in the configuration? The blender docs say:

To enable GPU rendering, go into the Preferences ‣ System ‣ Cycles Render Devices, and select either CUDA, OptiX, HIP, or Metal. Next, you must configure each scene to use GPU rendering in Properties ‣ Render ‣ Device.

[Quasimondo]

I dug around a bit and as it turns out the current code does only detect the GPU devices, but it does not activate them to be actually used. I found a code snippet (here: https://blender.stackexchange.com/questions/154249/how-setup-cycles-compute-device-in-console) which does the activation and changed the use_gpu() setter method in blender.py like this:

@use_gpu.setter
  def use_gpu(self, value: bool):
    self.blender_scene.cycles.device = "GPU" if value else "CPU"
    if value:
      # call get_devices() to let Blender detect GPU devices
      preferences = bpy.context.preferences
      cycles_preferences = preferences.addons['cycles'].preferences
      cuda_devices, opencl_devices = cycles_preferences.get_devices()
      cycles_preferences.compute_device_type = "CUDA"

      for device in cuda_devices:
        logger.info("Activating: %s", device.name)
        device.use = True

And voilá now I am seeing a 5x speedup for my larger renders - instead of 50 seconds it takes only 10 seconds.

[Wuziyi616]

I can confirm that the solution here works. nvidia-smi shows there is python process using GPU. Thanks @ganow @Quasimondo so much for the help!

FYI: I'm using 3090, the performance of rendering 128x128 images is similar, but 1024x1024 is much faster.

[Orchidaceae]

Since I have recently tried what's been provided in this thread I want to share what worked for me.

• I had a strange problem with applying the patch enable_cuda_patch.txt as described in @ganow 's workaround:

The following commit contains the actual changes: ganow@206eecd

my solution was to add -N to the patch command in Blender.Dockerfile

RUN cd blender && patch -N -p1 < /blenderpy/blender/enable_cuda_patch.txt

• Then I also needed to use another version of the CUDA docker image, since nvidia/cuda:11.4.1-devel-ubuntu20.04 is no longer available. I used the following in Blender.Dockerfile :

FROM nvidia/cuda:11.0.3-devel-ubuntu20.04 as build

...

FROM nvidia/cuda:11.0.3-devel-ubuntu20.04

• I also used the updated use_gpu() as provided by @Quasimondo.

To check available devices use the following:

import bpy

deviceList = bpy.context.preferences.addons["cycles"].preferences.get_devices()
for deviceTuple in deviceList:
    print("Devices:")
    for device in deviceTuple:
        print(f"\t{device.name} ({device.type}) {device.use}")

• Then I also encountered an issue with libopenexr when running kubric inside the docker environment. This was solved by following the steps here: undefined symbol: _ZTIN7Iex_2_27BaseExcE

Hope this helps anyone wanting to utilize the GPU powered kubric-docker.

[jiaming-ai]

@use_gpu.setter
def use_gpu(self, value: bool):
self.blender_scene.cycles.device = "GPU" if value else "CPU"
if value:
# call get_devices() to let Blender detect GPU devices
preferences = bpy.context.preferences
cycles_preferences = preferences.addons['cycles'].preferences
cuda_devices, opencl_devices = cycles_preferences.get_devices()
cycles_preferences.compute_device_type = "CUDA"

  for device in cuda_devices:
    logger.info("Activating: %s", device.name)
    device.use = True

Thanks @Orchidaceae , you comment is very helpful!

[Eliyas0007]

Thank you all for exposing and solving this problem!

Just to add: you need to change the blender.py in docker. For example you can bind your local file with the file in container by adding this line to docker/Kubruntu.Dockerfile:

COPY kubric/renderer/blender.py  /usr/local/lib/python3.9/dist-packages/kubric/renderer/blender.py

You can change the code locally, which is adding use_gpu() to blender.py, then re-build the container:

docker build -f docker/Kubruntu.Dockerfile -t kubricdockerhub/kubruntu:latest .

Maybe useful to some rookies like myself.