Failed Vulkan

[okamototk]

I got this error during vk::createInstance:

ggml_backend_vk_reg() -> Error: System error: vk::createInstance: ErrorLayerNotPresent

How can I resolve this problem?

Environment:

• commit 7b891bd (HEAD, tag: b4686)
• MacBook Apple M2 RAM 24G + podman krunkit(+MotenVK)
• Vulkan Backend

Result

During run llama-cli, I got error at vk::createInstance:

bash-5.2# build/bin/llama-cli  -m DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf -p "what is GPU" -no-cnv
ggml_vk_instance_init()
ggml_vulkan: Validation layers enabled
ggml_backend_vk_reg() -> Error: System error: vk::createInstance: ErrorLayerNotPresent

Information for Vulkan Device(GPU0):

bash-5.2# vulkaninfo --summary                                                                                                                              15:42:11 [40/1828]
'DISPLAY' environment variable not set... skipping surface info
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
ERROR: [../src/panfrost/vulkan/panvk_physical_device.c:608] Code 0 : WARNING: panvk is not a conformant vulkan implementation, pass PAN_I_WANT_A_BROKEN_VULKAN_DRIVER=1 if you
 know what you're doing. (VK_ERROR_INCOMPATIBLE_DRIVER)
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
==========
VULKANINFO
==========

Vulkan Instance Version: 1.3.296

Instance Extensions: count = 24
-------------------------------                                                                                                                             15:42:11 [25/1828]
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_headless_surface                : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 4
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1

Instance Layers: count = 1
--------------------------
VK_LAYER_MESA_device_select Linux device selection layer 1.3.211  version 1

Devices:
========
GPU0:
        apiVersion         = 1.2.0
        driverVersion      = 24.1.5
        vendorID           = 0x106b
        deviceID           = 0xf010208
        deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
        deviceName         = Virtio-GPU Venus (Apple M2)
        driverID           = DRIVER_ID_MESA_VENUS
        driverName         = venus
        driverInfo         = Mesa 24.1.5
        conformanceVersion = 1.3.0.0
        deviceUUID         = 26b49897-c8c9-cdea-542a-91890ff14f78
        driverUUID         = 05a3cceb-1991-3628-23bd-764bdc583f20
GPU1:
        apiVersion         = 1.3.278
        driverVersion      = 0.0.1
        vendorID           = 0x10005
        deviceID           = 0x0000
        deviceType         = PHYSICAL_DEVICE_TYPE_CPU
        deviceName         = llvmpipe (LLVM 18.1.6, 128 bits)
        driverID           = DRIVER_ID_MESA_LLVMPIPE
        driverName         = llvmpipe
        driverInfo         = Mesa 24.1.5 (LLVM 18.1.6)
        conformanceVersion = 1.3.1.1
        deviceUUID         = 6d657361-3234-2e31-2e35-000000000000
        driverUUID         = 6c6c766d-7069-7065-5555-494400000000

[0cc4m]

You enabled the Validation layer from within the backend during compilation, with GGML_VULKAN_VALIDATE, but you don't have the validation layer installed. If you actually want to use the layer, you need to install it. Otherwise, set the compile flag to OFF.

[0cc4m]

Yes, they probably failed, but as Jeff said there should be information about failing pipelines in the log above. It should look something like ggml_vulkan: Compute pipeline creation failed for ...

[okamototk]

I understand, I don't have any "ggml_vulkan: Compute pipeline creation failed for" or such messages include stderr.
During debugging, I think it's not compiled with "-g" optino.

# gdb build/bin/llama-cli
GNU gdb (Fedora Linux) 15.2-4.fc40
...
Reading symbols from build/bin/llama-cli...
**(No debugging symbols found in build/bin/llama-cli)**

I use this option for build:

cmake -DCMAKE_BUILD_TYPE=Debug -DGGML_VULKAN=ON
cmake --build build --config Debug

How can I build with "-g" option?

[slp]

Looks like this binary was built with -DGGML_VULKAN_RUN_TESTS=1. That's also broken for me as of 2eea03d. For doing normal inferencing, building just with -DGGML_VULKAN=1 (on a clean build directory) should work just fine (well, the output may be garbage, but that's a molten-vk problem).

[okamototk]

@slp

Looks like this binary was built with -DGGML_VULKAN_RUN_TESTS=1. That's also broken for me as of 2eea03d. For doing normal inferencing, building just with -DGGML_VULKAN=1 (on a clean build directory) should work just fine (well, the output may be garbage, but that's a molten-vk problem).

---

Thanks, I tried without -DGGML_VULKAN_RUN_TESTS but same error as follow. Could you share

• git commit id for llama.cpp
• MoltenVK version you success to run
• Apple Silicon(M1,M2,M3,M4...)
  ?

Here is failed log just use -DGGML_VULKAN=1

# cmake --build build --target clean
# cmake -DGGML_VULKAN=ON
CMake Warning:
  No source or binary directory provided.  Both will be assumed to be the
  same as the current working directory, but note that this warning will
  become a fatal error in future CMake releases.


-- ccache found, compilation results will be cached. Disable with GGML_CCACHE=OFF.
-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- Including CPU backend
-- ARM detected
-- ARM -mcpu not found, -mcpu=native will be used
cc1: error: unknown value 'native+nodotprod+noi8mm+nosve' for '-mcpu'
cc1: note: valid arguments are: cortex-a34 cortex-a35 cortex-a53 cortex-a57 cortex-a72 cortex-a73 thunderx thunderxt88p1 thunderxt88 octeontx octeontx81 octeontx83 thunderxt81 thunderxt83 ampere1 ampere1a ampere1b emag xgene1 falkor qdf24xx exynos-m1 phecda thunderx2t99p1 vulcan thunderx2t99 cortex-a55 cortex-a75 cortex-a76 cortex-a76ae cortex-a77 cortex-a78 cortex-a78ae cortex-a78c cortex-a65 cortex-a65ae cortex-x1 cortex-x1c neoverse-n1 ares neoverse-e1 octeontx2 octeontx2t98 octeontx2t96 octeontx2t93 octeontx2f95 octeontx2f95n octeontx2f95mm a64fx tsv110 thunderx3t110 neoverse-v1 zeus neoverse-512tvb saphira cortex-a57.cortex-a53 cortex-a72.cortex-a53 cortex-a73.cortex-a35 cortex-a73.cortex-a53 cortex-a75.cortex-a55 cortex-a76.cortex-a55 cortex-r82 cortex-a510 cortex-a520 cortex-a710 cortex-a715 cortex-a720 cortex-x2 cortex-x3 cortex-x4 neoverse-n2 cobalt-100 neoverse-v2 grace demeter generic generic-armv8-a generic-armv9-a
CMake Warning at ggml/src/ggml-cpu/CMakeLists.txt:151 (message):
  Failed to get ARM features
Call Stack (most recent call first):
  ggml/src/CMakeLists.txt:318 (ggml_add_cpu_backend_variant_impl)

It's error and seems need -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=native options for ARM:

# cmake -DGGML_VULKAN=ON -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=native

Then build:

# cmake --build build --config Debug

Run inference but same errror:

# build/bin/llama-cli  -m examples/server/tests/stories15M_MOE-F16.gguf  -p "what is GPU" -no-cnv -ngl 99
...
llama_init_from_model: KV self size  =   27.00 MiB, K (f16):   13.50 MiB, V (f16):   13.50 MiB
llama_init_from_model: Vulkan_Host  output buffer size =     0.12 MiB
llama_init_from_model:    Vulkan0 compute buffer size =    63.06 MiB
llama_init_from_model: Vulkan_Host compute buffer size =     8.57 MiB
llama_init_from_model: graph nodes  = 288
llama_init_from_model: graph splits = 2
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
terminate called after throwing an instance of 'std::out_of_range'
  what():  unordered_map::at
Aborted (core dumped)

[slp]

I suspect cmake didn't actually update the build config. I would try this:

1. Install clang (if this is Fedora, dnf install clang should be enough).
2. Remove the build directory.
3. CC=clang CXX=clang++ cmake -B build -DGGML_VULKAN=1
4. cmake --build build