[User] android clang does not support "-mfp16-format" argument

[RossComputerGuy]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

llama.cpp should compile for Android

Current Behavior

Fails with:

clang++: error: unknown argument: '-mfp16-format=ieee'
ninja: build stopped: subcommand failed.

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

Use case: Flutter library

• Physical (or virtual) hardware you are using, e.g. for Linux:

neon-fp-armv8

• Operating System, e.g. for Linux:

Android

• SDK version, e.g. for Linux:

NDK 23.1.7779620

Failure Information (for bugs)

Command executed:

/usr/local/lib/android/sdk/ndk/23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ --target=armv7-none-linux-androideabi16 --gcc-toolchain=/usr/local/lib/android/sdk/ndk/23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/usr/local/lib/android/sdk/ndk/23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/sysroot -DGGML_USE_K_QUANTS -DLLAMA_BUILD -DLLAMA_SHARED -Dllama_EXPORTS -I/home/runner/work/flutter_llama/flutter_llama/src/llama.cpp/. -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -march=armv7-a -mthumb -Wformat -Werror=format-security   -O2 -g -DNDEBUG -fPIC -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -mfpu=neon-fp-armv8 -mfp16-format=ieee -mno-unaligned-access -funsafe-math-optimizations -pthread -std=gnu++11 -MD -MT llama.cpp/CMakeFiles/llama.dir/llama.cpp.o -MF llama.cpp/CMakeFiles/llama.dir/llama.cpp.o.d -o llama.cpp/CMakeFiles/llama.dir/llama.cpp.o -c /home/runner/work/flutter_llama/flutter_llama/src/llama.cpp/llama.cpp

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. Try compiling llama.cpp for Android

Failure Logs

https://github.com/ExpidusOS/flutter_llama/actions/runs/5538864077/jobs/10109227613

[rdcoder33]

+1

[useCallback]

Any solution ?

[ghost]

I'd build excluding -mfp16-format=ieee. I compile llama.cpp daily with make and cmake.

[RossComputerGuy]

@JackJollimore Is there an option to remove that argument because it looks like there isn't in CMakeLists.txt?

[ghost]

I use

cmake -B build
cd build
cmake --build . --config Release

If the above doesn't work for your device then I think editing CMakeLists.txt is one way, // out the troublesome line.

Other than editing, I'm uncertain if there's an option to remove the arguement. Flutter isn't offically supported, but maybe @slaren can weigh in on if cmake can ignore that argument during build.

[RossComputerGuy]

Gotcha, I'm hoping to avoid forking if possible. I add llama.cpp as a submodule so patching isn't necessarily that easy. I could probably make a PR that adds an option to bypass the argument or check if the argument is supported by the compiler.

[ghost]

I could probably make a PR that adds an option to bypass the argument or check if the argument is supported by the compiler.

I don't speak for llama.cpp folk, but from what I've seen that's probably the most reasonable option assuming it's done in a way the devs value.

[ggerganov]

Adding a build flag for this use case is OK, so feel free to propose a PR

I add llama.cpp as a submodule so patching isn't necessarily that easy.

In general, 3rd party projects don't have to use llama.cpp's build system. In fact, I think it is best if the project builds llama.cpp with the optimum flags and options for that specific project.

[RossComputerGuy]

I made a PR, it seems on my M1 Pro with NixOS Asahi also faces this issue without the PR.