[python-package] Failing to build on mac using gcc

[sharmuz]

Description

Due to the OpenMP conflict issue, I'm attempting to build lightgbm from source using gcc.

However, I'm getting the following build error:

Using CPython 3.11.10
Creating virtual environment at: .venv
Resolved 220 packages in 750ms
error: Failed to prepare distributions
  Caused by: Failed to download and build `lightgbm==4.5.0`
  Caused by: Build backend failed to build wheel through `build_wheel` (exit status: 1)

[stdout]
*** scikit-build-core 0.10.7 using CMake 3.31.0 (wheel)
*** Configuring CMake...
loading initial cache file /var/folders/xs/m56qg795587f_dccm8bt12z80000gp/T/tmpkdomlx5a/build/CMakeInit.txt
-- The C compiler identification is GNU 14.2.0
-- The CXX compiler identification is GNU 14.2.0
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/local/bin/gcc-14 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Checking whether CXX compiler has -isysroot
-- Checking whether CXX compiler has -isysroot - yes
-- Checking whether CXX compiler supports OSX deployment target flag
-- Checking whether CXX compiler supports OSX deployment target flag - yes
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/local/bin/g++-14 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp
-- Found OpenMP_CXX: -fopenmp
-- Found OpenMP: TRUE
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done (6.4s)
-- Generating done (0.0s)
-- Build files have been written to: /var/folders/xs/m56qg795587f_dccm8bt12z80000gp/T/tmpkdomlx5a/build
*** Building project with Ninja...
[1/36] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o
FAILED: CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o
/usr/local/bin/g++-14 -DEIGEN_DONT_PARALLELIZE -DEIGEN_MPL2_ONLY -DMM_MALLOC -DMM_PREFETCH -DUSE_SOCKET -I/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/eigen -I/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fast_double_parser/include -I/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include -I/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/include -fopenmp -pthread -Wextra -Wall -Wno-ignored-attributes -Wno-unknown-pragmas -Wno-return-type -O3 -fPIC -funroll-loops -O3 -DNDEBUG -std=gnu++11 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX15.1.sdk -mmacosx-version-min=14.7 -MD -MT CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o -MF CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o.d -o CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o -c /Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/src/boosting/prediction_early_stop.cpp
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX15.1.sdk/usr/include/_wchar.h:90,
                 from /Library/Developer/CommandLineTools/SDKs/MacOSX15.1.sdk/usr/include/wchar.h:67,
                 from /usr/local/Cellar/gcc/14.2.0_1/include/c++/14/cwchar:44,
                 from /usr/local/Cellar/gcc/14.2.0_1/include/c++/14/bits/postypes.h:40,
                 from /usr/local/Cellar/gcc/14.2.0_1/include/c++/14/bits/char_traits.h:42,
                 from /usr/local/Cellar/gcc/14.2.0_1/include/c++/14/string:42,
                 from /Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/include/LightGBM/prediction_early_stop.h:10,
                 from /Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/src/boosting/prediction_early_stop.cpp:5:
/usr/local/Cellar/gcc/14.2.0_1/lib/gcc/current/gcc/x86_64-apple-darwin23/14/include-fixed/stdio.h:83:8: error: 'FILE' does not name a type
   83 | extern FILE *__stdinp;
      |        ^~~~
/usr/local/Cellar/gcc/14.2.0_1/lib/gcc/current/gcc/x86_64-apple-darwin23/14/include-fixed/stdio.h:81:1: note: 'FILE' is defined in header '<cstdio>'; this is probably fixable by adding '#include <cstdio>'
   80 | #include <sys/_types/_seek_set.h>
  +++ |+#include <cstdio>
   81 |
/usr/local/Cellar/gcc/14.2.0_1/lib/gcc/current/gcc/x86_64-apple-darwin23/14/include-fixed/stdio.h:84:8: error: 'FILE' does not name a type
   84 | extern FILE *__stdoutp;
      |        ^~~~
/usr/local/Cellar/gcc/14.2.0_1/lib/gcc/current/gcc/x86_64-apple-darwin23/14/include-fixed/stdio.h:84:8: note: 'FILE' is defined in header '<cstdio>'; this is probably fixable by adding '#include <cstdio>'
/usr/local/Cellar/gcc/14.2.0_1/lib/gcc/current/gcc/x86_64-apple-darwin23/14/include-fixed/stdio.h:85:8: error: 'FILE' does not name a type
   85 | extern FILE *__stderrp;
      |        ^~~~
<SNIP>
/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include/fmt/format-inl.h:1454:32: error: 'f' was not declared in this scope; did you mean 'fp'?
 1454 | FMT_FUNC void print(std::FILE* f, string_view text) {
      |                                ^
      |                                fp
/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include/fmt/format-inl.h:1454:47: error: expected primary-expression before 'text'
 1454 | FMT_FUNC void print(std::FILE* f, string_view text) {
      |                                               ^~~~
/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include/fmt/format-inl.h:1459:15: error: variable or field 'vprint' declared void
 1459 | FMT_FUNC void vprint(std::FILE* f, string_view fmt, format_args args) {
      |               ^~~~~~
/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include/fmt/format-inl.h:1459:27: error: 'FILE' is not a member of 'std'
 1459 | FMT_FUNC void vprint(std::FILE* f, string_view fmt, format_args args) {
      |                           ^~~~
/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include/fmt/format-inl.h:1459:33: error: 'f' was not declared in this scope
 1459 | FMT_FUNC void vprint(std::FILE* f, string_view fmt, format_args args) {
      |                                 ^
/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include/fmt/format-inl.h:1459:48: error: expected primary-expression before 'fmt'
 1459 | FMT_FUNC void vprint(std::FILE* f, string_view fmt, format_args args) {
      |                                                ^~~
/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include/fmt/format-inl.h:1459:65: error: expected primary-expression before 'args'
 1459 | FMT_FUNC void vprint(std::FILE* f, string_view fmt, format_args args) {
      |                                                                 ^~~~
/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include/fmt/format-inl.h: In function 'void fmt::v10::vprint(string_view, format_args)':
/Users/sharmuz/Library/Caches/uv/sdists-v6/pypi/lightgbm/4.5.0/4n6y_L3uPrDQInuJN7KOZ/src/external_libs/fmt/include/fmt/format-inl.h:1466:10: error: '__stdoutp' was not declared in this scope; did you mean 'stdout'?
 1466 |   vprint(stdout, fmt, args);
      |          ^~~~~~
ninja: build stopped: subcommand failed.

[stderr]
2024-11-12 10:30:05,930 - scikit_build_core - INFO - RUN: /usr/local/bin/cmake -E capabilities
2024-11-12 10:30:05,944 - scikit_build_core - INFO - CMake version: 3.31.0
2024-11-12 10:30:05,949 - scikit_build_core - INFO - Build directory: /private/var/folders/xs/m56qg795587f_dccm8bt12z80000gp/T/tmpkdomlx5a/build
2024-11-12 10:30:05,995 - scikit_build_core - INFO - RUN: /Users/sharmuz/Library/Caches/uv/builds-v0/.tmpFoM0io/lib/python3.11/site-packages/ninja/data/bin/ninja --version
2024-11-12 10:30:06,307 - scikit_build_core - INFO - Ninja version: 1.11.1
2024-11-12 10:30:06,312 - scikit_build_core - INFO - RUN: /usr/local/bin/cmake -S. -B/var/folders/xs/m56qg795587f_dccm8bt12z80000gp/T/tmpkdomlx5a/build -DCMAKE_BUILD_TYPE:STRING=Release -C/var/folders/xs/m56qg795587f_dccm8bt12z80000gp/T/tmpkdomlx5a/build/CMakeInit.txt -DCMAKE_INSTALL_PREFIX=/var/folders/xs/m56qg795587f_dccm8bt12z80000gp/T/tmpkdomlx5a/wheel/platlib -DCMAKE_MAKE_PROGRAM=/Users/sharmuz/Library/Caches/uv/builds-v0/.tmpFoM0io/lib/python3.11/site-packages/ninja/data/bin/ninja -D__BUILD_FOR_PYTHON:BOOL=ON
2024-11-12 10:30:12,780 - scikit_build_core - INFO - RUN: /usr/local/bin/cmake --build /var/folders/xs/m56qg795587f_dccm8bt12z80000gp/T/tmpkdomlx5a/build --target _lightgbm

*** CMake build failed

Reproducible example

• Install cmake and gcc:

brew install cmake
brew install gcc

• Use uv with following pyproject.toml to install lightgbm from source:

[project]
name = "minimal"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = "==3.11.10"
dependencies = [
    "lightgbm==4.5.0",
]

[tool.uv]
no-binary-package = ["lightgbm"]
package = true

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

• Remove any cached lightgbm installs:

uv cache clean lightgbm

• Install packages into environment:

CC=gcc-14 CXX=g++-14 uv sync --refresh

Environment info

LightGBM version: 4.5.0
Python version: 3.11.10
OS version: macOS Sonoma 14.7.1 (on a 2020 Intel MBP)
gcc/g++ version: 14.2.0 (also tried 11.5.0)

[jameslamb]

Thanks for the report and for using LightGBM.

Before we try to investigate, could you please tell us what happens when you remove these other intermediate layers you've introduced (uv and hatchling), and try building with gcc following the installation docs here (https://github.com/microsoft/LightGBM/tree/master/python-package)?

export CC=gcc-14
export CXX=g++-14 
pip install --no-binary 'lightgbm' 'lightgbm==4.5.0'