Improving pythran-config so SciPy et al can use it

[rgommers]

Hey @serge-sans-paille, I am looking at streamlining the SciPy build, and would like to start using pythran-config. To get that to work, I need a cleaner output from pythran-config --cflags though. What it looks like now for me:

$ pythran-config --cflags
-DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_BLAS -I/home/rgommers/code/tmp/pythran/pythran -I/home/rgommers/mambaforge/envs/scipy-dev-py312/lib/python3.12/site-packages/numpy/_core/include -I/home/rgommers/mambaforge/envs/scipy-dev-py312/lib/python3.12/site-packages/numpy/_core/include -I/home/rgommers/mambaforge/envs/scipy-dev-py312/include/python3.12

# or
$ pythran-config --cflags --no-python
-D__PYTHRAN__=3 -DPYTHRAN_BLAS_BLAS -I/home/rgommers/code/tmp/pythran/pythran

The first flavor is way too much; the second is better but requires:

1. -DENABLE_PYTHON_MODULE to still be there (this is pythran-specific, while the Python include paths are not)
2. -DPYTHRAN_BLAS_BLAS to turn into -DPYTHRAN_BLAS_NONE

Also, it should work without setuptools.

I'm happy to make the changes needed, but there's a decent amount of design freedom here so I thought it's better to open an issue to discuss first. Options I can think of:

• A separate --cflags-pythran-only flag combined with a --[no-]blas flag
• All in one options: --cflags-pythran-noblas and --cflags-pythran-blas
• New flags that influences the existing --cflags, e.g. --cflags --pythran-only --no-blas (this would be a bigger refactor it looks like)

The correct/desired output for SciPy as well as scikit-image looks like:

-DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_NONE -I/home/rgommers/code/tmp/pythran/pythran

Any preferences?

[serge-sans-paille]

Hey Ralf,
I agree with you: pythran-config needs improvement. We could learn from python-config

% python-config -h
Usage: /usr/bin/python3.12-x86_64-config --prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help|--abiflags|--configdir|--embed

and eventually pkg-confi, but man, the verbosity of its --help!

Maybe the output of llvm-config --help may be relevant there too

% llvm-config --help
usage: llvm-config <OPTION>... [<COMPONENT>...]

Get various configuration information needed to compile programs which use
LLVM.  Typically called from 'configure' scripts.  Examples:
  llvm-config --cxxflags
  llvm-config --ldflags
  llvm-config --libs engine bcreader scalaropts

Options:
  --assertion-mode  Print assertion mode of LLVM tree (ON or OFF).
  --bindir          Directory containing LLVM executables.
  --build-mode      Print build mode of LLVM tree (e.g. Debug or Release).
  --build-system    Print the build system used to build LLVM (e.g. `cmake` or `gn`).
  --cflags          C compiler flags for files that include LLVM headers.
  --cmakedir        Directory containing LLVM CMake modules.
  --components      List of all possible components.
  --cppflags        C preprocessor flags for files that include LLVM headers.
  --cxxflags        C++ compiler flags for files that include LLVM headers.
  --has-rtti        Print whether or not LLVM was built with rtti (YES or NO).
  --help            Print a summary of llvm-config arguments.
  --host-target     Target triple used to configure LLVM.
  --ignore-libllvm  Ignore libLLVM and link component libraries instead.
  --includedir      Directory containing LLVM headers.
  --ldflags         Print Linker flags.
  --libdir          Directory containing LLVM libraries.
  --libfiles        Fully qualified library filenames for makefile depends.
  --libnames        Bare library names for in-tree builds.
  --libs            Libraries needed to link against LLVM components.
  --link-shared     Link the components as shared libraries.
  --link-static     Link the component libraries statically.
  --obj-root        Print the object root used to build LLVM.
  --prefix          Print the installation prefix.
  --shared-mode     Print how the provided components can be collectively linked (`shared` or `static`).
  --system-libs     System Libraries needed to link against LLVM components.
  --targets-built   List of all targets currently built.
  --version         Print LLVM version.
Typical components:
  all               All LLVM libraries (default).
  engine            Either a native JIT or a bitcode interpreter.

What about

I think we at least need:

1. --cppflags
2. --cflags
3. --ldflags

It would be good to be able to pass extra configuration parameters like --config compiler.blas=none because some of them would have an influence on the output of the above flags.

In a similar manner to <COMPONENT> from llvm-config we could have a <TARGET> that could be python (the default) or nopython?

Any thoughts?

[rgommers]

and eventually pkg-config,

Yes indeed:) That said, pythran-config is nicer in Pythran's case because it's accessible when you install Pythran in a virtualenv or conda env; a pythran.pc file would not be.

Maybe the output of llvm-config --help may be relevant there too

And numpy-config and pybind11-config. Those have less knobs to turn, but are also Python packages so in some ways more relevant than python and llvm which aren't.

$ numpy-config --help
usage: numpy-config [-h] [--version] [--cflags] [--pkgconfigdir]

options:
  -h, --help      show this help message and exit
  --version       Print the version and exit.
  --cflags        Compile flag needed when using the NumPy headers.
  --pkgconfigdir  Print the pkgconfig directory in which `numpy.pc` is stored (useful for
                  setting $PKG_CONFIG_PATH).

$ pybind11-config --help
usage: pybind11-config [-h] [--version] [--includes] [--cmakedir] [--pkgconfigdir]

options:
  -h, --help      show this help message and exit
  --version       Print the version and exit.
  --includes      Include flags for both pybind11 and Python headers.
  --cmakedir      Print the CMake module directory, ideal for setting -Dpybind11_ROOT in CMake.
  --pkgconfigdir  Print the pkgconfig directory, ideal for setting $PKG_CONFIG_PATH.

I think we at least need:

1. `--cppflags`
2. `--cflags`
3. `--ldflags`

One question: since Pythran is C++ only, is --cflags actually needed, or only --cppflags?

Other than that, I think I agree. Although I wouldn't use --ldflags, --libs is probably a bit more appropriate (might come down to the same thing though).

The key point for --cppflags is that it should _not_contain anything for other dependencies, only compile flags that are specific to Pythran. Because anything else is going to be (a) duplicate, and (b) broken. The build system should be responsible for adding dependencies like Python and NumPy, which is going to pull in the correct compiler and linker flags. The current pythran-config --cflags is for example broken when cross compiling, since it's going to insert the build (native) env's include path rather than the host (non-native) ones for Python and NumPy (pybind11 gets this wrong too I think).
This is the type of thing where pkg-config tends to be better, since a pythran.pc can express "I depend on Python and NumPy", while in a custom -config tool that isn't really possible - trying to handroll the compile flags for another dependency isn't the solution though.

It would be good to be able to pass extra configuration parameters like --config compiler.blas=none because some of them would have an influence on the output of the above flags.

That sounds like a good idea to me. As does the <TARGET> for python/nopython.

Slightly related: can -D__PYTHRAN__=3 be made the default and the define dropped now? Python 2 is very much dead after all.

[serge-sans-paille]

One question: since Pythran is C++ only, is --cflags actually needed, or only --cppflags?

technically, only --cflags is needed, but it might be good to inherit the same cflags as the one from python-config to have the same optimization levels, -fPIC and such?

I don't see --libs outputing the linker path (-L) but let's follow common usage there.

[rgommers]

but it might be good to inherit the same cflags as the one from python-config to have the same optimization levels, -fPIC and such?

I don't think so. That will turn into a giant mess (as is python3-config). A build system (CMake, Meson, etc.) has a standard interface for this type of flag/behavior, with a reasonable default (e.g., see b_staticpic in this table of options for Meson) and with ways for distro packagers and end users to override the default per build (e.g, via CFLAGS). Individual tools like pythran-config, which are used by build systems to handle build targets that depend on Pythran, should not have any opinion on flags like -fPIC, that will lead to either duplicate or conflicting flags.

You want to be setting the flags that are essential for Pythran itself only. No general-purpose flags.

I don't see --libs outputing the linker path (-L) but let's follow common usage there.

There are no -L flags to add. These should be on the linker search path already. This is one that python3-config does seem to get right, this looks reasonable:

$ python3-config --libs
 -lpthread -ldl  -lutil -lm

-L isn't needed, and if it were then there's no way for pythran-config to know what the paths should be.

[serge-sans-paille]

You want to be setting the flags that are essential for Pythran itself only. No general-purpose flags.

ack

I don't see --libs outputing the linker path (-L) but let's follow common usage there. There are no -L flags to add. These should be on the linker search path already. This is one that python3-config does seem to get right, this looks reasonable: $ python3-config --libs -lpthread -ldl -lutil -lm -L isn't needed, and if it were then there's no way for pythran-config to know what the paths should be.

so it should provide location of libpython?

[rgommers]

so it should provide location of libpython?

No, I think at most it should return -lpython (semantically "Pythran-generated code has the need to link against libpython"), nothing else. libpython.so is normally located in <prefix>/lib so it's on the search path, and if it's elsewhere then it should be found through a dependency('python') call in the build system (Meson syntax, the same functionality is present in CMake, setuptools et al).

I'm honestly not even sure if -lpython is useful or a good idea. This is almost always needed ¹ and present when any build system builds an extension module, so it's kinda redundant - it won't be a problem for me to have it there, but I don't plan to use the output of --libs anyway because it's already taken care of. I'll note that pybind11-config and numpy-config both don't provide --libs.

Footnotes

1. Except the argument looks different for statically linking libpython, and there are platforms where one must not link libpython. Again this is something that Pythran can't really know, it's logic like https://github.com/mesonbuild/meson/blob/9f3f88feed74518f406839f4935b79a45d364540/mesonbuild/scripts/python_info.py#L66-L79 in build systems that deal with it. ↩