Expose compiler classes for customized use

[minrk]

Summary

The porting from distutils docs are a bit sparse, and could use some more specific examples. Specifically, for distutils.ccompiler and distutils.util.get_platform were the replacements I was looking for, and didn't find.

Maybe it would be good to start populating a FAQ section at or linked from the docs. Starting out with finishing the the table in the pep with actual recommended import substitutions that work (4 are there already), would be a huge help for migrators like myself.

OS / Environment

No response

Additional Information

In attempting to migrate https://github.com/zeromq/pyzmq away from distutils, I came up with some questions that don't appear to have clear answers in docs, and it seems like setuptools is the right place for such docs (if that's wrong, what is the right place?).

The pep lists setuptools as the thing to switch to for several APIs, such as ccompiler, etc., but at least ccompiler does not appear to be available from public setuptools APIs. Is importing form setuptools._distutils the 'right' way in general to get distutils functionality that hasn't been exposed at public level of setuptools yet? Or are there plans to expose these as top-level APIs (e.g. setutools.ccompiler)?

I can import new_compiler and customize_compiler from setuptools.command.build_ext, but that doesn't seem better than setuptools._distutils.ccompiler to me. If it were documented as the suggested alternative, I'd be happy, though.

Could we perhaps have a statement in the docs about when, if ever, importing from setuptools._distutils is appropriate? The text already there:

If a project relies on uses of distutils that do not have a suitable replacement above, please search the Setuptools issue tracker and file a request

suggests anything that requires importing from setuptools._distutils should result in an Issue (how I got here), but maybe could be more explicit about what to do while finding a resolution. A blessing of "import from setuptools._distutils" in the interim would ease some stress on migrators, but may not be what you all want to support.

The pep also lists the platform module as the replacement for distutils.util.get_platform, but in my experience this is not the right advice. Where I've used distutils.util.get_platform(), what I really wanted was the build platform string, which is available as dist.get_command_obj('build').plat_name. I don't always call it from within a command so this is a bit inconvenient, but I think I can manage. I don't think it's possible to construct this from the platform module.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[althonos]

I would also be interested in the migration process for distutils.ccompiler. I have several libraries that are using the interal CCompiler API to detect build-support for SIMD code. In my opinion, the distutils.ccompiler module has a lot of legacy code that doesn't always work (for instance, using a UNIX compiler on Windows through MinGW barely works) so I'd think it would be great to have new top-level APIs to work with. I'd be happy to contribute to this.

[minrk]

FWIW, this is how I've created compilers without calling private APIs:

from setuptools import setup
from setuptools import Extension
from setuptools.command.build_ext import build_ext

class GetCompiler(build_ext):
    def finalize_options(self):
        super().finalize_options()
        # need to have an Extension for `.run()` to create self.compiler
        self.extensions = [Extension("unused", ["unused.c"])]

    def build_extensions(self):
        """Empty to skip actual compilation"""
        # because we need self.extensions to be defined and unused,
        # we _also_ need to skip actually building them
    
    def run(self):
        super().run()
        print(f"Got my compiler! {self.compiler}")
        

setup(
    cmdclass={"compiler": GetCompiler},
)

Relying on build_ext's logic to create the compiler. The tedious bit is that build_ext only creates the compiler if it has some extensions to compile. So I have to pass it an unused Extension and also override build_extensions to then ignore those unused Extensions.

Then I replaced all my calls to new_compiler/configure_compiler to passing around copies of self.compiler.

To always get a fresh compiler, you can do the same thing by patching a build_ext on an empty Distribution:

from setuptools import Distribution, Extension

def get_compiler():
    d = Distribution()
    build_ext = Distribution().get_command_obj("build_ext")
    build_ext.finalize_options()
    # register an extension to ensure a compiler is created
    build_ext.extensions = [Extension("ignored", ["ignored.c"])]
    # disable building fake extensions
    build_ext.build_extensions = lambda: None
    # run to populate self.compiler
    build_ext.run()
    return build_ext.compiler

[neutrinoceros]

Hi @abravalheri, do you think this could get some "official" answer ?

[abravalheri]

Hi @neutrinoceros, compilers are still outside of my area of expertise in setuptools. Recently I got more acquainted with build_ext but I still don't feel like I can contribute directly on this discussion. Sorry for that.

I noticed that in #2372, @jaraco seems to be open to implement some changes in order to support the migration, e.g.:

• provide setuptools extensions to implement additional functionality where appropriate; setuptools could add extendable hooks if that helps.
• creating suitably-compatible versions of public interfaces entirely in the setuptools namespace and weaning users and packages off of import distutils*.
• explore what interfaces the users need for the distutils compiler classes/modules and consolidate the behaviours.

I think that one thing that can help in this process is to collect what are the requirements for the developers (and potentially, if you have any suggestions, what would be the desired interfaces).

Would you guys be happy if setuptools simply exposes part of the setuptools._distutils.ccompiler interface under setuptools.ccompiler (or another public module)? What are the functions/classes that are needed from this module? Is there a use case for using this API outside of a setuptools command?

---

Where I've used distutils.util.get_platform(), what I really wanted was the build platform string, which is available as dist.get_command_obj('build').plat_name. I don't always call it from within a command so this is a bit inconvenient, but I think I can manage. I don't think it's possible to construct this from the platform module.

What would be the use case for running get_platform outside of a setuptools command?

[neutrinoceros]

Thanks for your reply ! I will answer on behalf of yt

Would you guys be happy if setuptools simply exposes part of the setuptools._distutils.ccompiler interface under setuptools.ccompiler (or another public module)?

Pretty much, though any official recommendation would satisfy us.

What are the functions/classes that are needed from this module?

CCompiler and new_compiler are the only components that we use.

Is there a use case for using this API outside of a setuptools command?

Not that I know of.

[neutrinoceros]

As a side note, we also rely on distutils.sysconfig.customize_compiler. Should I open a different ticket or is this relevant enough to this conversation ?

[abravalheri]

Thank you very much @neutrinoceros. If the other maintainers are ok with exposing this interface, I can go ahead and try something out (please also feel free to submit a PR - I am a bit busy for July so it may take time...).

As a side note, we also rely on distutils.sysconfig.customize_compiler. Should I open a different ticket or is this relevant enough to this conversation ?

I would not mind to track them together...

[neutrinoceros]

@abravalheri @minrk I've opened a PR to try to address this: #3445
Your feedback is very welcome

[abravalheri]

Thank you very much @neutrinoceros, that is very helpful.

I think we should ask Jason to review the PR to see if he agrees with this direction.

[FirefoxMetzger]

I'm also interested in getting the CCompiler, because I want to write a custom command to compile a shared(!) library that is then accessed by ctypes. Clarifications on how this is meant to be done would help a ton 🚀

We can do this for statically linked archives (.lib) using the build_clib command and the libraries kwarg of setup(), but I couldn't find anything for dynamically linked libraries (.dll / .so). This might be slightly off-topic here, but would it be useful to discuss adding a build_shared_clib command to build such shared libraries?

[jaraco]

Thanks everybody for the patience. This issue had not risen to my attention until now, so bear with me as I catch up.

I appreciate the work done here, but I'd like to take this in a different direction.

A blessing of "import from setuptools._distutils" in the interim would ease some stress on migrators, but may not be what you all want to support.

Definitely not. That location is an implementation detail. If you wish to access distutils modules through the compatibility interface, use import distutils*. That is, import distutils.ccompiler and require that your users have setuptools installed when running that code.

I want to be cautious about simply mirroring distutils interfaces under similar names in Setuptools. I don't want to, for example, expose setuptools.ccompiler only to later realize that implies also exposing classes from setuptools.msvccompiler and setuptools._msvccompiler and setuptools.msvccompiler9.

I'm beginning to think that there may be a rationale here for having compiler functionality exposed independent of setuptools (i.e. in a compilers package), but I'm not yet sure, as the description above doesn't describe what the required use-case is. Can you help clarify the use-case for these classes/functions, i.e. under what conditions do you need to import a CCompiler? Under what conditions do you call new_compiler? What do you do with the results? Do you do this exclusively in service of building a package (alongside other Setuptools operations) or is it something your project does independently?

As a side note, we also rely on distutils.sysconfig.customize_compiler. Should I open a different ticket or is this relevant enough to this conversation ?

We're trying to deprecate everything in distutils.sysconfig in favor of sysconfig (stdlib). It's unclear to me what the need of customize_compiler is. Is it needed because one is constructing a CCompiler and needs to apply the customizations as implemented in distutils.sysconfig? Or do you need to override that function to apply additional customizations when it's called by build_clib and build_ext?

[althonos]

@jaraco : Thanks for tackling this issue!

Can you help clarify the use-case for these classes/functions, i.e. under what conditions do you need to import a CCompiler? Under what conditions do you call new_compiler? What do you do with the results? Do you do this exclusively in service of building a package (alongside other Setuptools operations) or is it something your project does independently?

I have the same use-case in several scientific computing packages, where I have several components:

• a C or C++ library that I'm writing bindings for
• a Cython extension exposing a Python interface
• several additional files which are platform-specific (e.g. a specialized SSE2 implementation of some algorithms).

In general, I need to have access to a C compiler before building the Cython extension so that I can detect some platform-specific functions and headers (like a configure script would), and also to check if I can build the specialized code (i.e. does the compiler support SSE2/AVX/NEON etc.). I have a (non-minimal) example here: https://github.com/althonos/pytrimal/blob/main/setup.py

At the moment I'm never creating a CCompiler directly, I'm actually extending the build_ext command so that I can use self.compiler, which is a CCompiler configured with customize_compiler. But if a dedicated compilers package was available, I'd likely use that instead 👍

[neutrinoceros]

our use case for yt is also very well described by @althonos' answer: we use distutils to build Cython extensions.
Here's the setupext.py file we want to port https://github.com/yt-project/yt/blob/main/setupext.py

If you wish to access distutils modules through the compatibility interface, use import distutils*. That is, import distutils.ccompiler

Wait, this is news to me. Does it mean that distutils can be imported even in Python 3.12 if setuptools is installed ? If I'm reading this right, and if you guys plan to maintain this interface, this would actually mean that nothing more needs be done to keep our setupext.py working when distutils leaves the standard library ? (though it sounds like you do want to ditch distutils.sysconfig)

[FirefoxMetzger]

My main use-case is similar to that of @althonos. Set platform-specific flags before building extensions.

The other use-case that I have is that I have a package that offers ctypes bindings into a C library. Traditionally, we installed the python package and the C library separately; however, now I would like to switch and build and package the C library at the same time we package the python package. Unfortunately, there does not seem to be a built-in way of compiling shared libraries, so I am writing my own command in which I have to call CCompiler.link_shared_lib. At the moment this is a subclass of build_clib, because it has self.compiler, but that seems like a terrible hack x)

Another thing that I would be curious to try is to use intel's C compiler when building the extensions instead of MSVC on Windows. I have to date never attempted this, but (or rather because) this would involve creating a custom CCompiler and hooking it into the packaging process. This would be a really cool thing, but maybe treat it as more of a nice-to-have ... at least from my side.

I'm beginning to think that there may be a rationale here for having compiler functionality exposed independent of setuptools (i.e. in a compilers package), but I'm not yet sure

I had the same thought while putting together the package to build the shared library for my ctypes project thingy. I think a dedicated library for this would be an excellent idea and if it does get enough support to make it happen I'm happy to contribute :)

[jaraco]

Does it mean that distutils can be imported even in Python 3.12 if setuptools is installed ?

Yes. That's correct. The only exception is that if SETUPTOOLS_USE_DISTUTILS=stdlib is set, then no distutils will be found. However, the default since Setuptools 60 is SETUPTOOLS_USE_DISTUTILS=local, so that default behavior remains unchanged across Python versions, including 3.12+.

if you guys plan to maintain this interface

We do, though it's our goal to obviate the need for accessing distutils altogether (eventually deprecating and removing it). Meaning that we do want to identify use-cases like the ones described above and provide a robust, long-term solution that doesn't involve distutils. That's why I'd like to avoid simply exposing distutils interfaces in Setuptools and instead step back and think about the problems that these approaches address. I need to create a milestone to track these efforts.

It seems to me from the responses above that there are basically these use-cases we want to meet:

• Detect and expose platform-specific configuration and capabilities relating to the compiler.
• Set platform-specific flags before building extensions (under Setuptools).
• Compile shared libraries to be included in a distribution.
• [enhancement] Provide a customized compiler to the build process (for experimentation or customization).

I'm thinking we should create new issues for each and close this issue as "invalid" as the compatibility shim already exposes the compiler classes. Does that sound right?

[FirefoxMetzger]

@jaraco Sounds like a workable solution from my side. Thanks for taking the time to move this issue forward!

[minrk]

@jaraco that sounds like a good summary. To answer your questions directly:

under what conditions do you need to import a CCompiler?

I haven't needed this.

Under what conditions do you call new_compiler?

Only during setup.py, in a pre-build step I call setup.py configure.

What do you do with the results?

Compile test libraries and executables for detection/discovery of dependencies, etc. Similar to autotools' feature/package discovery. These produce information used to configure the 'real' compiler used in build_ext. The result affects Extension objects passed to build_ext, mostly include_dirs, libraries, define_macros, and even which Extensions are to be built.

Do you do this exclusively in service of building a package (alongside other Setuptools operations) or is it something your project does independently?

I do it exclusively during package building.

• Detect and expose platform-specific configuration and capabilities relating to the compiler.
• Set platform-specific flags before building extensions (under Setuptools).
• Compile shared libraries to be included in a distribution.
• [enhancement] Provide a customized compiler to the build process (for experimentation or customization).

The first two points cover my use case.

[andyfaff]

I wouldn't rule out the possibility that people would want to compile C-code during program execution, e.g. for construction of compiled plugins.

[minrk]

Yes, I think that's a totally valid and reasonable use case. I can't speak for it myself, and can't immediately find an example. cffi, for example instantiates Distribution and Extension objects, invoking build_ext without reaching directly into the compiler.

[minrk]

Cython's inline is another runtime-compilation example where build_ext and Extension are the consumed APIs and ccompiler only comes up in their tests.

[jonathanunderwood]

Related to this, at package build time, from setup.py I detect the compiler that is in use, using get_default_compiler() from distutils, and based on that set extra_compile_args. What's the migration path here?

from distutils import ccompiler
...
if compiler == 'msvc':
    extension_kwargs['extra_compile_args'] = [
        '/Ot',
        '/Wall',
        '/wd4711',
        '/wd4820',
    ]
elif compiler in ('unix', 'mingw32'):
    if liblz4_found:
        extension_kwargs = pkgconfig_parse('liblz4')
    else:
        extension_kwargs['extra_compile_args'] = [
            '-O3',
            '-Wall',
            '-Wundef'
        ]
else:
    print('Unrecognized compiler: {0}'.format(compiler))
    sys.exit(1)

I don't see any get_default_compiler() replacement in the setuptools API.

[tfardet]

@jonathanunderwood did you find a replacement for get_default_compiler? I would also need one ^^"

[Code7R]

Same here. I fail to see how the current situation is usable as a replacement of plain access to distutils.ccompiler. setuptools._distutils.ccompiler feels like something where an architect with common sense would add at least one factory function, which would get the user access to the implementation (subclass) describing a specified compiler, or even the default compiler chosen for the OS (and/or detected as best one in doubt). I.e. have the analogous or even better version of get_default_compiler.

Our usecase here is actually simple - let Cython generate the descriptions and then amend the settings inside - by using those from get_default_compiler() and appending a few flags. This is a simple operation. And the only replacement which I can see with the current setuptools is something like sub-classing the default interface manually and adapting something (a poorly documented something). This is NOT user-friendly.

[jaraco]

In pypa/distutils#295, I've started work on refactoring the compilers functionality in distutils such that it can be exposed separately and publicaly in a new namespace. Along with the other recent work to allow Setuptools and distutils to have dependencies, consolidating MSVC support in distutils, and removing deprecated compilers, this behavior gets us closer to having an independent and re-usable library for compiler behavior. There's still some work to do as far as decoupling some of the distutils-specific behaviors, but it's getting close.

[Avasam]

Late addition to this discussion: pywin32 replaces the ccompiler.new_compiler method to wedge in its own MSVCCompiler subclass: https://github.com/mhammond/pywin32/blob/c717bfa4b06bab5984ba4c9eb62fcc4e6dcf99ef/setup.py#L921-L932