Support for custom build steps

[jaraco]

I've seen cases where users wish to add custom steps to the build process (example, example). Historically, users have managed to work around the issue by overriding the build or install commands (using cmdclass), but this approach requires replacing the default command just to inject some behavior. Preferable would be to avoid having to override the default behavior and for Setuptools to present a protocol for extending build steps.

And indeed, distutils already provides a mechanism to extend build steps through sub_commands.

My proposal is that the build command present an interface to append or prepend commands to the build subcommands such that a third-party provider could provide an independent command, expose it through distutils.commands, and either implicitly as part of the import or explicitly by the end user, append or prepend that command to the subcommands.

I think it should be possible to demonstrate this behavior today just by mutating the sub_commands list directly, though I'd probably recommend a proper API for users to use.

[jaraco]

In demo-custom, I've created a project that relies on a custom-build command without any overriding of build or install commands.

This approach required the explicit addition of the command in the setup.py for demo-custom.

But it does demo the concept:

draft $ pip wheel -v git+https://github.com/jaraco/demo-custom 2>&1 | tail -n 32
Created temporary directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-unpack-4tfm8tmb
Building wheels for collected packages: demo-custom
  Created temporary directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-wheel-xpii40jz
  Destination directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-wheel-xpii40jz
  Building wheel for demo-custom (PEP 517): started
  Running command /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp3izxy0r8
  running bdist_wheel
  running build
  running custom-build
  brrrrrrrrr
  installing to build/bdist.macosx-10.9-x86_64/wheel
  running install
  running install_egg_info
  running egg_info
  writing demo_custom.egg-info/PKG-INFO
  writing dependency_links to demo_custom.egg-info/dependency_links.txt
  writing top-level names to demo_custom.egg-info/top_level.txt
  writing manifest file 'demo_custom.egg-info/SOURCES.txt'
  Copying demo_custom.egg-info to build/bdist.macosx-10.9-x86_64/wheel/demo_custom-0.1.dev1+gffa9928-py3.9.egg-info
  running install_scripts
  creating build/bdist.macosx-10.9-x86_64/wheel/demo_custom-0.1.dev1+gffa9928.dist-info/WHEEL
  creating '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-wheel-xpii40jz/tmpi7uk802o/demo_custom-0.1.dev1+gffa9928-py3-none-any.whl' and adding 'build/bdist.macosx-10.9-x86_64/wheel' to it
  adding 'demo_custom-0.1.dev1+gffa9928.dist-info/METADATA'
  adding 'demo_custom-0.1.dev1+gffa9928.dist-info/WHEEL'
  adding 'demo_custom-0.1.dev1+gffa9928.dist-info/top_level.txt'
  adding 'demo_custom-0.1.dev1+gffa9928.dist-info/RECORD'
  removing build/bdist.macosx-10.9-x86_64/wheel
  Building wheel for demo-custom (PEP 517): finished with status 'done'
  Created wheel for demo-custom: filename=demo_custom-0.1.dev1+gffa9928-py3-none-any.whl size=1147 sha256=e9ce3ed5131a843264242225b5e80c25e4a4b1e39f3eddedb5d946208951eafa
  Stored in directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ephem-wheel-cache-o0hxe9tb/wheels/2b/94/c3/ca414870dfa5f5a106895a32bc0b3837efe8cda48776c7cf71
Successfully built demo-custom
Removed build tracker: '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-tracker-2ku320jw'

Note running custom-build and brrrrrrrrr.

[Nick-Hall]

This functionality would be useful for building the Gramps software. We have additional build steps for translations and manual pages.

[basil96]

Thank you, @jaraco , for providing an example of this functionality. I was struggling to implement a custom build step for PyInstaller using cmdclass in setup.cfg with a custom override of build. I tried implementing that using #2571, but it fails for editable pip install -e . installs with a ModuleNotFoundError when it encounters the name of the custom module in setup.cfg.

Your approach as shown here is much cleaner and actually works. Looking forward to the finalized API.

[abravalheri]

This approach required the explicit addition of the command in the setup.py for demo-custom.

I think we can get around requiring the users to explicitly add the sub-command, by changing the plugin to use finalize_distribution_option to automatically insert the sub-command.

I did created a proof of concept in https://github.com/abravalheri/experiment-setuptools-plugin.

@jaraco what do you think of this approach?

I think in the long term we will want to go for a different entry-point, however, this solution seems to be usable nowadays without any extra change.

[ax3l]

With Python 3.12 on our doorstep, I am searching right now for recommended ways to replace (#2928)
https://github.com/AMReX-Codes/pyamrex/blob/23.08/setup.py#L8-L9

from distutils.command.build import build
from distutils.command.clean import clean

for steps such as
https://github.com/AMReX-Codes/pyamrex/blob/23.08/setup.py#L22C1-L47C1

class CopyPreBuild(build):
    def initialize_options(self):
        build.initialize_options(self)
        # We just overwrite this because the default "build" (and "build/lib")
        # clashes with directories many developers have in their source trees;
        # this can create confusing results with "pip install .", which clones
        # the whole source tree by default
        self.build_base = "_tmppythonbuild"

    def run(self):
        # remove existing build directory
        c = clean(self.distribution)
        # ...

I do not understand the example above to replace these with setuptools, as the migration guide suggests
https://peps.python.org/pep-0632/#migration-advice

Since #2928 is closed, what is the current recommended practice to migrate such logic? :)

[abravalheri]

Hi @ax3l , if you are using the latest versions of setuptools the existing code should still keep working even on Python 3.12.

You can use setuptools.command.build.build and maybe shutil.rmtree to replace the distutils imports if you want? Does that work for you.