python3: proper syntax for Windows patches

[greg-hellings]

Windows patches were being added with a shell glob, which was dying with
an error. See #351007. Use filesystem methods to walk the patch set and
retrieve the patch names.

Also, update the patches from the Fedora project to the latest for
Python 3.11.9 and add filtering for any that are already applied to the
3.11.10 in the current cycle.

python312 still fails to cross-compile after this patch, but at least
3.11 will again be available

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[wolfgangwalther]

Only looked at the code, didn't try to build.

LGTM except for the comment with the version number?

[FliegendeWurst]

It seems the patches are outdated (building pkgsCross.{mingw32,mingwW64}.python3)

Running phase: unpackPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking source archive /nix/store/60ma51qz0f36kjs1wxyzbmkddlvkphnn-Python-3.12.8.tar.xz
source root is Python-3.12.8
setting SOURCE_DATE_EPOCH to timestamp 1733251361 of file Python-3.12.8/pyconfig.h.in
Running phase: patchPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
applying patch /nix/store/s5myj3ybdn7wrsfpydj48mm24vnvw6bc-no-ldconfig.patch
patching file Lib/ctypes/util.py
Hunk #1 succeeded at 271 (offset 3 lines).
applying patch /nix/store/r112dk8w7zvdjipki58ch00m825li7fq-virtualenv-permissions.patch
patching file Lib/venv/__init__.py
Hunk #1 succeeded at 509 with fuzz 2 (offset 130 lines).
applying patch /nix/store/lpfs02z2i2hamz9f50p4bz6v0fs0600j-mimetypes.patch
patching file Lib/mimetypes.py
Hunk #1 succeeded at 46 (offset 6 lines).
applying patch /nix/store/7f26mgj9vx5izbkbfirkd4m0dyxlrkvv-platform-triplet-detection.patch
patching file configure.ac
Hunk #1 succeeded at 941 (offset 5 lines).
Hunk #2 succeeded at 1174 (offset 5 lines).
applying patch /nix/store/6ldh5mz1w8nyfp5xhfbp9hwhc093yfrk-mingw-python-patches/mingw-python3_distutils.patch
can't find file to patch at input line 4
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -rupN --no-dereference Python-3.11.9/Lib/distutils/ccompiler.py Python-3.11.9-new/Lib/distutils/ccompiler.py
|--- Python-3.11.9/Lib/distutils/ccompiler.py	2024-04-02 10:25:04.000000000 +0200
|+++ Python-3.11.9-new/Lib/distutils/ccompiler.py	2024-08-26 10:10:38.769671701 +0200
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
3 out of 3 hunks ignored
can't find file to patch at input line 35
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -rupN --no-dereference Python-3.11.9/Lib/distutils/command/build_ext.py Python-3.11.9-new/Lib/distutils/command/build_ext.py
|--- Python-3.11.9/Lib/distutils/command/build_ext.py	2024-04-02 10:25:04.000000000 +0200
|+++ Python-3.11.9-new/Lib/distutils/command/build_ext.py	2024-08-26 10:10:38.770671699 +0200
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
3 out of 3 hunks ignored
can't find file to patch at input line 77
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -rupN --no-dereference Python-3.11.9/Lib/distutils/cygwinccompiler.py Python-3.11.9-new/Lib/distutils/cygwinccompiler.py
|--- Python-3.11.9/Lib/distutils/cygwinccompiler.py	2024-04-02 10:25:04.000000000 +0200
|+++ Python-3.11.9-new/Lib/distutils/cygwinccompiler.py	2024-08-26 10:10:38.770671699 +0200
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
5 out of 5 hunks ignored
can't find file to patch at input line 140
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -rupN --no-dereference Python-3.11.9/Lib/distutils/sysconfig.py Python-3.11.9-new/Lib/distutils/sysconfig.py
|--- Python-3.11.9/Lib/distutils/sysconfig.py	2024-04-02 10:25:04.000000000 +0200
|+++ Python-3.11.9-new/Lib/distutils/sysconfig.py	2024-08-26 10:10:38.771671698 +0200
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
2 out of 2 hunks ignored
can't find file to patch at input line 163
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -rupN --no-dereference Python-3.11.9/Lib/distutils/util.py Python-3.11.9-new/Lib/distutils/util.py
|--- Python-3.11.9/Lib/distutils/util.py	2024-04-02 10:25:04.000000000 +0200
|+++ Python-3.11.9-new/Lib/distutils/util.py	2024-08-26 10:10:38.771671698 +0200
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
1 out of 1 hunk ignored

[greg-hellings]

It seems the patches are outdated (building pkgsCross.{mingw32,mingwW64}.python3)

Correct. The patches Nix uses are pulled from Fedora. Fedora has not updated to 3.12 yet. Most of the bad patches are because of dropping distutils in 3.12. However, that shouldn't stop us from at least enabling 3.11 to cross compile with this change set.

If you specifically build pkgsCross.mingwW64.python311 it should work with this change in place.

[FliegendeWurst]

That one fails due to pthreads not compiling:

x86_64-w64-mingw32-ar: `u' modifier ignored since `D' is the default (see `U')
x86_64-w64-mingw32-ranlib fakelib/libgcc.a
rm -f fakelib/libgcc_eh.a
x86_64-w64-mingw32-ar cru fakelib/libgcc_eh.a  
x86_64-w64-mingw32-ar: `u' modifier ignored since `D' is the default (see `U')
x86_64-w64-mingw32-ranlib fakelib/libgcc_eh.a
rm -f fakelib/libgcc_s.a
x86_64-w64-mingw32-ar cru fakelib/libgcc_s.a  
x86_64-w64-mingw32-ar: `u' modifier ignored since `D' is the default (see `U')
x86_64-w64-mingw32-ranlib fakelib/libgcc_s.a
/nix/store/2l2galvq6zxjafp43ar4z87znk3xwccn-bash-5.2p37/bin/bash ./libtool  --tag=CC   --mode=link x86_64-w64-mingw32-gcc -Wall -DWIN32_LEAN_AND_MEAN  -g -O2 -no-undefined -version-info 1:0:0 -L./fakelib -Wc,-no-pthread  -o libwinpthread.la -rpath /nix/store/d4gbq2xccvr2z7d4kdx09hf42sy5ijv5-mingw_w64-pthreads-x86_64-w64-mingw32-12.0.0/lib src/libwinpthread_la-barrier.lo src/libwinpthread_la-cond.lo src/libwinpthread_la-misc.lo src/libwinpthread_la-mutex.lo src/libwinpthread_la-rwlock.lo src/libwinpthread_la-spinlock.lo src/libwinpthread_la-thread.lo src/libwinpthread_la-ref.lo src/libwinpthread_la-sem.lo src/libwinpthread_la-sched.lo src/libwinpthread_la-clock.lo src/libwinpthread_la-nanosleep.lo src/version.lo  
libtool: link: x86_64-w64-mingw32-gcc -shared  src/.libs/libwinpthread_la-barrier.o src/.libs/libwinpthread_la-cond.o src/.libs/libwinpthread_la-misc.o src/.libs/libwinpthread_la-mutex.o src/.libs/libwinpthread_la-rwlock.o src/.libs/libwinpthread_la-spinlock.o src/.libs/libwinpthread_la-thread.o src/.libs/libwinpthread_la-ref.o src/.libs/libwinpthread_la-sem.o src/.libs/libwinpthread_la-sched.o src/.libs/libwinpthread_la-clock.o src/.libs/libwinpthread_la-nanosleep.o src/.libs/version.o   -L./fakelib  -g -O2 -no-pthread   -o .libs/libwinpthread-1.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libwinpthread.dll.a
/nix/store/rcxfzrx1vk7kcv81x3r7gy01dlzy4ddv-x86_64-w64-mingw32-binutils-2.43.1/bin/x86_64-w64-mingw32-ld: src/.libs/libwinpthread_la-thread.o: in function `pthread_create_wrapper':
/build/mingw-w64-v12.0.0/mingw-w64-libraries/winpthreads/src/thread.c:1544:(.text+0xb38): undefined reference to `__intrinsic_setjmpex'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:706: libwinpthread.la] Error 1
make[2]: Leaving directory '/build/mingw-w64-v12.0.0/mingw-w64-libraries/winpthreads'
make[1]: *** [Makefile:908: all-recursive] Error 1
make[1]: Leaving directory '/build/mingw-w64-v12.0.0/mingw-w64-libraries/winpthreads'
make: *** [Makefile:514: all] Error 2

[greg-hellings]

That one fails due to pthreads not compiling:

x86_64-w64-mingw32-ar: `u' modifier ignored since `D' is the default (see `U')
x86_64-w64-mingw32-ranlib fakelib/libgcc.a
rm -f fakelib/libgcc_eh.a
x86_64-w64-mingw32-ar cru fakelib/libgcc_eh.a  
x86_64-w64-mingw32-ar: `u' modifier ignored since `D' is the default (see `U')
x86_64-w64-mingw32-ranlib fakelib/libgcc_eh.a
rm -f fakelib/libgcc_s.a
x86_64-w64-mingw32-ar cru fakelib/libgcc_s.a  
x86_64-w64-mingw32-ar: `u' modifier ignored since `D' is the default (see `U')
x86_64-w64-mingw32-ranlib fakelib/libgcc_s.a
/nix/store/2l2galvq6zxjafp43ar4z87znk3xwccn-bash-5.2p37/bin/bash ./libtool  --tag=CC   --mode=link x86_64-w64-mingw32-gcc -Wall -DWIN32_LEAN_AND_MEAN  -g -O2 -no-undefined -version-info 1:0:0 -L./fakelib -Wc,-no-pthread  -o libwinpthread.la -rpath /nix/store/d4gbq2xccvr2z7d4kdx09hf42sy5ijv5-mingw_w64-pthreads-x86_64-w64-mingw32-12.0.0/lib src/libwinpthread_la-barrier.lo src/libwinpthread_la-cond.lo src/libwinpthread_la-misc.lo src/libwinpthread_la-mutex.lo src/libwinpthread_la-rwlock.lo src/libwinpthread_la-spinlock.lo src/libwinpthread_la-thread.lo src/libwinpthread_la-ref.lo src/libwinpthread_la-sem.lo src/libwinpthread_la-sched.lo src/libwinpthread_la-clock.lo src/libwinpthread_la-nanosleep.lo src/version.lo  
libtool: link: x86_64-w64-mingw32-gcc -shared  src/.libs/libwinpthread_la-barrier.o src/.libs/libwinpthread_la-cond.o src/.libs/libwinpthread_la-misc.o src/.libs/libwinpthread_la-mutex.o src/.libs/libwinpthread_la-rwlock.o src/.libs/libwinpthread_la-spinlock.o src/.libs/libwinpthread_la-thread.o src/.libs/libwinpthread_la-ref.o src/.libs/libwinpthread_la-sem.o src/.libs/libwinpthread_la-sched.o src/.libs/libwinpthread_la-clock.o src/.libs/libwinpthread_la-nanosleep.o src/.libs/version.o   -L./fakelib  -g -O2 -no-pthread   -o .libs/libwinpthread-1.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libwinpthread.dll.a
/nix/store/rcxfzrx1vk7kcv81x3r7gy01dlzy4ddv-x86_64-w64-mingw32-binutils-2.43.1/bin/x86_64-w64-mingw32-ld: src/.libs/libwinpthread_la-thread.o: in function `pthread_create_wrapper':
/build/mingw-w64-v12.0.0/mingw-w64-libraries/winpthreads/src/thread.c:1544:(.text+0xb38): undefined reference to `__intrinsic_setjmpex'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:706: libwinpthread.la] Error 1
make[2]: Leaving directory '/build/mingw-w64-v12.0.0/mingw-w64-libraries/winpthreads'
make[1]: *** [Makefile:908: all-recursive] Error 1
make[1]: Leaving directory '/build/mingw-w64-v12.0.0/mingw-w64-libraries/winpthreads'
make: *** [Makefile:514: all] Error 2

Hmm, then that's a change since I made this branch and needs to be addressed in a different PR. I can look into that later. But I've just rebuilt it again here, locally, directly out of my branch. Are you testing a merge of this PR into master?

[FliegendeWurst]

I merged into staging. But if the patches are correct for you (you tested with latest 3.11, right?), then this PR can be merged.

[greg-hellings]

It was 3.11.10 at the time, which was latest. It looks like 3.11.11 is out. A quick scan through the Fedora patchset looks like it hasn't changed at all in that update, only in the auto-generated comments listing the version numbers in the file pathnames that their tooling used to generate the patches. That won't affect application of the patch, plus when I just tried rebasing to master (saw the same winpthread failure you did), it it still building 3.11.10 so it looks like 3.11.11 hasn't made it into nixpkgs yet.

[greg-hellings]

@FliegendeWurst I opened a new issue to track the winpthreads failure and put my initial work into there. I have been able to work around the build failure, but it required me to change which C library all of the MinGW stack was building against. So that seems a tad drastic.