Update package info for system python3 on Cygwin #33078

slel · 2021-12-25T04:51:05Z

Sage 9.5.beta9 fails to use system Python 3.9 on Cygwin,
unless configured with

./configure --with-python=/usr/bin/python3

Upstream report:
https://cygwin.com/pipermail/cygwin/2021-December/250302.html

Upstream: Fixed upstream, in a later stable release.

CC: @mkoeppe @slel @darijgr

Component: porting: Cygwin

Author: Matthias Koeppe

Branch/Commit: 65418de

Reviewer: Darij Grinberg

Issue created by migration from https://trac.sagemath.org/ticket/33078

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-12-25T17:52:02Z

comment:1

logs please

mkoeppe · 2021-12-27T07:15:26Z

Branch: u/mkoeppe/use_system_python_3_on_cygwin

mkoeppe · 2021-12-27T07:32:28Z

Author: Matthias Koeppe

mkoeppe · 2021-12-27T07:32:28Z

New commits:

`9cba975`	`build/pkgs/{_prereq,python3}/distros/cygwin.txt: Switch to python 3.9`

mkoeppe · 2021-12-27T07:32:28Z

Commit: 9cba975

slel · 2021-12-28T01:00:22Z

comment:4

Fails here. Something may be wrong with my Cygwin.

Before getting Python 3.9 there, Sage would build
using system Python 3.8.12.

It now deems system Python 3.9 unusable, and builds
a lacking Python, see #33079.

Logs for failed build of Sage 9.5.beta9 + #33078:

https://aij.ovh/sage/logs/slel-2021-12-27-cyg-sage-9-5-b9-33078.zip

mkoeppe · 2021-12-28T01:06:29Z

comment:5

from config.log:

g++ -shared -Wl,--enable-auto-image-base conftest.dir/temp.cygwin-3.3.3-x86_64-3.9/conftest.o -L/home/s/sage-9-5-b9-33078/conftest_venv/lib/python3.9/config -L/usr/lib -lpython3.9 -o conftest.dir/lib.cygwin-3.3.3-x86_64-3.9/config_check_distutils_cxx.cpython-39-x86_64-cygwin.dll
	CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1"
configure:35820: result: no, this is a misconfigured Python whose sysconfig compiler/linker flags contain -I or -L options, which may cause wrong versions of libraries to leak into the build of Python packages - see https://github.com/sagemath/sage-prod/issues/31132; to use it anyway, use ./configure --with-python=/usr/bin/python3

It's the -I/usr/include/uuid/ above that our configure script does not like.

./configure --with-python=/usr/bin/python3 as you wrote in the ticket description is a good workaround.

mkoeppe · 2021-12-28T01:07:55Z

comment:6

It might be worth reporting this to Cygwin as a bug

mkoeppe · 2021-12-28T01:22:45Z

Upstream: Not yet reported upstream; Will do shortly.

mkoeppe · 2021-12-28T01:25:02Z

comment:8

This is similar to the recent bug in Homebrew's packaging of python - #31132.

mkoeppe · 2021-12-28T01:29:01Z

Changed upstream from Not yet reported upstream; Will do shortly. to Reported upstream. No feedback yet.

mkoeppe · 2021-12-28T16:57:53Z

comment:11

New cygwin package version - https://cygwin.com/pipermail/cygwin/2021-December/250304.html

mkoeppe · 2021-12-28T19:46:59Z

comment:13

Could someone who can run a local cygwin please test this branch with this cygwin package?

darijgr · 2021-12-28T19:48:55Z

comment:14

What exactly should I test?

mkoeppe · 2021-12-28T20:29:02Z

comment:15

Install python39-3.9.9-3 and python39-devel-3.9.9-3 (these are test packages),
then on this branch here run ./bootstrap && ./configure --with-system-python3=force && make build

darijgr · 2021-12-28T20:56:13Z

comment:16

Hmm. Will this trigger recompilation of all other packages, or are these standalone?

mkoeppe · 2021-12-28T22:50:15Z

comment:17

Make that ./bootstrap && ./configure --with-system-python3=force --with-sage-venv && make build. If you are already on 9.5.beta9, then this will only rebuild Python packages in a new virtual environment.

darijgr · 2021-12-29T11:19:05Z

comment:18

Sorry but I need more details. I have a freshly installed 9.5.beta9 with the regular python (and a couple patches for packages that would otherwise not have compiled). What should I type and in what order? And what should I do after that to get back to my previous install?

darijgr · 2021-12-30T19:03:51Z

comment:33

OK! I'm at the make build stage now. Hoping it doesn't again take 2 days...

New commits:

`65418de`	`.github/workflows/ci-cygwin*.yml: Use python39`

darijgr · 2021-12-31T17:00:30Z

comment:34

Confirming: mkoeppe's recipe (comment:19) works:

sage: python_help()

Welcome to Python 3.9's help utility!
[...]

I haven't tested the branch, though.

I assume I don't need to go back to the old version, seeing that it has worked?

mkoeppe · 2021-12-31T18:52:16Z

comment:35

Thanks for testing! I'll report to the cygwin maintainers that this version seems to be working well (and solves our problem).

mkoeppe · 2021-12-31T18:52:43Z

comment:36

Replying to @darijgr:

I assume I don't need to go back to the old version, seeing that it has worked?

I agree

mkoeppe · 2021-12-31T18:53:08Z

Changed upstream from Reported upstream. No feedback yet. to Fixed upstream, but not in a stable release.

mkoeppe · 2022-01-02T19:26:26Z

comment:38

Let's wait for the fixed package to arrive in the main Cygwin repository - I assume there will be an announcement in https://cygwin.com/pipermail/cygwin/2022-January/thread.html

mkoeppe · 2022-01-25T00:53:35Z

comment:40

There is now python39-3.9.10-1, needs testing

mkoeppe · 2022-03-21T19:54:30Z

comment:41

Let's move this forward please

darijgr · 2022-03-22T17:12:39Z

comment:42

Back to my office, so I'll start the install now. Expect an answer in 2 days or so (cygwin installs are slow even if lots of packages are preinstalled).

darijgr · 2022-03-24T18:48:29Z

comment:43


real    2869m24.160s
user    651m22.088s
sys     542m26.524s
***************************************************************
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make all-build'):

* package:         scipy-1.7.3
  last build time: 2390901 Mar 24
  log file:        /home/dg899/sage/logs/pkgs/scipy-1.7.3.log

Here is the end of this log file:

  gcc -shared -Wl,--enable-auto-image-base -shared -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -D_GNU_SOURCE build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_misc.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/global_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/io_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/libqhull_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/mem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/merge_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/qset_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/random_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/rboxlib_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/stat_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/user_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/usermem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_rbox_r.o -L/home/dg899/sage/local/lib -L/home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/config -L/usr/lib -Lbuild/temp.cygwin-3.3.4-x86_64-3.9 -lopenblas.dll -lopenblas.dll -lpython3.9 -o build/lib.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.cpython-39-x86_64-cygwin.dll
  /usr/lib/gcc/x86_64-pc-cygwin/11/../../../../x86_64-pc-cygwin/bin/ld: final link failed: Device or resource busy
  collect2: error: ld returned 1 exit status
  error: Command "gcc -shared -Wl,--enable-auto-image-base -shared -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -D_GNU_SOURCE build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_misc.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/global_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/io_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/libqhull_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/mem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/merge_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/qset_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/random_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/rboxlib_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/stat_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/user_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/usermem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_rbox_r.o -L/home/dg899/sage/local/lib -L/home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/config -L/usr/lib -Lbuild/temp.cygwin-3.3.4-x86_64-3.9 -lopenblas.dll -lopenblas.dll -lpython3.9 -o build/lib.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.cpython-39-x86_64-cygwin.dll" failed with exit status 1

  ########### EXT COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt.cache_flush[809] : write cache to path -> /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src/build/temp.cygwin-3.3.4-x86_64-3.9/ccompiler_opt_cache_ext.py

  ########### CLIB COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt.cache_flush[809] : write cache to path -> /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src/build/temp.cygwin-3.3.4-x86_64-3.9/ccompiler_opt_cache_clib.py
  error: subprocess-exited-with-error

  × Building wheel for scipy (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/dg899/sage/local/var/lib/sage/venv-python3.9/bin/python3 /home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpb06dquwe
  cwd: /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src
  Building wheel for scipy (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for scipy
Failed to build scipy
ERROR: Failed to build one or more wheels

This looks like a parallel-compile-related heisenbug, so I'll now try "make build" again.

darijgr · 2022-03-25T18:35:29Z

comment:44

As I hoped, it has now compiled without further problems and is working:

dg899@MATH-6264WD ~/sage
$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.6.beta5, Release Date: 2022-03-12               │
│ Using Python 3.9.10. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

mkoeppe · 2022-03-25T18:50:03Z

comment:45

Thanks for testing!

mkoeppe · 2022-03-25T18:50:03Z

Changed upstream from Fixed upstream, but not in a stable release. to Fixed upstream, in a later stable release.

mkoeppe · 2022-04-11T16:51:37Z

comment:46

I interpret this as a positive review

mkoeppe · 2022-04-11T16:51:37Z

Reviewer: Darij Grinberg

vbraun · 2022-04-21T21:07:30Z

Changed branch from u/mkoeppe/use_system_python_3_on_cygwin to 65418de

slel added this to the sage-9.5 milestone Dec 25, 2021

slel added c: porting: cygwin labels Dec 25, 2021

mkoeppe added the s: needs review label Dec 27, 2021

mkoeppe changed the title ~~Use system Python 3 on Cygwin~~ Update package info for system python3 on Cygwin Dec 27, 2021

This comment has been minimized.

Sign in to view

mkoeppe added s: needs work and removed s: needs review labels Dec 28, 2021

This comment has been minimized.

Sign in to view

mkoeppe added s: needs review and removed s: needs work labels Dec 31, 2021

mkoeppe modified the milestones: sage-9.5, sage-9.6 Jan 12, 2022

mkoeppe added p: critical / 2 and removed p: major / 3 labels Apr 11, 2022

mkoeppe added p: blocker / 1 and removed p: critical / 2 labels Apr 19, 2022

vbraun removed the s: positive review label Apr 21, 2022

vbraun closed this as completed in b5a7bd1 Apr 21, 2022

slel mentioned this issue Jan 14, 2022

Fix python3 build failure on Cygwin #33079

Closed

Update package info for system python3 on Cygwin #33078

Update package info for system python3 on Cygwin #33078

Comments

slel commented Dec 25, 2021

mkoeppe commented Dec 25, 2021

mkoeppe commented Dec 27, 2021

mkoeppe commented Dec 27, 2021

mkoeppe commented Dec 27, 2021

mkoeppe commented Dec 27, 2021

slel commented Dec 28, 2021

This comment has been minimized.

mkoeppe commented Dec 28, 2021

mkoeppe commented Dec 28, 2021

mkoeppe commented Dec 28, 2021

mkoeppe commented Dec 28, 2021

This comment has been minimized.

mkoeppe commented Dec 28, 2021

mkoeppe commented Dec 28, 2021

mkoeppe commented Dec 28, 2021

darijgr commented Dec 28, 2021

mkoeppe commented Dec 28, 2021

darijgr commented Dec 28, 2021

mkoeppe commented Dec 28, 2021

darijgr commented Dec 29, 2021

This comment has been minimized.

darijgr commented Dec 30, 2021

darijgr commented Dec 31, 2021

mkoeppe commented Dec 31, 2021

mkoeppe commented Dec 31, 2021

mkoeppe commented Dec 31, 2021

mkoeppe commented Jan 2, 2022

mkoeppe commented Jan 25, 2022

mkoeppe commented Mar 21, 2022

darijgr commented Mar 22, 2022

darijgr commented Mar 24, 2022

darijgr commented Mar 25, 2022

mkoeppe commented Mar 25, 2022

mkoeppe commented Mar 25, 2022

mkoeppe commented Apr 11, 2022

mkoeppe commented Apr 11, 2022

vbraun commented Apr 21, 2022