Support 3.11 builds on Windows

[samuelcolvin]

There doesn't seem to be a 3.11 interpreter available for builds on windows, see this build.

Full output from that step of the action

Run messense/maturin-action@v1
  with:
    target: x86_64
    manylinux: auto
    args: --release --out dist --interpreter 3.7 3.8 3.9 3.10 3.11
    rust-toolchain: stable
    token: ***
    command: build
  env:
    pythonLocation: C:\hostedtoolcache\windows\Python\3.10.8\x64
    PKG_CONFIG_PATH: C:\hostedtoolcache\windows\Python\3.10.8\x64/lib/pkgconfig
    Python_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.10.8\x64
    Python[2](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:2)_ROOT_DIR: C:\hostedtoolcache\windows\Python\[3](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:3).10.8\x6[4](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:4)
    Python3_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.10.8\x64
Found maturin version requirement maturin>=0.13,<0.14 specified in pyproject.toml
Found maturin release from manifest: v0.13.6
Install Rust target
  C:\Users\runneradmin\.cargo\bin\rustup.exe override set stable
  info: using existing install for 'stable-x86_64-pc-windows-msvc'
  info: override toolchain for 'D:\a\pydantic-core\pydantic-core' set to 'stable-x86_64-pc-windows-msvc'
  
    stable-x86_64-pc-windows-msvc unchanged - rustc 1.64.0 (a[5](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:5)5dd71d5 2022-09-19)
  
  C:\Users\runneradmin\.cargo\bin\rustup.exe component add llvm-tools-preview
  info: downloading component 'llvm-tools-preview'
  info: installing component 'llvm-tools-preview'
Install maturin
  Installing 'maturin' from tag 'v0.13.[6](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:6)'
  "C:\Program Files\PowerShell\[7](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:7)\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command "$ErrorActionPreference = 'Stop' ; try { Add-Type -AssemblyName System.IO.Compression.ZipFile } catch { } ; try { [System.IO.Compression.ZipFile]::ExtractToDirectory('D:\a\_temp\2ff2b9bd-26a3-41ad-b214-ff0cf[8](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:8)161d73', 'D:\a\_temp\3c4ce4[9](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:9)7-8c17-43be-b3b8-37862b7a509f', $true) } catch { if (($_.Exception.GetType().FullName -eq 'System.Management.Automation.MethodException') -or ($_.Exception.GetType().FullName -eq 'System.Management.Automation.RuntimeException') ){ Expand-Archive -LiteralPath 'D:\a\_temp\2ff2b9bd-26a3-41ad-b214-ff0cf8161d73' -DestinationPath 'D:\a\_temp\3c4ce497-8c17-43be-b3b8-37862b7a509f' -Force } else { throw $_ } } ;"
  Installed 'maturin' to D:\a\_temp\3c4ce497-8c17-43be-b3b8-37862b7a509f\maturin.exe
  D:\a\_temp\3c4ce497-8c17-43be-b3b8-37862b7a509f\maturin.exe --version
  maturin 0.13.6
D:\a\_temp\3c4ce497-8c17-43be-b3b8-37862b7a509f\maturin.exe build --release --out dist --interpreter 3.7 3.8 3.9 3.[10](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:10) 3.[11](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:11) --target x86_64-pc-windows-msvc
📦 Including license file "LICENSE"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
💥 maturin failed
  Caused by: The given list of python interpreters is invalid
  Caused by: Python interpreter `3.11` doesn't exist
Error: The process 'D:\a\_temp\3c4ce497-8c[17](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:18)-43be-b3b8-[37](https://github.com/pydantic/pydantic-core/actions/runs/3336013427/jobs/5520653893#step:7:40)862b7a509f\maturin.exe' failed with exit code 1

[messense]

Python 3.11 isn't in tool cache yet, have you tried install it first using the setup-python action?

[samuelcolvin]

No, I didn't know I needed to, I'll try.

[messense]

See also https://github.com/actions/runner-images/blob/win22/20221024.1/images/win/Windows2022-Readme.md#python

[samuelcolvin]

Humm, see pydantic/pydantic-core#311 - installing 3.11 seems to have removed 3.7 (and presumably all other python installs).

actions/setup-python#98 seems related, but has no solution.

[samuelcolvin]

Also note that as per here and here github don't say that 3.11 is available on macos or linux, but builds are working.

I assume that's because docker is used for linux builds, but I don't really understand what's going on with macos?

[messense]

Directory: C:\hostedtoolcache\windows\Python\3.11.0\x64

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          10/27/2022  3:02 PM                DLLs
d----          10/27/2022  3:02 PM                Doc
d----          10/27/2022  3:02 PM                include
d----          10/27/2022  3:02 PM                Lib
d----          10/27/2022  3:02 PM                libs
d----          10/27/2022  3:02 PM                Scripts
d----          10/27/2022  3:02 PM                tcl
d----          10/27/2022  3:02 PM                Tools
-a---          10/24/2022  6:35 PM          32763 LICENSE.txt
-a---          10/24/2022  6:37 PM        [14](https://github.com/messense/maturin-action/actions/runs/3338505754/jobs/5526258575#step:13:15)39691 NEWS.txt
-a---          10/26/2022 10:16 AM       25[15](https://github.com/messense/maturin-action/actions/runs/3338505754/jobs/5526258575#step:13:16)7416 python-3.11.0-amd64.exe
-a---          10/24/2022  6:34 PM         101760 python.exe
-a---          10/24/2022  6:34 PM          65920 python3.dll
la---          10/27/2022  3:02 PM              0 python3.exe ->
                                                  C:\hostedtoolcache\windows\Python\3.11.0\x64\python.exe
-a---          10/24/2022  6:34 PM        5758328 python311.dll
-a---          10/24/2022  6:34 PM         100216 pythonw.exe
-a---          10/24/2022  6:35 PM          38304 vcruntime140_1.dll
-a---          10/24/2022  6:35 PM          98736 vcruntime140.dll

    Directory: C:\hostedtoolcache\windows\Python\3.11.0\x64\Scripts

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          10/27/[20](https://github.com/messense/maturin-action/actions/runs/3338505754/jobs/5526258575#step:13:21)[22](https://github.com/messense/maturin-action/actions/runs/3338505754/jobs/5526258575#step:13:23)  3:02 PM         108414 pip.exe
-a---          10/[27](https://github.com/messense/maturin-action/actions/runs/3338505754/jobs/5526258575#step:13:28)/2022  3:02 PM         108[41](https://github.com/messense/maturin-action/actions/runs/3338505754/jobs/5526258575#step:13:42)4 pip3.10.exe
-a---          10/27/2022  3:02 PM         108414 pip3.11.exe
-a---          10/27/2022  3:02 PM         108414 pip3.exe

Ah, there isn't a python3.11.exe, see https://github.com/messense/maturin-action/actions/runs/3338505754/jobs/5526258575

[samuelcolvin]

ye, sorry meant to comment, as per latest commit on pydantic/pydantic-core#311, even with python 3.11 installed via the setup-python action, python3.11 still isn't available for maturin.

[messense]

That shouldn't be a issue because maturin use the py launcher to find it, I will investigate.

[messense]

"py" "-3.11-64" "-c" "import json\r\nimport platform\r\nimport sys\r\nimport sysconfig\r\nimport struct\r\n\r\nif platform.python_implementation() == "PyPy":\r\n # Workaround for PyPy 3.6 on windows:\r\n # - sysconfig.get_config_var("EXT_SUFFIX") differs to importlib until\r\n # Python 3.8\r\n # - PyPy does not load the plain ".pyd" suffix because it expects that's\r\n # for a CPython extension module\r\n #\r\n # This workaround can probably be removed once PyPy for Python 3.8 is the\r\n # main PyPy version.\r\n import importlib.machinery\r\n\r\n ext_suffix = importlib.machinery.EXTENSION_SUFFIXES[0]\r\nelse:\r\n ext_suffix = sysconfig.get_config_var("EXT_SUFFIX")\r\n\r\nmetadata = {\r\n "executable": sys.executable or None,\r\n "major": sys.version_info.major,\r\n "minor": sys.version_info.minor,\r\n "abiflags": sysconfig.get_config_var("ABIFLAGS"),\r\n "interpreter": platform.python_implementation().lower(),\r\n "ext_suffix": ext_suffix,\r\n "abi_tag": (sysconfig.get_config_var("SOABI") or "-").split("-")[1] or None,\r\n "platform": sysconfig.get_platform(),\r\n # This one isn't technically necessary, but still very useful for sanity checks\r\n "system": platform.system().lower(),\r\n # This one is for generating a config file for pyo3\r\n "pointer_width": struct.calcsize("P") * 8,\r\n}\r\n\r\nprint(json.dumps(metadata))\r\n"

Ok(Output { status: ExitStatus(ExitStatus(101)), stdout: "", stderr: "Unable to create process using 'C:\hostedtoolcache\windows\Python\3.11.0\x64\py" -c "import json\r\r\nimport platform\r\r\nimport sys\r\r\nimport sysconfig\r\r\nimport struct\r\r\n\r\r\nif platform.python_implementation() == \"PyPy\":\r\r\n # Workaround for PyPy 3.6 on windows:\r\r\n # - sysconfig.get_config_var(\"EXT_SUFFIX\") differs to importlib until\r\r\n # Python 3.8\r\r\n # - PyPy does not load the plain \".pyd\" suffix because it expects that's\r\r\n # for a CPython extension module\r\r\n #\r\r\n # This workaround can probably be removed once PyPy for Python 3.8 is the\r\r\n # main PyPy version.\r\r\n import importlib.machinery\r\r\n\r\r\n ext_suffix = importlib.machinery.EXTENSION_SUFFIXES[0]\r\r\nelse:\r\r\n ext_suffix = sysconfig.get_config_var(\"EXT_SUFFIX\")\r\r\n\r\r\nmetadata = {\r\r\n \"executable\": sys.executable or None,\r\r\n \"major\": sys.version_info.major,\r\r\n \"minor\": sys.version_info.minor,\r\r\n \"abiflags\": sysconfig.get_config_var(\"ABIFLAGS\"),\r\r\n \"interpreter\": platform.python_implementation().lower(),\r\r\n \"ext_s\r\n" })

Odd, https://github.com/PyO3/maturin/actions/runs/3342317805/jobs/5534561600

[messense]

"cmd" "/C" "py" "-3.11-64" "-c" "import json\r\nimport platform\r\nimport sys\r\nimport sysconfig\r\nimport struct\r\n\r\nif platform.python_implementation() == "PyPy":\r\n # Workaround for PyPy 3.6 on windows:\r\n # - sysconfig.get_config_var("EXT_SUFFIX") differs to importlib until\r\n # Python 3.8\r\n # - PyPy does not load the plain ".pyd" suffix because it expects that's\r\n # for a CPython extension module\r\n #\r\n # This workaround can probably be removed once PyPy for Python 3.8 is the\r\n # main PyPy version.\r\n import importlib.machinery\r\n\r\n ext_suffix = importlib.machinery.EXTENSION_SUFFIXES[0]\r\nelse:\r\n ext_suffix = sysconfig.get_config_var("EXT_SUFFIX")\r\n\r\nmetadata = {\r\n "executable": sys.executable or None,\r\n "major": sys.version_info.major,\r\n "minor": sys.version_info.minor,\r\n "abiflags": sysconfig.get_config_var("ABIFLAGS"),\r\n "interpreter": platform.python_implementation().lower(),\r\n "ext_suffix": ext_suffix,\r\n "abi_tag": (sysconfig.get_config_var("SOABI") or "-").split("-")[1] or None,\r\n "platform": sysconfig.get_platform(),\r\n # This one isn't technically necessary, but still very useful for sanity checks\r\n "system": platform.system().lower(),\r\n # This one is for generating a config file for pyo3\r\n "pointer_width": struct.calcsize("P") * 8,\r\n}\r\n\r\nprint(json.dumps(metadata))\r\n"

Ok(Output { status: ExitStatus(ExitStatus(0)), stdout: "", stderr: "" })

Exit success with cmd.exe /C, but no output captured, so confusing.

[messense]

Interesting, install Python 3.11 will change py --list-paths output from

 -3.10-64       C:\\hostedtoolcache\\windows\\Python\\3.10.0\\x64\\python.exe
 -3.10-32       C:\\hostedtoolcache\\windows\\Python\\3.10.8\\x86\\python.exe
 -3.9-64        C:\\hostedtoolcache\\windows\\Python\\3.9.13\\x64\\python.exe
 -3.9-32        C:\\hostedtoolcache\\windows\\Python\\3.9.13\\x86\\python.exe
 -3.8-64        C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\python.exe
 -3.8-32        C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x86\\python.exe
 -3.7-64        C:\\hostedtoolcache\\windows\\Python\\3.7.9\\x64\\python.exe
 -3.7-32        C:\\hostedtoolcache\\windows\\Python\\3.7.9\\x86\\python.exe

to

 -V:3.11 *        C:\\hostedtoolcache\\windows\\Python\\3.11.0\\x64\\python.exe
 -V:3.10          C:\\hostedtoolcache\\windows\\Python\\3.10.0\\x64\\python.exe
 -V:3.10-32       C:\\hostedtoolcache\\windows\\Python\\3.10.8\\x86\\python.exe
 -V:3.9           C:\\hostedtoolcache\\windows\\Python\\3.9.13\\x64\\python.exe
 -V:3.9-32        C:\\hostedtoolcache\\windows\\Python\\3.9.13\\x86\\python.exe
 -V:3.8           C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\python.exe
 -V:3.8-32        C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x86\\python.exe
 -V:3.7           C:\\hostedtoolcache\\windows\\Python\\3.7.9\\x64\\python.exe
 -V:3.7-32        C:\\hostedtoolcache\\windows\\Python\\3.7.9\\x86\\python.exe
 -V:ContinuumAnalytics/Anaconda39-64 C:\\Miniconda\\python.exe

https://docs.python.org/3/whatsnew/3.11.html#windows-py-exe-launcher-improvements

[messense]

Please try again, I think it's fixed in #1222 and released in v0.13.7.

[ashb]

This worked for me, yes.

Working build here https://github.com/ashb/windows-fonts/actions/runs/3352900266/jobs/5555349536

(Had to have actions/setup-python run to install 3.11 as expected)

[messense]

Great! Feel free to reopen if it still occurs.