Meson errors with Python 3.10

[lazka]

It randomly fails when calling into meson scripts:

1. mesa:

FAILED: src/util/u_unfilled_gen.c
"D:/M/msys64/mingw64/bin/meson" "--internal" "exe" "--capture" "src/util/u_unfilled_gen.c" "--" "D:\M\msys64\mingw64\bin/python3.EXE" "../../src/util/indices/u_unfilled_gen.py"

FAILED: src/gallium/targets/osmesa/osmesa.dll.p/osmesa.dll.symbols
"D:/M/msys64/mingw64/bin/meson" "--internal" "symbolextractor" "C:/_/mingw-w64-mesa/src/mesa-22.1.2/build/windows-x86_64" src/gallium/targets/osmesa/osmesa.dll "src/gallium/targets/osmesa/osmesa.dll.a" src/gallium/targets/osmesa/osmesa.dll.p/osmesa.dll.symbols

2. gobject-introspection

FAILED: girepository/libgirepository-1.0-1.dll.p/libgirepository-1.0-1.dll.symbols
"D:/a/msys64/clang64/bin/meson" "--internal" "symbolextractor" "C:/M/mingw-w64-gobject-introspection/src/build-x86_64-w64-mingw32" girepository/libgirepository-1.0-1.dll "girepository/libgirepository-1.0.dll.a" girepository/libgirepository-1.0-1.dll.p/libgirepository-1.0-1.dll.symbols

There was one more packages during the Python rebuild that failed that way (with "symbolextractor" I think), but I didn't write it down sadly.

[eli-schwartz]

It's completely bizarre that the symbolextractor fails without emitting an error message of any sort. It's reminiscent of pypa/pip#10875 (was a bug in the launcher *.exe files pip/distlib uses on Windows, reproducible in CI but not running interactively).

[lazka]

thanks, good point. The last meson version before was built with the same setuptools version though, so there shouldn't be any difference regarding the launcher.

I can try porting the meson package from setuptools to builder/installer so it gets a newer launcher. Maybe that makes a difference, or at least gives some more insight.

[naveen521kk]

(was a bug in the launcher *.exe files pip/distlib uses on Windows, reproducible in CI but not running interactively).

Hmm, meson was built with setuptools launcher which shouldn't have this issue afaik.

[lazka]

same error with the python-installer launcher (#11865)

[eli-schwartz]

For the record, last I saw the latest version of distlib was the one with the wonky binaries, and pip specifically worked around this by downgrading the bundled distlib and rereleasing.

I'm not sure what other installers do on Windows specifically. They may bundle different versions of distlib.

[lazka]

I fetched the meson build dir from a failed CI build, but there is nothing helpful in there.

Just to point out why it might be a different issue: This happens randomly, it sometimes works, sometimes doesn't in CI. It started with the Update to Python 3.10 over the last week, could be a coincidence, but mesa built fine 2 weeks ago.

I'll (hackily) try forcing a sys.stderr for symbolextractor in CI and see if that helps next.

[lazka]

After some more printf debugging, the internal meson command runs through fine, so something after that fails somehow.. I'm giving up for now..

[kcgen]

@lazka , after seeing you mention the issue was intermittent, I've placed our Meson builds in a diaper script to re-run on failure.

Data points:

• direct CI launch: run: meson -C build --> 60 jobs, all failing in CI, not one passed
• direct CI launch: run: ninja -C build --> 18 jobs failing in CI, not one passed
• script-wrapped launch: run: scripts/retry_command.sh 2 meson compile -C build --> 18 back-to-back good runs, no retries needed.

What's interesting is that the retry logic is never needed, making me suspect that running inside a shell wrapper is altering the parent-level runtime space/behavior in a way that Python likes (perhaps stacksize or other low-level stuff inside MSYS2) versus the "direct-launch" that's giving it troubles.

Perhaps to reproduce this locally, you'd need to launch the CI YAML run: command using whatever mechanism GitHub is using. Perhaps their CI code is Go(?) or Python(?) launching command as a system-call (and probably wrapped in some security-hardened WSL Docker instance).

[lazka]

I've forward ported an old ninja PR (ninja-build/ninja#1805) to show the exit status and it's STATUS_ACCESS_VIOLATION. So that could either be the launcher crashing or python itself. I'd guess python itself (?).

[shermp]

Perhaps to reproduce this locally, you'd need to launch the CI YAML run: command using whatever mechanism GitHub is using. Perhaps their CI code is Go(?) or Python(?) launching command as a system-call (and probably wrapped in some security-hardened WSL Docker instance).

They may be using either .NET or NodeJS. That's what their self hosted runner and actions are using respectively at least.

[naveen521kk]

So that could either be the launcher crashing or python itself. I'd guess python itself (?).

Oh wow, interesting.

[kcgen]

A user running GDAL was hit with STATUS_ACCESS_VIOLATION on Windows, and configured the stack and environment settings prior to launch:

if __name__ == "__main__":
    if sys.platform[:3] == "win":
        sys.setrecursionlimit(10000000)
        threading.stack_size(200000000)
        thread = threading.Thread(target=driver)
        thread.start()

    # launch ...

https://stackoverflow.com/questions/41000945/why-do-i-get-a-status-access-violation-when-i-run-this-python-script-on-windows

[lb90]

ELI5? Some Meson internal commands fail randomly and it's unclear whether the problem is in Ninja, Meson or Python 3.10?

[lb90]

FWIF, the error in #11915 can be reproduced reliably, meaning that it's not random, it just happens anytime I run ninja. It may be useful to investigate this issue further

EDIT: maybe it's not related? The error message is: failed to load doclet 'D:/a/msys64/mingw64/lib/valadoc-0.56\doclets\devhelp'

[Biswa96]

Yeah, the vala error is different topic. I can try to create a PR today to workaround that issue.

[lb90]

Ah, thanks! That would be great. I'm not well-versed in valadoc 🙂

[lb90]

Not sure, but maybe those links provide infos about this bug:

• pip3 install fails on Windows with exit code 3221225477 actions/runner-images#5027
• Add fix for Windows caching of pip actions/setup-python#332

[lazka]

So that could either be the launcher crashing or python itself. I'd guess python itself (?).

it's python itself..

edit: the faulthandler doesn't print any stacktraces, despite python crashing..

edit: I'm out of ideas now..

[anarazel]

@lazka:

Enabling JIT debugging (6ff1a1f) makes the error go away. But at this point I'm just happy to have a workaround for CI.

I suspect that might be because setting MSYS=winjitdebug causes the error-mode for children to be set to 0. Normally that's set to SEM_FAILCRITICALERRORS | SEM_NOOPENFILEERRORBOX (in normal windows at least). Not having SEM_FAILCRITICALERRORS will cause a bunch of behavioural differences, which could easily prevent the crash from occurring.

myadmin@win10-nojoin UCRT64 ~
$ python -c "import msvcrt; print(hex(msvcrt.GetErrorMode()))"
0x3

$ export MSYS=winjitdebug
$ exec bash

myadmin@win10-nojoin UCRT64 ~
$ python -c "import msvcrt; print(hex(msvcrt.GetErrorMode()))"
0x0

[anarazel]

Turns out it's actually the presence of SEM_NOOPENFILEERRORBOX - I can, very very occasionally, reproduce crashes after os._exit() and before process exit. So this looks very likely to be a microsoft C runtime (or kernel) bug. In my case no msys environment is involved, fwiw.

[lb90]

Could it be a DLL doing weird things from their DLL_PROCESS_DETACH then? (or the same from a TLS callback)

Actually in DLL_PROCESS_DETACH a DLL should check whether the process is exiting (lpvReserved != nullptr), and in that case do nothing at all. That's because the program state is undefined at that time:

• https://devblogs.microsoft.com/oldnewthing/20150213-00/?p=44683
• https://devblogs.microsoft.com/oldnewthing/20070503-00/?p=27003

Also documented on MSDN:

• https://docs.microsoft.com/en-us/windows/win32/dlls/dllmain
• https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-best-practices#best-practices-for-synchronization

Unfortunately many DLLs don't check lpvReserved

[anarazel]

Could it be a DLL doing weird things from their DLL_PROCESS_DETACH then? (or the same from a TLS callback)

A quick grep through the python sources doesn't show a problematic DLL_PROCESS_DETACH callback that could be involved here. None of them do anything in the DLL_PROCESS_DETACH case. I didn't see any tls callbacks at all, but might be missing something (not a windows person).

Given the crash only happens with SEM_NOGPFAULTERRORBOX set it seems more likely to be a bug in the CRT or kernel to me, since that being set should lead to less application involvement.

Edit: I copy-pastoed SEM_NOOPENFILEERRORBOX instead of SEM_NOGPFAULTERRORBOX in an earlier version

[lb90]

I can, very very occasionally, reproduce crashes after os._exit() and before process exit

Can you reproduce locally? If so, may I ask how are you invoking python?

[anarazel]

Can you reproduce locally? If so, may I ask how are you invoking python?

Unfortunately only in CI, windows 2019 in my case. The method of invoking python doesn't seem to matter, it happens even when invoking via path/to/python.exe path/to/script.py

[jeremyd2019]

Thought I'd try to correct the record a bit here since this got referenced again. error mode 0x3 is actually SEM_FAILCRITICALERRORS|SEM_NOGPFAULTERRORBOX. Both relate to not showing a message box if an error occurs. Unfortunately, MS hooked just-in-time debugging (and WER) to the "GP fault error box" path, so if you disable that you cannot get those either. This is why the option I added is called winjitdebug (because that's what I was trying to enable). Also, mode 0x0 should be the OS default (though the default is also to inherit from the parent process, so you can't necessarily assume what they will be).

https://learn.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-seterrormode#parameters

[anarazel]

Also, mode 0x0 should be the OS default (though the default is also to inherit from the parent process, so you can't necessarily assume what they will

Oh, indeed. I saw 0x8001 because I was using the 'terminal app' to start cmd, rather than just cmd.exe directly. Extremely unhelpful by terminal to start the "sub terminals" with a different error mode than normal...

Thought I'd try to correct the record a bit here since this got referenced again. error mode 0x3 is actually SEM_FAILCRITICALERRORS|SEM_NOGPFAULTERRORBOX.

The reference to SEM_FAILCRITICALERRORS | SEM_NOOPENFILEERRORBOX was not about the 0x3, but what I saw in the windows terminal. 0x3 / SEM_FAILCRITICALERRORS|SEM_NOGPFAULTERRORBOX is what some msys code sets when MSYS=winjitdebug isn't set.

[lb90]

Regarding this, how could one go about setting up Github action that can reproduce the error?

[vintagepc]

It seems to happen fairly consistently with the MSYS build in my project repo linked in the referenced issue from 6h ago as it does a fairly involved Meson/Python build.

vintagepc/MINI404#116 has been giving me grief most recently. You're more than welcome to fork it and hack up the workflow to dig in.

[vintagepc]

I'll add in that I just tried force-downgrading to the mingw64 python 3.9 package and it seems to have made the problem go away - will rerun the workflow a few times to confirm.

[lazka]

We've made some progress finding a potential cause here: #17415

With a bit of luck the update to 3.11 will fix this.

[lazka]

I'm closing this in favor of #17415

[lazka]

One last comment here: We've updated to Python 3.11 yesterday, so this should no longer be an issue.