Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build failed because docker is too old or is not working properly: json.decoder.JSONDecodeError #2074

Closed
mcleantom opened this issue Nov 6, 2024 · 16 comments · Fixed by #2076

Comments

@mcleantom
Copy link

Description

I am running cibuildwheel version 2.21.3 and I am trying to see if my project will compile for linux wheels, from windows, using the command:

cibuildwheel --platform linux --debug-traceback

Which fails with the error

Build failed because docker is too old or is not working properly.

Because of a json decode error:


     _ _       _ _   _       _           _
 ___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
|  _| | . | | | | | . | | | |   | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|

cibuildwheel version 2.21.3

Build options:
  platform: linux
  allow_empty: False
  architectures: x86_64
  build_selector:
    build_config: cp*
    skip_config: cp*-musllinux* pp*
    requires_python: >=3.11
    prerelease_pythons: False
    free_threaded_support: False
  output_dir: C:\Users\tom.mclean\src\Nemo\wheelhouse
  package_dir: C:\Users\tom.mclean\src\Nemo
  test_selector:
    skip_config:
  before_all:
    rm -rf vcpkg
    rm -rf vcpkg_installed
    yum -y install curl zip unzip tar
    git clone https://github.com/microsoft/vcpkg.git
    ./vcpkg/bootstrap-vcpkg.sh
    ./vcpkg/vcpkg install
  before_build:
  before_test:
  build_frontend: None
  build_verbosity: 0
  config_settings:
  container_engine: docker
  dependency_constraints: pinned
  environment:
  manylinux_images:
    x86_64: quay.io/pypa/manylinux_2_28_x86_64:2024.10.07-1
    i686: manylinux_2_28
    pypy_x86_64: quay.io/pypa/manylinux_2_28_x86_64:2024.10.07-1
    aarch64: quay.io/pypa/manylinux_2_28_aarch64:2024.10.07-1
    ppc64le: quay.io/pypa/manylinux_2_28_ppc64le:2024.10.07-1
    s390x: quay.io/pypa/manylinux_2_28_s390x:2024.10.07-1
    pypy_aarch64: quay.io/pypa/manylinux_2_28_aarch64:2024.10.07-1
    pypy_i686: manylinux_2_28
  musllinux_images:
    x86_64: quay.io/pypa/musllinux_1_2_x86_64:2024.10.07-1
    i686: quay.io/pypa/musllinux_1_2_i686:2024.10.07-1
    aarch64: quay.io/pypa/musllinux_1_2_aarch64:2024.10.07-1
    ppc64le: quay.io/pypa/musllinux_1_2_ppc64le:2024.10.07-1
    s390x: quay.io/pypa/musllinux_1_2_s390x:2024.10.07-1
    armv7l: quay.io/pypa/musllinux_1_2_armv7l:2024.10.07-1
  repair_command: auditwheel repair -w {dest_dir} {wheel}
  test_command:
  test_extras:
  test_requires:

Cache folder: C:\Users\tom.mclean\AppData\Local\pypa\cibuildwheel\Cache

Here we go!

Starting container image quay.io/pypa/manylinux_2_28_x86_64:2024.10.07-1...

info: This container will host the build for cp311-manylinux_x86_64, cp312-manylinux_x86_64, cp313-manylinux_x86_64...
+ docker version -f '{{json .}}'

                                                                       ✕ 0.14s
Error: Build failed because docker is too old or is not working properly.

Traceback (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 105, in _check_engine_version
    version_info = json.loads(version_string.strip())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 49, in main
    main_inner(global_options)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 184, in main_inner
    build_in_directory(args)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 351, in build_in_directory
    platform_module.build(options, tmp_path)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\linux.py", line 460, in build
    with OCIContainer(
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 230, in __enter__
    _check_engine_version(self.engine)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 145, in _check_engine_version
    raise OCIEngineTooOldError(msg) from e
cibuildwheel.errors.OCIEngineTooOldError: Build failed because docker is too old or is not working properly.

The output of my docker version command is:

{
   "Client":{
      "CloudIntegration":"v1.0.35+desktop.10",
      "Version":"25.0.2",
      "ApiVersion":"1.44",
      "DefaultAPIVersion":"1.44",
      "GitCommit":"29cf629",
      "GoVersion":"go1.21.6",
      "Os":"windows",
      "Arch":"amd64",
      "BuildTime":"Thu Feb  1 00:24:09 2024",
      "Context":"default"
   },
   "Server":{
      "Platform":{
         "Name":"Docker Desktop 4.27.1 (136059)"
      },
      "Components":[
         {
            "Name":"Engine",
            "Version":"25.0.2",
            "Details":{
               "ApiVersion":"1.44",
               "Arch":"amd64",
               "BuildTime":"Thu Feb  1 00:23:17 2024",
               "Experimental":"false",
               "GitCommit":"fce6e0c",
               "GoVersion":"go1.21.6",
               "KernelVersion":"5.15.133.1-microsoft-standard-WSL2",
               "MinAPIVersion":"1.24",
               "Os":"linux"
            }
         },
         {
            "Name":"containerd",
            "Version":"1.6.28",
            "Details":{
               "GitCommit":"ae07eda36dd25f8a1b98dfbf587313b99c0190bb"
            }
         },
         {
            "Name":"runc",
            "Version":"1.1.12",
            "Details":{
               "GitCommit":"v1.1.12-0-g51d5e94"
            }
         },
         {
            "Name":"docker-init",
            "Version":"0.19.0",
            "Details":{
               "GitCommit":"de40ad0"
            }
         }
      ],
      "Version":"25.0.2",
      "ApiVersion":"1.44",
      "MinAPIVersion":"1.24",
      "GitCommit":"fce6e0c",
      "GoVersion":"go1.21.6",
      "Os":"linux",
      "Arch":"amd64",
      "KernelVersion":"5.15.133.1-microsoft-standard-WSL2",
      "BuildTime":"2024-02-01T00:23:17.000000000+00:00"
   }
}

So I am not sure why it is failing.

Build log

No response

CI config

No response

@mcleantom mcleantom changed the title Build failed because docker is too old or is not working properly. Build failed because docker is too old or is not working properly: json.decoder.JSONDecodeError Nov 6, 2024
@mcleantom
Copy link
Author

I could run the command from inside WSL, but I thought this command had previously worked when ran from windows

@mcleantom
Copy link
Author

Actually it should work from windows as per #1782 so not sure whats happening

@mcleantom mcleantom reopened this Nov 6, 2024
@Czaki
Copy link
Contributor

Czaki commented Nov 6, 2024

@mcleantom could you run the cibuildwheel using traceback-with-variables

$ traceback-with-variables cibuildwheel --platform linux --debug-traceback

It should add additional context to traceback to allow understand the problem.

@mcleantom
Copy link
Author

@Czaki Hey, here is the output of that:

Traceback with variables (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\pathlib.py", line 1059, in read_text
    return f.read()
      self = WindowsPath('C:/Users/tom.mclean/AppData/Local/miniconda3/envs/nemo/Scripts/cibuildwheel.EXE')
      encoding = 'locale'
      errors = None
      f = <_io.TextIOWrapper name='C:\\Users\\tom.mclean\\AppData\\Local\\miniconda3\\envs\\nemo\\Scripts\\cibuildwheel.EXE' mode='r' encoding='cp1252'>
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
      self = <encodings.cp1252.IncrementalDecoder object at 0x0000029F4C652950>(errors='strict')
      input = b'MZ\x90\x00\x03\x00\x00\x00\x04\x00\x00\x00\xff\xff\x00\x00\xb8\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf8\x00\x00\x00\x0e\x1f\xba\x0e\x00\xb4\t\xcd!\xb8\x01L\xcd!This program cannot be run in DOS mode.\r\r\n$\x00\x00\x00\x00\x00\x00\x00\xa3\xfa`v\xe7\x9b\x0e%\xe7\x9b\x0e%\xe7\x9b\x0e%t\xd5\x96%\xe6\x9b\x0e%\xfc\x06\xa5%\xc6\x9b\x0e%\xfc\x06\xa4%\x91\x9b\x0e%\xfc\x06\x90%\xee\x9b\x0e%\xee\xe3\x9d%\xe2\x9b\x0e%\xe7\x9b\x0f%\xb8\x9b\x0e%\xfc\x06\xa0%\xe6\x9b\x0e%\xfc\x06\x94%\xe6\x9b\x0e%\xfc\x06\x93%\xe6\x9b\x0e%Rich\xe7\x9b\x0e%\x00\x00\x00\x00\x00\x00\x00\x00\x0...\', \'\', sys.argv[0])\n    sys.exit(main())\nPK\x01\x02\x14\x00\x14\x00\x00\x00\x00\x00\xf5zfY\x1aa\x8bP\xc9\x00\x00\x00\xc9\x00\x00\x00\x0b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\x01\x00\x00\x00\x00__main__.pyPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x009\x00\x00\x00\xf2\x00\x00\x00\x00\x00'
      final = True
builtins.UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 2: character maps to <undefined>

@Czaki
Copy link
Contributor

Czaki commented Nov 7, 2024

Hm. It looks like it invoke exe instead of a python file. What with:

$ traceback-with-variables cibuildwheel.__main__ --platform linux --debug-traceback

@mcleantom
Copy link
Author

That output is:

(nemo) C:\Users\tom.mclean\src\Nemo>traceback-with-variables cibuildwheel.__main__ --platform linux --debug-traceback

     _ _       _ _   _       _           _
 ___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
|  _| | . | | | | | . | | | |   | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|

cibuildwheel version 2.21.2

Build options:
  platform: linux
  allow_empty: False
  architectures: x86_64
  build_selector:
    build_config: cp*
    skip_config: cp*-musllinux* pp*
    requires_python: >=3.11
    prerelease_pythons: False
    free_threaded_support: False
  output_dir: C:\Users\tom.mclean\src\Nemo\wheelhouse
  package_dir: C:\Users\tom.mclean\src\Nemo
  test_selector:
    skip_config:
  before_all:
    rm -rf vcpkg
    rm -rf vcpkg_installed
    yum -y install curl zip unzip tar
    git clone https://github.com/microsoft/vcpkg.git
    ./vcpkg/bootstrap-vcpkg.sh
    ./vcpkg/vcpkg install
  before_build:
  before_test:
  build_frontend: None
  build_verbosity: 0
  config_settings:
  container_engine: docker
  dependency_constraints: pinned
  environment:
  manylinux_images:
    x86_64: quay.io/pypa/manylinux_2_28_x86_64:2024.10.01-1
    i686: manylinux_2_28
    pypy_x86_64: quay.io/pypa/manylinux_2_28_x86_64:2024.10.01-1
    aarch64: quay.io/pypa/manylinux_2_28_aarch64:2024.10.01-1
    ppc64le: quay.io/pypa/manylinux_2_28_ppc64le:2024.10.01-1
    s390x: quay.io/pypa/manylinux_2_28_s390x:2024.10.01-1
    pypy_aarch64: quay.io/pypa/manylinux_2_28_aarch64:2024.10.01-1
    pypy_i686: manylinux_2_28
  musllinux_images:
    x86_64: quay.io/pypa/musllinux_1_2_x86_64:2024.10.01-1
    i686: quay.io/pypa/musllinux_1_2_i686:2024.10.01-1
    aarch64: quay.io/pypa/musllinux_1_2_aarch64:2024.10.01-1
    ppc64le: quay.io/pypa/musllinux_1_2_ppc64le:2024.10.01-1
    s390x: quay.io/pypa/musllinux_1_2_s390x:2024.10.01-1
    armv7l: quay.io/pypa/musllinux_1_2_armv7l:2024.10.01-1
  repair_command: auditwheel repair -w {dest_dir} {wheel}
  test_command:
  test_extras:
  test_requires:

Cache folder: C:\Users\tom.mclean\AppData\Local\pypa\cibuildwheel\Cache

Here we go!

Starting container image quay.io/pypa/manylinux_2_28_x86_64:2024.10.01-1...

info: This container will host the build for cp311-manylinux_x86_64, cp312-manylinux_x86_64, cp313-manylinux_x86_64...
+ docker version -f '{{json .}}'

                                                                       ✕ 0.19s
Error: Build failed because docker is too old or is not working properly.

Traceback (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 105, in _check_engine_version
    version_info = json.loads(version_string.strip())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 49, in main
    main_inner(global_options)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 184, in main_inner
    build_in_directory(args)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 351, in build_in_directory
    platform_module.build(options, tmp_path)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\linux.py", line 460, in build
    with OCIContainer(
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 230, in __enter__
    _check_engine_version(self.engine)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 145, in _check_engine_version
    raise OCIEngineTooOldError(msg) from e
cibuildwheel.errors.OCIEngineTooOldError: Build failed because docker is too old or is not working properly.

@Czaki
Copy link
Contributor

Czaki commented Nov 7, 2024

Ok. So this module do not overwrite traceback.print_exc(file=sys.stderr) function.

traceback.print_exc(file=sys.stderr)

I need to think about other debug options..

@mcleantom
Copy link
Author

mcleantom commented Nov 7, 2024

I cloned the repo and ran it within pycharm debugger so that it picks up the cibuildwheel that I cloned and it ran fine. Not sure why its not working when I just run from the terminal

Even if I use pip install . inside the cloned repo, then run the terminal command it doesnt work.

I modified the code to print the version string variable and it says:

+ docker version -f '{{json .}}'
failed to get console mode for stdout: The handle is invalid.
{"Client":{"CloudIntegration":"v1.0.35+desktop.10","Version":"25.0.2","ApiVersion":"1.44","DefaultAPIVersion":"1.44","GitCommit":"29cf629","GoVersion":"go1.21.6","Os":"windows","Ar
ch":"amd64","BuildTime":"Thu Feb  1 00:24:09 2024","Context":"default"},"Server":{"Platform":{"Name":"Docker Desktop 4.27.1 (136059)"},"Components":[{"Name":"Engine","Version":"25.
0.2","Details":{"ApiVersion":"1.44","Arch":"amd64","BuildTime":"Thu Feb  1 00:23:17 2024","Experimental":"false","GitCommit":"fce6e0c","GoVersion":"go1.21.6","KernelVersion":"5.15.
133.1-microsoft-standard-WSL2","MinAPIVersion":"1.24","Os":"linux"}},{"Name":"containerd","Version":"1.6.28","Details":{"GitCommit":"ae07eda36dd25f8a1b98dfbf587313b99c0190bb"}},{"N
ame":"runc","Version":"1.1.12","Details":{"GitCommit":"v1.1.12-0-g51d5e94"}},{"Name":"docker-init","Version":"0.19.0","Details":{"GitCommit":"de40ad0"}}],"Version":"25.0.2","ApiVer
sion":"1.44","MinAPIVersion":"1.24","GitCommit":"fce6e0c","GoVersion":"go1.21.6","Os":"linux","Arch":"amd64","KernelVersion":"5.15.133.1-microsoft-standard-WSL2","BuildTime":"2024-02-01T00:23:17.000000000+00:00"}}

@mcleantom
Copy link
Author

mcleantom commented Nov 7, 2024

Seems to be related to docker/for-win#13891

Adding this:

version_string = version_string.strip("failed to get console mode for stdout: The handle is invalid.\n")

"fixes" it, but I guess I should downgrade/upgrade docker

@Czaki
Copy link
Contributor

Czaki commented Nov 7, 2024

Hm.
Reading pointed issue suggest that failed to get console mode for stdout: The handle is invalid is printed to stderr.

Maybe we should prepare stderr handler that may handle it so it will not be rediretedet to stdout?

@henryiii?

@mayeut
Copy link
Member

mayeut commented Nov 9, 2024

@mcleantom,

can you try something like the following. If it's the same as on macOS, I suspect the one with shell=True to fail:

python3.11 -c "import json; import subprocess; print(json.loads(subprocess.run(['docker', 'version', '-f', '{{json .}}'], check=True, shell=False, universal_newlines=True, stdout=subprocess.PIPE).stdout))"
python3.11 -c "import json; import subprocess; print(json.loads(subprocess.run(['docker', 'version', '-f', '{{json .}}'], check=True, shell=True, universal_newlines=True, stdout=subprocess.PIPE).stdout))"

If the one with shell=False also fails, can you try it with the full path to the docker executable please ?

@mcleantom
Copy link
Author

Hey sure, here are those outputs:

(Nemo) PS C:\Users\tom.mclean\src\Nemo> python3.11 -c "import json; import subprocess; print(json.loads(subprocess.run(['docker', 'version', '-f', '{{json .}}'], check=True, shell=False, universal_newlines=True, stdout=subprocess.PIPE).stdout))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

And the second one:

(Nemo) PS C:\Users\tom.mclean\src\Nemo> python3.11 -c "import json; import subprocess; print(json.loads(subprocess.run(['docker', 'version', '-f', '{{json .}}'], check=True, shell=True, universal_newlines=True, stdout=subprocess.PIPE).stdout))"     
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

And with the full path to docker:

(Nemo) PS C:\Users\tom.mclean\src\Nemo> python3.11 -c "import json; import subprocess; print(json.loads(subprocess.run([r'C:\Program Files\Docker\Docker\resources\bin\docker.exe', 'version', '-f', '{{json .}}'], check=True, shell=False, universal_newlines=True, stdout=subprocess.PIPE).stdout))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

For reference, it is still outputting failed to get console mode for stdout:

(Nemo) PS C:\Users\tom.mclean\src\Nemo> python3.11 -c "import json; import subprocess; print(subprocess.run([r'C:\Program Files\Docker\Docker\resources\bin\docker.exe', 'version', '-f', '{{json .}}'], check=True, shell=False, universal_newlines=True, stdout=subprocess.PIPE).stdout)"            
failed to get console mode for stdout: The handle is invalid.
{"Client":{"CloudIntegration":"v1.0.35+desktop.10","Version":"25.0.2","ApiVersion":"1.44","DefaultAPIVersion":"1.44","GitCommit":"29cf629","GoVersion":"go1.21.6","Os":"windows","Ar
ch":"amd64","BuildTime":"Thu Feb  1 00:24:09 2024","Context":"default"},"Server":{"Platform":{"Name":"Docker Desktop 4.27.1 (136059)"},"Components":[{"Name":"Engine","Version":"25.
0.2","Details":{"ApiVersion":"1.44","Arch":"amd64","BuildTime":"Thu Feb  1 00:23:17 2024","Experimental":"false","GitCommit":"fce6e0c","GoVersion":"go1.21.6","KernelVersion":"5.15.
133.1-microsoft-standard-WSL2","MinAPIVersion":"1.24","Os":"linux"}},{"Name":"containerd","Version":"1.6.28","Details":{"GitCommit":"ae07eda36dd25f8a1b98dfbf587313b99c0190bb"}},{"N
ame":"runc","Version":"1.1.12","Details":{"GitCommit":"v1.1.12-0-g51d5e94"}},{"Name":"docker-init","Version":"0.19.0","Details":{"GitCommit":"de40ad0"}}],"Version":"25.0.2","ApiVer
sion":"1.44","MinAPIVersion":"1.24","GitCommit":"fce6e0c","GoVersion":"go1.21.6","Os":"linux","Arch":"amd64","KernelVersion":"5.15.133.1-microsoft-standard-WSL2","BuildTime":"2024-02-01T00:23:17.000000000+00:00"}}

@mcleantom
Copy link
Author

mcleantom commented Nov 9, 2024

If I also add stderr=subprocess.PIPE it doesnt throw an error:

(Nemo) PS C:\Users\tom.mclean\src\Nemo> python3.11 -c "import json; import subprocess; print(json.loads(subprocess.run([r'C:\Program Files\Docker\Docker\resources\bin\docker.exe', 'version', '-f', '{{json .}}'], check=True, shell=False, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stdout))"
{'Client': {'CloudIntegration': 'v1.0.35+desktop.10', 'Version': '25.0.2', 'ApiVersion': '1.44', 'DefaultAPIVersion': '1.44', 'GitCommit': '29cf629', 'GoVersion': 'go1.21.6', 'Os':
 'windows', 'Arch': 'amd64', 'BuildTime': 'Thu Feb  1 00:24:09 2024', 'Context': 'default'}, 'Server': {'Platform': {'Name': 'Docker Desktop 4.27.1 (136059)'}, 'Components': [{'Nam
e': 'Engine', 'Version': '25.0.2', 'Details': {'ApiVersion': '1.44', 'Arch': 'amd64', 'BuildTime': 'Thu Feb  1 00:23:17 2024', 'Experimental': 'false', 'GitCommit': 'fce6e0c', 'GoV
ersion': 'go1.21.6', 'KernelVersion': '5.15.133.1-microsoft-standard-WSL2', 'MinAPIVersion': '1.24', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.6.28', 'Details': {'GitCo
mmit': 'ae07eda36dd25f8a1b98dfbf587313b99c0190bb'}}, {'Name': 'runc', 'Version': '1.1.12', 'Details': {'GitCommit': 'v1.1.12-0-g51d5e94'}}, {'Name': 'docker-init', 'Version': '0.19
.0', 'Details': {'GitCommit': 'de40ad0'}}], 'Version': '25.0.2', 'ApiVersion': '1.44', 'MinAPIVersion': '1.24', 'GitCommit': 'fce6e0c', 'GoVersion': 'go1.21.6', 'Os': 'linux', 'Arch': 'amd64', 'KernelVersion': '5.15.133.1-microsoft-standard-WSL2', 'BuildTime': '2024-02-01T00:23:17.000000000+00:00'}}

@mayeut
Copy link
Member

mayeut commented Nov 9, 2024

does the following works ? (or as a simple test, modifying your previous working test to use shell=True)

diff --git a/cibuildwheel/util.py b/cibuildwheel/util.py
index bfbe50c7..c72a3ddf 100644
--- a/cibuildwheel/util.py
+++ b/cibuildwheel/util.py
@@ -131,9 +131,16 @@ def call(
     if capture_stdout:
         kwargs["universal_newlines"] = True
         kwargs["stdout"] = subprocess.PIPE
-    result = subprocess.run(args_, check=True, shell=IS_WIN, env=env, cwd=cwd, **kwargs)
+        kwargs["stderr"] = subprocess.PIPE
+    try:
+        result = subprocess.run(args_, check=True, shell=IS_WIN, env=env, cwd=cwd, **kwargs)
+    except subprocess.CalledProcessError as e:
+        if capture_stdout:
+            sys.stderr.write(e.stderr)
+        raise
     if not capture_stdout:
         return None
+    sys.stderr.write(result.stderr)
     return typing.cast(str, result.stdout)
 
 

@mcleantom
Copy link
Author

mcleantom commented Nov 11, 2024

That worked

@mayeut
Copy link
Member

mayeut commented Nov 11, 2024

So I guess #2076 should fix the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants