Bug with GNU make on windows jobs

[felipec]

Description

It took me a long time to figure out what was the problem, but it seems there's a problem in make when the configured shell ends in .exe, for example bash works, but not bash.exe, and the shell I see by default is sh.exe.

I tried configuring the sell manually:

make SHELL='C:/Program Files/Git/usr/bin/bash'

And that works, but only on the top level Makefile, when doing a submake with $(MAKE) -C the variable is reset to sh.exe, and then I see this error:

Windows Subsystem for Linux has no installed distributions.

Distributions can be installed by visiting the Microsoft Store:

https://aka.ms/wslstore

So presumably make is ignoring the default SHELL variable and running C:/Windows/System32/bash.exe for some reason.

I found this curious message online for CMake:

  sh.exe was found in your PATH, here:

  C:/Program Files/Git/usr/bin/sh.exe

  For MinGW make to work correctly sh.exe must NOT be in your path.

  Run cmake from a shell that does not have sh.exe in your PATH.

  If you want to use a UNIX shell, then use MSYS Makefiles.

I see a ton of people mentioning, but no one explains why make doesn't work, but I can confirm that it doesn't.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

2.302.1

Is it regression?

I don't know

Expected behavior

I would expect make to just work (even with sub-makes).

Actual behavior

make doesn't work, it's using a wrong shell (not bash).

Repro steps

1. Run a job with windows-latest
2. Run make

If you need an way to reproduce this, I probably can create a very simple repo that reproduces it, but any project using make should be affected.

[felipec]

I forgot to mention something else, the failure only happens when the Makefile runs a bash script directly, like:

./script.sh

If it's a command it works fine (IIRC), like:

bash script.sh

Or:

./script.sh || true

[sergei-pyshnoi]

Hello @felipec. Can you please provide more detailed repro steps with code and pipeline example?

[felipec]

@sergei-pyshnoi OK, I created a very simple repo that reproduces the issue: felipec/make-simple. It literally does nothing but call a simple script in many ways.

Here's a test run. As you can see it works perfectly fine in ubuntu, but not windows.

## info
main SHELL C:/Program Files/Git/bin/sh.exe
main MAKE C:/ProgramData/Chocolatey/lib/mingw/tools/install/mingw64/bin/mingw32-make.exe
## test
sub/foo.t
Windows Subsystem for Linux has no installed distributions.

mingw32-make: *** [makefile:[10]: test] Error 1
Distributions can be installed by visiting the Microsoft Store:

https://aka.ms/wslstore

Error: Process completed with exit code 1.

The only way to make the test in the subdir work is to specify -C sub in the command, and the SHELL variable:

make SHELL='C:/Program Files/Git/usr/bin/bash' -C sub

Here's a run where that works.

I don't know how to tell GitHub actions to run all the steps regardless (I'm don't know powershell, although I guess I could use bash and do || true).

[felipec]

Ah, make -i ignores all errors and now I was able to test all the steps in this run.

test	sh.exe/bash.exe	sh/bash
test (sub/foo.t)	Fail	OK
straight (bash -c sub/foo.t)	OK	OK
sub-test (./foo.t)	Fail	OK
sub-straight-test (bash -c ./foo.t)	OK	OK

As you can see the default shell just doesn't work. I don't know how make is picking it.

In theory I could just do make SHELL='C:/Program Files/Git/usr/bin/bash' but as you can see in the step "Test with shell" that variable doesn't get carried to the sub-test ($(MAKE) -C sub test), I don't know why.

[sergei-pyshnoi]

Thanks for detailed information. We will take a look.

[ilia-shipitsin]

@felipec , git bash is "mingw" app while "make" is native win app. I'm afraid that here we observe one of mingw interoperation side effects ((

[felipec]

What do you mean native win app? I did make -v everywhere (on the runner shell, in the make shell, and even $(MAKE) -v, all of them show the same version:

GNU Make 4.3
Built for x86_64-w64-mingw32
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[felipec]

As you can see from this job, the issue doesn't just happen with sh.exe, but bash.exe as well.

This works: 'C:/Program Files/Git/bin/sh', this doesn't 'C:/Program Files/Git/bin/sh.exe'.

Maybe make is checking the extension and doing something different?

[ilia-shipitsin]

I planned to run make with debug -d and compare output. Maybe it will give some clue

[felipec]

@ilia-shipitsin Here it is: test with debug.

The interesting part is this with sh:

CreateProcess(NULL,C:/Program Files/Git/bin/sh -c sub/foo.t,...)

And this with sh.exe:

CreateProcess(NULL,bash D:\a\make-simple\make-simple\sub\foo.t,...)

[ilia-shipitsin]

can you try bash as well in order to compare "bash" and "bash" ?

[felipec]

I'm not sure I follow, but I called bash sub/foo.t directly in this job.

With C:/Program Files/Git/bin/sh it's:

CreateProcess(NULL,C:/Program Files/Git/bin/sh -c "bash sub/foo.t",...)

With C:/Program Files/Git/bin/sh.exe it's:

CreateProcess(C:\Program Files\Git\bin\bash.exe,bash sub/foo.t,...)

[sergei-pyshnoi]

Hello @felipec . Sorry for the late response. Please, can you take a look on result of this job . Look like every tests was passed. I am just changed #!/bin/bash => #!/bin/sh in the script. Look like it was reason why is make choses wrong executor of shell.

[felipec]

Look like every tests was passed. I am just changed #!/bin/bash => #!/bin/sh in the script.

Yes, I thought that might work, but /bin/bash should work as well.

Look like it was reason why is make choses wrong executor of shell.

No, this comes from the Unix world, a script that starts with #!/bin/bash is supposed to be interpreted by a program /bin/bash: see shebang.

For example a program /tmp/test.perl like this:

#!/bin/perl

print("foo\n")

Is supposed to be interpreted by perl, so doing /tmp/test.perl or perl /tmp/test.perl should be equivalent.

If the shebang was #!/bin/python then that should be the interpreter, but it could even be #!/bin/cat.

GNU make isn't making a mistake here, a foo.t script that starts !#/bin/bash is meant to be interpreted by bash. Note that in some Linux distributions this does make a difference, as /bin/sh could be a POSIX-compatible shell, not necessarily bash, so a script with bash-specific features would fail with sh but succeed with bash.

---

The question is why does this fail:

CreateProcess(NULL,bash D:\a\make-simple\make-simple\sub\foo.t,...)

But this succeed:

CreateProcess(NULL,sh D:\a\make-simple\make-simple\sub\bar.t,...)

[sergei-pyshnoi]

@felipec After some testing, I found that GNU make + GitBash works a little differently on Windows than it does on UNIX systems. For using bash as interpreter in this case you can put #!/usr/bin/env bash as shebang to your script. Here is success result of running make command.

[felipec]

After some testing, I found that GNU make + GitBash works a little differently on Windows than it does on UNIX systems.

Yes, that's explained in the documentation of GNU make: Choosing a Shell in DOS and Windows.

For using bash as interpreter in this case you can put #!/usr/bin/env bash as shebang to your script.

Yes, that changes this:

CreateProcess(NULL,bash D:\a\make-simple\make-simple\sub\foo.t,...)

To this:

CreateProcess(NULL,env bash D:\a\make-simple\make-simple\sub\foo.t,...)

But the question still remains: why doesn't the first one works? I did some tests and I don't think the $PATH environment variable is wrong, so plain bash should work.

This seems to be a problem with Windows itself, maybe there's some interaction with WSL?

For the record, env should not be needed in GNU make, its purpose is to avoid typing #!/usr/bin/python, which hard-codes the path in Linux, when the user might want to change $PATH and use a different binary, for example /usr/local/bin/python.

[sergei-pyshnoi]

@felipec . After investigation, we found that is after disabling WSL feature, make starts running the correct bash. Other setting like installation path end PATH variable is correct. The main problem is how exactly make running bash process. Unfortunately we can not do anything with this and WSL can`t be disabled in runtime. Please use provided workarounds. I am closing issue for now, but if you have additional question feel free to ask.