pwsh does not support -Command positionals, is incompatible with intended set shell and positional-arguments usage

[cspotcode]

Looking through various issues, I see a misunderstanding of how pwsh -Command accepts positionals. Unfortunately, it appears to accept them but actually does not (explained below) which makes it incompatible with the intended usage of set shell and set positional-arguments.

PowerShell Core · Issue #1554
Make positional-arguments default to true next edition · Issue #1367
#1050 (comment)

and powershell doesn't like unused positional arguments.

That assessment is perfectly understandable, but the truth is a bit different. powershell -Command does not accept positional arguments at all! Rather, it interprets all positional arguments as being parts of a single command.

A summary of the issues:

• positionals are not available by any means. pwsh equivalent to $@, $args, is empty
• positionals are part of the command, are eval'd as powershell syntax (this extends beyond simple issues with the dollar sign)
• empty positionals are dropped
• workarounds do not solve the above, and also make recipes non-portable (worst of both worlds)

Bash example for comparison

When we do:

$ bash -c 'echo $1' foo bar baz
bar

bash is (roughly) executing "echo $1" as if it were a script named foo passed 2 positionals: bar and baz
$0 is foo
$1 is bar
$2 is baz
$@ is a bash array containing bar and baz

There is clear separation between the command and the positionals.

$ bash -c 'printf "<%s> " "$@"' foo bar baz
<bar> <baz>

$ bash -c 'echo ${#}' foo bar baz
2

because the length of the $@ array is 2.

Powershell is not like that

Powershell does not behave like bash at all. To illustrate, we'll do the exact same thing we just did in bash, ported to powershell, replacing ${#} with powershell equivalent $args.count:

$ pwsh -nologo -noprofile -command 'echo $args.count' foo bar baz
0
foo
bar
baz

Powershell does not put foo bar baz into an args array and pass it to the script. Rather, the positionals are part of the command.
So it executes this line of code: echo $args.count foo bar baz

This means powershell is not compatible with intended usage of positional-arguments, unless you want the recipe name and all positionals to be evaluated as powershell syntax.

To get behavior akin to bash, you might try wrapping your command in a script block, making your recipe incompatible with bash:

This powershell syntax:

& { echo $args.count } foo bar baz

will output 3 because positionals after the script block have been passed to the block as $args, equivalent to $@

Note that this still breaks for empty strings.

$ pwsh -nologo -noprofile -command '&{echo $args.count}' '' '' '' 'fourtharg'
1

It also evaluates everything as powershell syntax, because it is all part of the command.

$ pwsh -nologo -noprofile -command '&{echo $args}' '' '([environment])' '' 'fourtharg'

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    Environment                              System.Object
fourtharg

or passing to an external executable:

$ pwsh -nologo -noprofile -command '&{node -p process.argv @args}' '' '([environment])' '' 'fourtharg'
[
  '/home/cspotcode/.volta/tools/image/node/18.15.0/bin/node',
  'System.Environment',
  'fourtharg'
]

A more concrete example:

# justfile for a python project
set shell := ["pwsh", "-nologo", "-noprofile", "-command"]
set positional-arguments
test *args:
  pytest foo

$ just test --ignore-glob='*{disabled,skipped}*'
pytest foo
ParserError:
Line |
   1 |  pytest foo test --ignore-glob=*{disabled,skipped}*
     |                                          ~
     | Missing argument in parameter list.
error: Recipe `test` failed on line 4 with exit code 1

[cspotcode]

This isn't necessarily a bug in just, but I think it's worth understanding. It might mean changes in the README, or it might mean that positional-arguments cannot become default behavior until just is able to execute commands without going through a shell. This would be similar to how make invokes commands directly, not calling out to a shell. Funny enough, after wrestling with these issues, I'm feeling like make is a more reliably option on Windows, since I can scoop install make and write cross-platform makefiles, no need to deal with shell issues. I would much prefer using just, of course!

yarn, a nodejs package manager, uses its own bash-like shell implementation to execute build scripts.

https://github.com/yarnpkg/berry/tree/master/packages/yarnpkg-shell#yarnpkgshell

Yarn uses a bash-like portable shell to make package scripts portable across of Windows, Linux, and macOS

Not suggesting just should go that far, but it would be nice if there were better ways to avoid dealing with a shell at all.

[runeimp]

First, thanks for looking into the issue so deeply. It is much appreciated. 😃

But make does use the shell. The Scoop install of make likely just includes a Bourne/POSIX shell with it. You could simply use Scoop to install a Bourne/POSIX shell, Cygwin, MSYS, Git Bash, etc. for your cross platform experience. When I have to actually do long term dev on a Windows box I install Git Bash anyways so done deal on those rare cases (for me). Or I use PowerShell by default because I regularly need to use Just on Windows systems but do the lion's share of dev on macOS. So I install PowerShell Core on my Mac and use PowerShell Desktop already installed on the Windows systems. In any case cross-platform is, unfortunately, a weakness for both tools.

[cspotcode]

But make does use the shell.

I stand corrected, you're right.

---

For open-source, I need a justfile that will work for most strangers wanting to contribute. My goal is to create a justfile that will work with zero or minimal effort, and hopefully zero modification since it's versioned by git.

You could simply use Scoop to install a Bourne/POSIX shell

Do you know of one on scoop? I can't find one. It doesn't have sh or zsh, and I can't use bash since that's already claimed by WSL2's shim. And it needs to invoke Windows tools, not Linux, so calling into WSL won't work.

I think Git for Windows doesn't put the bundled sh.exe onto PATH by default(?), so most people won't have it there. I can almost assume it'll live at C:\program files\git\bin\sh.exe and hardcode that into the justfile. Won't work for Github Desktop users since it installs to a different location; they'll have to modify the justfile and be careful not to commit the change.

I tried a conditional set shell where I check a couple well-known locations for sh.exe, but that seems not to be supported.

[runeimp]

I'm pretty sure Git for Windows/Git Bash does put their sh.exe in the PATH on install. Or maybe that's an install option?

Open Source Projects, that is a tough one...

1. make is likely your best bet if you're really trying to help users avoid installing project dependencies as they will likely have that already installed for most platforms and is likely on Windows just from other projects.
2. just is very popular. I have no idea what the numbers are but seems to be up there, so they may already have it. If not the extra step of installing a shell becomes the issue. But they are likely to have Git for Windows/Git Bash or Cygwin, MSYS, etc. already installed or will need it installed for other open source projects as well.
3. True cross-platform task/command runner. Not likely to support a make like experience. May be configuration only and no scriptability. Less likely to already be installed? Once again I don't know any theoretical numbers on this. Just a guess.

I was about to create my own make-alike when I found Just. It covers most of my bases but there is simply nothing that is like Just and truly cross-platform without also needing to install a scripting language as well such as Node, Ruby, PowerShell, Python, etc. The one task runner to rule them all doesn't exist yet to my knowledge. But Just does have plans to eventually have Python 3 built in. At that point it will be the unstoppable!

[cspotcode]

Or maybe that's an install option?

Yeah, it's an option with Git for Windows. Github Desktop doesn't offer the option.

If not the extra step of installing a shell becomes the issue

For open source, the issue is mostly 4 factors:

• documentation: can they easily google how to set it up?
• support: I can't offer support if they get confused
• time it takes to install: if it takes too long, they'll decide it's not worth contributing to the project
• risk it will fail: e.g. if it requires mucking with env vars. Installs should be mistake-proof

Asking someone to install just is good per these criteria, because the process is well documented, fast, and reliable: get scoop, install just, done.

They probably already have git for windows, but without sh on the path. The issue is, if someone googles "How do I get sh for this justfile?" they should find something. The just readme says "On Windows, just works with the sh provided by ..." which is technically true, but likely to fail without additional setup.

[runeimp]

To be honest if the setup is too tough for a developer then they may not be qualified to contribute. That is a very broad over simplification of course. And eliminating all reasonable barriers to entry should be a goal. But it is tough if the project isn't POSIX only or Windows only. The simple confusion of mixed platform parts (Windows batch files and POSIX shell scripts, etc.) can be very confusing to many who've only ever had experience with one platform. No matter what platform that is. It really is a tough problem.

[cspotcode]

To be honest if the setup is too tough for a developer then they may not be qualified to contribute.

Or if they're qualified then they can spend their time on any number of projects, and may focus on one with a smoother on-ramp. You have to make things smooth for people if you want them to contribute.

[runeimp]

I completely agreed with you when I said

And eliminating all reasonable barriers to entry should be a goal.

[casey]

@cspotcode Thanks so much for explaining this in depth! It sounds like it would be pretty hard to provide useful and consistent behavior for powershell. It sounds like, for powershell, we would need to both pass arguments to command blocks, and quote them:

& { echo $args.count } "foo" "bar" "baz"

[Aankhen]

I’m not entirely certain I’ve understood correctly, but it seems to me that script blocks like the one in your example would be enough to provide consistent behaviour; only the arguments need to be escaped for PowerShell, which means adding backticks before a fixed set of characters. That would provide the correct list of arguments and count within the block as well as avoid evaluating each argument as PowerShell.