Remove dependency on sh

[casey]

sh is just's only significant dependency. I've been thinking for a long time that it would be nice to get rid of it.

I think the best way would be to write a shell library in rust. It would probably be easiest to make it a drop in replacement for dash. It could be integrated as a library to avoid having to fork a new binary to run each command.

Some benefits of this would be:

• Faster execution time, since there is no need to fork a new binary to run each command
• Works the same independent of the system-provided sh
• Support windows
• Can tokenize and parse recipes ahead of time, and thus emit errors regarding malformed recipes without having to actually run them
• Currently, variable assignment and cding in recipes don't work as one would expect, since each line is executed by a new shell. If instead some kind of sh library were used, these could be made to work with the expected semantics
• Support pipefail
• Can better integrate just variables and shell variables, perhaps allowing both to be accessed with $NAME
• Can check for use of undefined shell variables
• Can provide better error messages
• Since the sh library will be doing the parsing, multi-line constructs would not need to be backslash escaped

Additionally, I think that such a project would have a lot of utility outside of just. sh is one of the most important programs on a system, and writing it in a more secure language like rust would have a lot of benefits. It might be possible to add a lot of aggressive optimizations, features which aren't feasible in C. Also, although the base library would have a very small feature set, and would be a drop in replacement for dash, with the ability to add plugins, it could be the starting point for a more feature rich shell.

One difficult constraint is that we currently depend on Cygwin on Windows. Cygwin binaries contain a system call translation layer, which allows them to use unix paths and a unix userland on windows. If we wrote our own shell we would have to figure out how to continue to provide this functionality.

[jD91mZM2]

I think a cool way would be to let you choose shell. And then it would start the shell and pass commands to its STDIN. This way it wouldn't fork it every time, and also wouldn't be too much work, AND also would be simpilest to implement.

[casey]

@legolord208 I think that would be a neat, although I'm hoping to make just as cross-platform as possible. If we allowed using any shell, then it would also require that that shell be installed wherever the justfile was run.

[jD91mZM2]

@casey OK. I might be able to set up some parser PR some time later. How would you like the syntax?

EDIT: Like, what happens if you mix quote and not quote like "hello "world? In bash this would become "hello world". Should we do the same or throw an error?
Should it support continuing over multiple lines? (Either with unclosed quote or trailing backslash)

[casey]

@legolord208 I've been wanting to do this for a while, but I think it's too massive of a project to be done in a single PR. Also, I'm hoping for strict compatibility with an existing shell, either dash (the debian almquist shell, and the standard sh on most unices), or rc which is a simple sh alternative

[tbodt]

The exact functionality of sh is detailed in the POSIX spec, http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html

[jD91mZM2]

@tbodt Ouch, that sounds like a lot of work :O

[casey]

Yeah, shells do a lot of stuff! If you're interested in tackling writing a shell in rust, I'd suggest making it its own project.

[jD91mZM2]

@casey Oh, yeah! Perhaps there could be a flag to use ion as shell. Not sure if it has any way to communicate with Rust, but it would be cross-platform 👍

[tbodt]

A posix shell parsing library in Rust sounds like a useful thing. Not sure if it exists.

[bugabinga]

@jD91mZM2 ion is not aiming for POSIX compatability (which I assume to be important for Just based on previous comments). There is cicada, which calls itself a subset of bash, but as of today it is rather a POC. Neither support windows.

For anyone seriously attempting this, I suggest taking a look at oil. It aims to be a superset of bash in order to gradually replace it with saner syntax. It is a C/Python project but take a look at andyc's blog to understand what a massive endeavor this kind of project really is.

[casey]

I just released v0.5.0, which includes the ability to set the shell used for recipes and backticks on a justfile-wide basis. This gives just rudimentary support for cmd.exe and powershell, by adding set shell := ["cmd.exe", "/c"] or set shell := ["powershell", "-c"]. (see #530 and the readme for more details)

I think this sort-of addresses this issue, since sh is no longer a hard dependency. However, a big part of what's been discussed in this issue is the desire to have Just be usable with no external dependencies, and to be able to write cross-platform justfiles.

I opened #531 to track the issue of cross-platform justfiles (I also included a summary of the discussion and design from this thread) and #534 to track the wishlist item of bundling a shell and userland with just.

I hope the new set shell := ... feature is useful! Let me know if you have problems with it, or have suggestions for improving it.