REPL shell mode calls the shell with incorrectly escaped arguments

[vtjnash]

shell> ./julia -E "println(ARGS)"
bash: syntax error near unexpected token `('

shell> ./julia -E \"println(ARGS)\"
"println(ARGS)"

shell> ./julia -E println\(ARGS\)
bash: syntax error near unexpected token `('

[vtjnash]

@StefanKarpinski #4635 seems to have had a long tail of bug fixes, but it still doesn't work work for any non-trivial argument list (see above). Is there any chance of fixing this behavior? Or is nobody actually using this feature?

Brainstorming with Jeff, it seems we might need a custom command parser that handles $ and \$, but leaves ' and " alone, and wraps the results of interpolating any values in quotes and then calls Base.shell_escape (after accounting for iterables), unless the result is a Cmd in which case it shouldn't wrap the value in quotes before calling Base.shell_escape. Have I missed anything?

ping other possible interested parties from looking at who's contributed to this function: @stevengj @staticfloat @Keno @JeffBezanson

since nobody else seems to notice this, perhaps we should just reclaim the ; key? I'm pretty sure I could find some new uses for it.

[staticfloat]

I use ; all the time, but I guess I've just never tried to run shell commands with parens; I'm in favor of fixing this properly. I'm too busy right now to dig into the problematic behavior, but as finnicky as quoting rules are, I think we will be able to arrive on the "proper" behavior.

[vtjnash]

do you just run simple commands like ls and cd? we probably don't need to be invoking the shell just to handle those.

[staticfloat]

I tend to use pipes and redirects pretty often as well, so the shell mode is definitely useful.

[tkelman]

Please ask for a second opinion before adding the backport pending tag on anything non-trivial.

[eschnett]

@tkelman Apologies. Removed.

[tkelman]

We can maybe put it back, but interactive shell-mode related changes need a lot of hands-on testing time and are at higher-than-usual risk of platform differences so I don't want to rush things between RC and final. Maybe post-0.4.0.

[lkuper]

I think I just stumbled over this bug while trying to use run:

julia> run(`./example.sh`)
./example.sh: 2: ./example.sh: Syntax error: "(" unexpected
ERROR: failed process: Process(`./example.sh`, ProcessExited(2)) [2]
 in run at ./process.jl:531

where example.sh is essentially the following:

var1="foo:bar"
var2=(${var1//:/ })
echo $var2

It works fine run from the command line with bash:

$ ./example.sh 
foo

Incidentally, Syntax error: "(" unexpected is also the error raised by the above script when run with dash, not bash, and so at first I thought that Julia was somehow calling out to dash!

(Edit: turns out we can work around the bug by calling run(./example-wrapper.sh) where example-wrapper.sh is just a script that calls ./example.sh. So this isn't a showstopper.)

(Edit 2: On second thought, the issue was that there was a comment in the file before #!/bin/bash. It worked fine from the command line, and not from within Julia.)

[tkelman]

what is the shebang line set to in the script? julia isnt calling the shell, it's calling uv_spawn

[vtjnash]

please do not hijack this issue to discuss unrelated questions. use Stack-Overflow or julia-users instead.

[lkuper]

@vtjnash Sorry. I wouldn't have brought it up if I didn't think my issue was the same fundamental issue.

[ihnorton]

@lkuper no worries, and thanks for using Julia. The shell situation still clearly has a few rough edges.