Consider making just variables environment variables 🔥

[casey]

I've long thought that it was good for just variables to be distinct from environment variables. However, I've grown increasingly unsure of that decision.

Pros of just variables being distinct from environment variables:

• Compile time undefined-variable errors: Misspelled variables in interpolations {{..}} are reported at compile time, not at runtime. (However, because just invokes sh with the -u flag, misspelled variables would produce an error at runtime, instead of evaluating to the empty string.)
• Easier access in shebang recipes: Can use "{{VAR}}" instead of import os; os.getenv("VAR").
• Avoids polluting child process environment: Variables do not pollute the downstream process environment, unless prefixed with export.
• Lines are echoed with the values of variables, e.g. echo foo instead of echo $variable_name.

Cons of just variables being distinct from environment variables:

• Can't use a=b just foo to set variable a, must use just a=b foo or just --set a b foo.
• Verbose: Must use {{VAR}} instead of the shorter and more common $VAR.
• Complex quoting interactions: Depending on the shell or shebang-recipe interpreter, interpolations may require quotes, for example if the variable value contains spaces, quotes, or special characters.
• Unquoted variables that evaluate to the empty string or whitespace will not be picked up as arguments: If FOO is empty, then cmd {{FOO}} a b c will run cmd a b c, whereas if FOO were an environment variable, cmd $FOO a b c would run cmd "" a b c.

On balance, I think it's better for just variables to be environment variables. The benefits aren't huge, and quoting and escaping feel like a thorny issue.

My current thinking is that this should be supported with a per-justfile setting, and a command line flag, either of which would trigger export of all variables and parameters:

set export := true

foo := "bar"

# `a` and `foo` are exported, so this works:
baz a:
  echo $foo
  echo $a

This way, users who preferred this behavior could opt in to it. Eventually, if we so desired, it could be made the default behavior after a deprecation period.

What do people think? There was an issue that I can't find at the moment where someone wanted this (I think it was @Fleshgrinder) and I argued for the opposite, but I've started to come around.

[runeimp]

I like that the just environment isn't polluted with the system environment variables by default. This can be extremely useful. For the few ENV Vars I need I can specify access via the env_var() and env_var_or_default() functions. And you can export those variables as well. The only problem I see is that if I wanted to have access to my entire system environment it would be a pain. But a logical fix could be a function export_env() which could pull all system ENV Vars as if I'd setup export HOME = env_var("HOME") for every system ENV Var. Maybe with the option of supplying a list of ENV Vars to exclude. And possibly a different function import_env() which would do the same thing but not export them, making them only available via the {{VAR}} syntax.

Or maybe command line switch with that sort of functionality. But that would be more cumbersome. I use just so I don't have to remember all the command line switch for everything in my projects. But still better than having all system ENV vars always populating the just environment.

[casey]

Here's an interesting idea, how about if prefixing a recipe argument with a $ caused it to be exported:

foo $bar:
  echo $bar

$ just foo baz
echo $baz
baz

This would allow users to choose if they wanted arguments to be exported. It would also fix the asymmetry where it's possible to export variables with export foo := "bar" but it isn't possible to do the same with parameters.

[casey]

@cc Fleshgrinder

[Fleshgrinder]

This was in #245 where I was talking about it.

GNU make does not pollute the environment by default unless there is a lonely export (a line that contains only the keyword) somewhere, which switches to export-all. This is weird behavior and definitely something we don’t want to copy. That being said, an export-all mode is extremely handy but it should have a dedicated keyword.

It’s own environment does get polluted with the shell’s environment but I’ve never in more than 10 years had a problem with that. Either one allows overrides (VAR ?= default) or not (VAR := default), very simple.

There are also per target variables and exports:

build: export SOURCE_DATE_EPOCH ?= $(shell git show -s --format=%ct)
build: # …
    $(CC) # …

[casey]

@Fleshgrinder What do you think about $ syntax for opting into exported parameters?

foo $bar:
  echo $bar

$ just foo baz
echo $baz
baz

[Fleshgrinder]

Would my example from above work with your idea?

build $SOURCE_DATE_EPOCH=`git show -s --format=%ct`:
    gcc # …

In other words, is SOURCE_DATE_EPOCH also exported if not given via the command line?

[casey]

Not by default. You'd have to use the much messier:

build $SOURCE_DATE_EPOCH=env_var_or_default("SOURCE_DATE_EPOCH", `git show -s --format=%ct`):
    gcc # …

Something like:

build: export SOURCE_DATE_EPOCH ?= $(shell git show -s --format=%ct)

Is difficult to support in the parser, since after the parser sees build: it is expecting a list of dependencies, and disambiguating a list of dependencies vs an export would be rough.

For scoped variables on a per recipe basis, possibilities include modules:

mod build {
  export SOURCE_DATE_EPOCH ?=  $(shell git show -s --format=%ct)

  build: # …
    $(CC) # …
}

In a possible future with modules, just build with the above justfile would resolve to build::build

Some way of introducing a scope on a per-recipe basis in which variables can be constructed is a feature that's been requested a few times. It could be supported with something like:

# recipe with scope has a `{` at the end of the line
foo: a b c {
  export SOURCE_DATE_EPOCH ?= `shell git show -s --format=%ct`

  # hypothetical syntax for recipe lines inside of blocks
  > echo foo
  > echo bar

  # another hypothetical syntax:
  : echo foo
  : echo bar

  # maybe with a dedicated line-continuation syntax:
  > echo foo
  | bar
  # the above turns into to "echo foo bar"

  # same effect as:
  # foo: a b c
  #   echo foo
  #   echo bar
}

or

foo: a b c {
  export SOURCE_DATE_EPOCH ?=  `shell git show -s --format=%ct`

  # inside of a block recipe, you can use `recipe:`
  recipe:
    echo foo
}

I think I am warming up to a ?= construct. I might spell it $=, to make clear that it involves looking up an environment variable.

[Fleshgrinder]

I see, I think that foo $bar: is still a good start to have auto-exported recipe parameters. It’s a shame that we already have the export keyword because otherwise we could really make everything super symmetrical by always using $ for environment.

# export to env
$VAR = value

# assign from env if set otherwise use default
VAR $= default

# export to env and use value from env if set otherwise use default
$VAR $= default

# same for args:
recipe $V0 = value V1 $= default $V2 $= default:

Looks a little weird with the many dollar signs…

Regarding scopes, I assume the recipe in your last example would need to be called as just foo::recipe, correct? Would it not be possible to simply support anonymous scopes:

{
    export SOURCE_DATE_EPOCH ?= `shell git show -s --format=%ct`
    build:
        gcc # …
}

[casey]

The export keyword is very readable, and I think would still serve an important purpose if more $-based syntax is introduced:

# create just variable
foo := "bar"

# create just variable and export
export foo := "bar"

# create exported environment variable only
$foo := "bar"

Anonymous scopes in that case are a bit unintuitive for me. It's not immediately obvious to me why recipes in that scope escape, but variables don't. I think that's why I favor syntax that puts the recipe name outside of the block:

build: {
    export SOURCE_DATE_EPOCH ?= `shell git show -s --format=%ct`
    : gcc # …
}

[Fleshgrinder]

But what if I would like to share certain variables in a scope with multiple recipes?

This is possible in GNU make:

t1:: export EV1 ?= v1
t1 t2:: export EV2 ?= v2
t1 t2 t3:: export EV3 ?= v3

t1::
    echo "$@ ==> $(EV1) $(EV2) $(EV3)"

t2::
    echo "$@ ==> $(EV1) $(EV2)"

t3::
    echo "$@ ==> $(EV1)"

[casey]

Thanks, that's an interesting example. That's pretty nutty, I didn't know about the extra :.

I'm not entirely sure how that would be best supported. I can see how that's a useful feature for make though.

I can't think of another language that has similar functionality, i.e. a statement that makes values visible in other constructs selectively. It's almost like an export to statement. It would be like if C had:

insert_into t1 { int t1 = 0; }

insert_into t1, t2 { int t1 = 0; }

insert_into t1, t2, t3 { int t1 = 0; }

int t1() {
  return v1 + v2 + v3;
}

int t2() {
  return v2 + v3;
}

int t3() {
  return v3;
}

Make's syntax for this could not be adopted directly. Since make doesn't have arguments, in A B C:: and A B C:, A, B, and C are all targets. But in just, recipes can have arguments, so in the same construct A is a recipe and B and C are parameters. As such, the parser would need unbounded lookahead to disambiguate A B C: and A B C::.

I would probably reach for some kind of block syntax:

a b c {
  export := "foo"
}

But again, we would have unbounded lookahead to disambiguate between a b c ... { and a b c ... :.

Commas might be the answer:

a, b, c {
  export := "foo"
}

Since then NAME COMMA is always an export-into block.

[Fleshgrinder]

The extra colon is a pretty handy feature in general. I mostly use make as a task runner (that’s why I like just [besides the fact that it’s made with Rust]) and almost all my targets are using double colons because it also makes the target unconditional. Kind of invalidates the argument against make in the just README. 😋

[casey]

Yeah, but it's probably better not to have to learn the obscure and non-obvious difference between :: and : to begin with, and I think BSD make differs fro GNU make in this regard.

[Fleshgrinder]

💯% agree, GNU make is a very weird piece of software but as to date the most efficient task runner available. Now we just have to make sure that just becomes more efficient. 😉

[casey]

@bb010g & @NickeZ I notice that you thumbs down this issue. If you wouldn't mind, could you explain why you would prefer Just variables to not be environment variables? I continue to be on the fence, so any user feedback is about this is very valuable to me.

[NickeZ]

I reread the issue, and it sounds like you propose two features. 1. Importing environment variables by default and making them just variables and 2. exporting variables by default making just variables available to sub-processes.

I'm mostly against importing variables into just because that means that hidden state from the environment leaks into your build process. If you on the other hand are required to specify them as arguments they will always be obvious.

For exporting variables to sub-processes I think it would make sense to have some special syntax to avoid polluting the sub-process with unintended variables.

[casey]

@NickeZ Thanks for clarifying.

I think 1. isn't necessary, since we have the env_var() function to access variables from the environment.

For 2., I updated the issue with a proposal for a setting which would make trigger export for all variables and parameters, namely:

set export := true

foo := "bar"

# `a` and `foo` are exported, so this works:
baz a:
  echo $foo
  echo $a

(I recently added the set construct for justfile-wide settings, although it hasn't made it into a release yet.)

This would be a fairly simple feature, and would be convenient for people who wanted to opt into export for an entire justfile.

I'm definitely sympathetic to the argument that it's not ideal for just to pollute sub-processes with variables that they won't use. However, the quoting issue seems rather thorny, i.e. that variable interpolations with spaces and special characters must be quoted. Switching to environment variables would effectively solve the quoting problem, and although polluting downstream environments with extra variables is not ideal, I'm not sure if it's something which often has a negative effect or causes usability problems in practice.

I could see set export := true eventually becoming the default, after a deprecation period. (And, people could opt out with set export := false.)

[casey]

I think I'm going to close this. I personally find the argument of not polluting subprocess environments to be compelling. I would however like some syntax for exporting recipe arguments (Like foo $BAR: ... to export the argument BAR.) If that were added, users could opt-in to exporting any variable or argument, which I think would be better than exporting variables by default.

[casey]

I just released v0.8.6, with two features that allow exporting more Just variables as environment variables.

Parameters can now be prefixed with $ to export them as environment variables:

foo $bar:
  echo $bar

And adding set export anywhere in the justfile causes all variables to be exported without needing export var := ... or $:

set export

a := "foo"

baz $b:
  echo $a
  echo $b

[casey]

Also, there's another good reason not to make just variables environment variables.

Just a few days ago, @alefbragin mentioned that it would be nice if there was a way to pass arguments as positional arguments. If this is implemented, this will provide another way of dealing with hard-to-escape values, i.e. refer to them as positional arguments, e.g. $1 / "$@".