Consider ability to reload .env files between recipes

[tomasol]

Maybe there is a better way, but I would like to modify the .env file from within a recipe and use it afterwards.
The only working solution I found was following:

modify:
    cp .env.template .env
    echo "TIME=$(date +%s)" >> .env
    cat .env

read:
    echo "Expected: $(grep TIME .env)"
    echo $TIME

# could this just be "all: modify read"?
all: modify
    bash -c 'unset TIME && just --dotenv-filename .env read'

Ideally there would be a flag to auto reload .env files between recipes. Right now they are read at startup and even calling just recursively does not work as the old environment variables will have precedence before the .env file.

[casey]

Would it be better if there was some way to modify variables without modifying the .env file?

[tomasol]

Thanks for quick reply. I would like to be able to run the recipes one by one, which works now, but also together. I need the persistence to keep an execution ID around. Can it be done without modifying the .env file?

[casey]

Thanks for quick reply. I would like to be able to run the recipes one by one, which works now, but also together. I need the persistence to keep an execution ID around. Can it be done without modifying the .env file?

I see, so persistence is a part of it.

You could do something like this:

id := `cat .id`

init:
  echo xyz > .id

You would have to run just init first, to initialize .id, but then on subsequent runs, you'd have it in the id variable.

Does that work?

[tomasol]

Not really. The solution works same way as an env file - it works if recipes are invoked separately, but breaks when invoked in a single run:

time := `cat .time`

init:
  echo $(date +%s) > .time
  cat .time

dep: init
  echo {{time}}

$ touch .time
$ just dep
echo $(date +%s) > .time
cat .time
1635542204
echo

$ just dep
echo $(date +%s) > .time
cat .time
1635542212
echo 1635542204
1635542204

What I really want is to generate and pass a persistent variable that works on separate recipes and on a chain.

[casey]

Yeah, that's tough. I can't think of a great way to do this, but I'll leave the issue open in case anyone thinks of something clever.

[page-down]

... I would like to modify the .env file from within a recipe and use it afterwards.
... I need the persistence to keep an execution ID around. Can it be done without modifying the .env file?
... generate and pass a persistent variable that works on separate recipes and on a chain.

So you want the ID to be generated in a recipe, and then use it in other recipes, right? This ID should be consistent throughout the current execution.

Your question is really about inter-process communication and information sharing. The easiest way to do this is to use shared temporary files.

# example
set dotenv-load := true

# just process id
#exec_id := `printf $PPID`

# UUID
exec_id := `uuidgen`
export EXEC_ID := exec_id

@_default: run

clean:
    rm -f {{exec_id}}.*

init:
    date +%s > {{exec_id}}.time

read:
    cat {{exec_id}}.time

#THE_TIME := '`just exec_id=$EXEC_ID read`'
THE_TIME := '$(<$EXEC_ID.time)'

dep: init
    # echo [TIME:`cat {{exec_id}}.time`] do something...
    # echo [TIME:`just exec_id={{exec_id}} read`] do something...
    echo [TIME:{{THE_TIME}}] do something...

run: dep && clean 
    echo running...

date +%s > 2F6A3669-E8AA-49FD-834D-FA4FF807D5CE.time
# echo [TIME:`cat 2F6A3669-E8AA-49FD-834D-FA4FF807D5CE.time`] do something...
# echo [TIME:`just exec_id=2F6A3669-E8AA-49FD-834D-FA4FF807D5CE read`] do something...
echo [TIME:$(<$EXEC_ID.time)] do something...
[TIME:1635949252] do something...
echo running...
running...
rm -f 2F6A3669-E8AA-49FD-834D-FA4FF807D5CE.*

As in the case above, you can use something like {{THE_TIME}} in any recipe, as long as it is in the shell, to get the time initialized in init.

@casey
We have functions related to system information arch(), os(), can you please add functions related to the current execution process? For example, process id, start timestamp, etc. This eliminates the need to rely on executing a third-party program, such as a shell, to get the parent process id $PPID.

Also, may I ask if you think the uuid generator function is worth adding? It won't add much code, but it's a useful function. Thank you very much.

---

The next step is to see if we can get just to establish a two-way connection between third-party (shell) programs instead of the current one-way execution.

One way to do this is to open a new file descriptor (e.g. /dev/fd/63) in just and listen for commands. the program executing in recipe, which will inherit the file descriptor by default, sends command execution information to just via e.g.

printf 'var := "value-from-init-recipe";' >&63

Gives the program in the recipe the opportunity to modify variables in the just process and even perform other set, export, ... and so on, to export variables to the environment in the subsequent recipe executions.

Also need to implement a similar named pipe approach on Windows.

In addition to file descriptor anonymous pipes, there are also named pipes, sockets, and other forms of communication. Just make sure it's cross-platform, lightweight, and performs well.

[casey]

One way to do this is to open a new file descriptor (e.g. /dev/fd/63) in just and listen for commands. the program executing in recipe, which will inherit the file descriptor by default, sends command execution information to just via e.g.

That's a really interesting idea. Would that work on Windows? Is it possible to open a file descriptor by number without opening /dev/fd/N, which doesn't work on macOS or BSDs? Sockets are probably the most cross platform, but you would have to write to them with nc or telnet, which would mean an additional dependency.

Another possibility is some kind of line annotation that would tell just that the output of that line was a directive, that just should parse and execute:

var := "hello"

foo:
  > echo var=goodbye

Scoping is pretty confusing, for example, in the following justfile, just would have to know which bar to modify:

var := "hello"

foo var:
  > echo var=goodbye

I opened up other issues to discuss the functions you mentioned, those seem like good ideas.

[page-down]

That's a really interesting idea. Would that work on Windows?

Yes, Windows support for inter-process communication is complete and covers anonymous pipes as well. PowerShell also has full access to dot net functions.

System.IO.Pipes.AnonymousPipeClientStream

Is it possible to open a file descriptor by number without opening /dev/fd/N, which doesn't work on macOS or BSDs?

As far as I know macOS has full support and can be created and used without problems.

Sockets are probably the most cross platform, but you would have to write to them with nc or telnet, which would mean an additional dependency.

No, since there is already just, then just will act as both server and client. At startup, a unix socket or tcp socket is opened, and the listening socket address is passed to recipes via an environment variable. When executing the client just --send, the client determines if the socket address environment variable is available, then connects and sends the command. an additional dependency does not exists.

Think of just as a simple in-memory key-value storage service. recipes would be able to share information among themselves, however, I think this is a complete over kill. Yet it is also the easiest to implement and the most stable way.

Another possibility is some kind of line annotation that would tell just that the output of that line was a directive, that just should parse and execute:

In fact, each line in recipes is a separate process. The core is still inter-process communication. Written this way, I don't see how just can get the data it wants to return from the child processes. If you don't open a new file descriptor, it already takes up stdout and stderr.

---

If you do implement a socket, then you are actually turning just into a tool with network capabilities. It has daemon server mode. You can connect via just --client and execute any recipe command, and modify any key-value data built in.

For example, you have 3 steps of recipes that are executed by 3 different remote servers in distributed parallel, and after execution, the variable data or even the file is sent back, and then the local steps continue.