Module improvement tracking issue

[casey]

This issue tracks improvements and missing features of submodules

• Recipes defined in one module can be depended on from another
• Variables defined in one module can be referenced from another
• Inline modules #2442
• Share settings between modules #2271
• Allow submodules to load .env files
• Variables defined in a submodule should be overridable on the command line #2119
• Make all errors, not just compilation errors, refer to module source file path as appropriate
• Make just --show work with modules

This thread should probably be used for general discussion. If you're interested in one of the above features, mention it in this thread and I'll create a dedicated issue for it.

[vasdee]

I am definitely interested in seeing modules fleshed out a bit more. I'm coming from being a fairly heavy make user where our team could organize make 'stubs' and include them on a per-repo basis.

Something like this was nice and handy in Make-land, and is definitely nearly there with Just

Makefile

-include .env
-include make.d/*.make

make.d/docker.make

docker/build:
   docker build ...

docker/login:
   docker login ....

The recipe naming with the / gave us faux-namespacing, which worked fairly well, but is arguably better implemented with Just modules. The one thing that keeps me from using them in this Make style way, is that variables from main/root Justfile don't cascade into the sub-modules

This has made it difficult to fully embrace modules, so we resort to using imports in Just and doing a more 'restrictive' faux-namespacing in the recipe names.

I would therefore +1 all the following, as I think they all contribute

• Recipes defined in one module can be depended on from another
• Variables defined in one module can be referenced from another
• Allow creating inline modules that are declared and defined in a single module
• Add a setting which allows a parent module to propagate settings to submodules
• Consider adding a reverse setting, which allows a child module to inherit settings from its parent

As an aside, I think I'd be perfect happy as well if recipe names could include some special characters that allow the 'faux' namespacing I mention, / is ideal ( or even \, but even a . would suffice. Then I could just use imports

[prasannavl]

Adding this one to the list: #2271

[expelledboy]

We have nested just mods that goes:

./Justfile
./iac/mod.just
./iac/docker/mod.just
./iac/docker/stack/mod.just

Which allows us to call just iac docker stack deploy

However from ./iac/docker/stack/mod.just I want to call a task defined in ./iac/docker/stack/mod.just and not have to use the fully qualified just iac docker stack <local-task> because this module is used in multiple projects with different file hierarchies.

How exactly could I call a local task, without using the hierarchy?

To highlight the issue in a different way:

I have a parent git repo, which has git submodules. Each submodule I wish to have a Justfile that works independently for those development teams. However I wish to expose that submodule tasks to the parent monorepo without modifications to the submodule Justfile. This is not possible because as soon as I include it as a module I now have to call the fully qualified module path

Eg When running submodule/Justfile in isolation I can call just package without issue

build:
  # do stuff
  
package:
  just build

But when I include it in a monorepo, and try to expose its tasks to a parent Justfile just build must be modified to just submodule build

./monorepo/Justfile

mod submodule

./monorepo/submodule/Justfile

build:
  # do stuff
  
package:
  just build # <---- this now breaks, and has to be aware of its position in the module hierarchy

[expelledboy]

I have put some thought into this. Should calls to just defined in a nested module not default to recipes in the current file, and then navigate up until it finds a recipe that matches? Why does a submodule need to be aware of its location in the module hierarchy?

And similarly you can should be able to call a dependancy to a module, relative to the currently module hierarchy. ie

Given

./Justfile
./iac/mod.just
./iac/docker/mod.just
./iac/docker/stack/mod.just

In ./iac/docker/mod.just you should be able to create a dep to its modules, without having any need to know where its being included in the parent hierarchy.

deploy env="dev": (stack/deploy env)

# or 

deploy env="dev":
  just stack deploy {{env}}
  
# not

deploy env="dev":
  just iac docker stack deploy {{env}}

Not, as "docker/mod.just" would now have to know that it resides in iac, and can be installed anywhere else but in that exact hierarchy.

[flokoe]

I am particularly interested in "Allow submodules to load .env files". This is especially useful in Monorepo Setups. For example:

Imagine a directory structure like this:

.
├── common.just
├── .env
├── justfile
├── project1
│   ├── .env
│   └── justfile
└── project2
    ├── .env
    └── justfile

I use modules, to namespace the two projects and use the same recipe names (same ux) defined in common.just for each project.

The corresponding justfiles would look something like this:

• justfile

  set dotenv-load := true
  mod project1 'project1/justfile'
  mod project2 'project2/justfile'

• project1/justfile

  set dotenv-load := true
  import '../common.just'

• project2/justfile

  set dotenv-load := true
  import '../common.just'

If I execute just project1 foo just already hands down environment variables to the called module. It would be awesome if the module would also load its own env file and maybe even override existing environment variables.

I know there is the path syntax just project1/foo, but this only loads the env in the project1 directory, which is fine, and I would not expect it otherwise (this is useful on its own).

The cherry on top would be to be able to call modules directly as dependencies, maybe like this:

tests: project1::test project2::test

But this is not really necessary, as recursive calling just works fine:

tests:
  @just project1 test
  @just project2 test

[jszopi]

+1 to recipe dependencies between modules. I have a server plus database cloud deployment use case. I'm new to the stack, so I'm basically slapping commands together and I'd like just to be flexible while I'm figuring out what the dependencies actually are.

Here's a subproblem from the project to illustrate my point. I'm keeping the Django and Google Cloud Platform details in case someone wants to debate whether the use case is legit. I have three recipes factored into two modules. I find these smaller files easier to maintain and I can use the module names as namespaces, so that I have a e.g.client::deploy and server::deploy. The three recipes form a dependency chain:

server::deploy --depends-on--> db::schema-update --depends-on--> server::publish

because:

• the server version in server::deploy may be depending on the new database schema (but the schema is backwards-compatible with older server versions).
• db::schema-update runs the image published in server::publish

To break the circular dependency between the modules, I've put the server-publish recipe in a module called common. In lieu of cross-module dependencies, I can invoke just:

# ==============================================================================
# In server/justfile

mod common
mod db

deploy:
  just db::schema-update
  gcloud run services update # ...

publish:
  just common::server-publish

# ==============================================================================
# In server/db.just
mod common

schema-update:
  just common::server-publish
  # Invokes `python manage.py migrate`
  gcloud run jobs execute migrate # ...

# ==============================================================================
# In server/common.just
server-publish:
  python manage.py makemigrations
  gcloud builds submit # ...

The downside is that the just invocations work outside its dependency order resolutions system. While just can prevent a circular dependency like:

foo: bar

bar: foo

it can't detect:

foo:
  just bar

bar:
  just foo

Similarly, when there are multiple modules, the raw just invocations can prevent the detection of circular dependencies between modules.

Here's a thought though: in the case of my simple dependency graph, should I have to factor the modules into a DAG? What if I could simply write:

# ==============================================================================
# In server/justfile

mod db

deploy: db::schema-update
  gcloud run services update # ...

publish:
  python manage.py makemigrations
  gcloud builds submit # ...

# ==============================================================================
# In server/db.just

mod server "justfile"

schema-update: server::publish
  gcloud run jobs execute migrate # ...

At the moment, modules only allow referencing of recipes, and not settings, variables etc. As long as there's no cycle between recipes, circular dependencies between modules could be allowed as a convenience. After all, it's a command runner, not a build system or a programming language.

I appreciate that allowing circular dependencies between modules would prevent the implementation of a few of the features put forward by @casey, such as sharing variables (because their evaluation could trigger a cycle) or inheriting settings. But perhaps that's the scope of imports instead? AFAICT modules don't support any evaluation aside from the recipe dependency order, so there's a chance that its current features and implementation don't yet require the modules to form a DAG.

[W1M0R]

I'm also interested in "Allow submodules to load .env files".

[vimota]

I haven't seen the ability to --list module recipes, so that would be great to add!

$ just --list
Available recipes:
    foo ... # foo is a great module!
$ just foo --list
Available recipes:
    bar

[laniakea64]

I haven't seen the ability to --list module recipes,

It does exist: for your example just --list=foo

[vimota]

Ah thanks - that's good to know!

I still think supporting running --list on modules with just mod-name --help would be more familiar to users (ie. similar to other tools like git commit --help , docker run --help applying the --help to the subcommand).

[casey]

@vimota I definitely agree that would be nice. --help is a valid recipe argument though, so we would have to see that the path provided on the command line pointed to a submodule, and then re-parse the following arguments as flags and options, which I think would be kind of a mess.

You could always make the default recipe in a module call just --list MODULE, like so:

# in foo.mod
default:
  just --list foo

And then you can do just foo and see the recipes in that module.

[vimota]

@casey Agreed, I can see how that'd be tricky. And that's a great idea - thanks!!

[psibi]

Is it possible to add the resolved filename of a module as part of the json dump ?

Motivation: For adding modules support in the Emacs extension: psibi/justl.el#57

Right now the extension invokes this to find metadata about the justfile:

just --unstable --dump --dump-format=json

This includes the module information currently which is helpful for extending. Currently this is the format:

{
"modules": {
  "module_name_one": {
    "doc": "My nice project"
    ...
}
}
}

Apart from the above information, I would also like the file path of the module. So something like this:

{
"modules": {
  "module_name_one": {
    "doc": "My nice project",
    "source": "/home/sibi/my_project/folder1/module.justfile"
    ...
}
}
}