Add escript_incl_priv so escriptize priv dir inclusion works with _checkouts and profiles

[mxxk]

When developing relx and testing changes with rebar3, it is convenient to make relx a checkout dependency.

However, the current implementation of escript_incl_extra expects the output directory of relx to be under _build/<profile>/lib/, which is not the case if relx is a checkout dependency.

Fixes #2600

Note

Due to the change in rebar.config, this is a backwards-incompatible change when using an older version of rebar3 to build a new version of rebar3. But because the recommended way to build rebar3 is with ./bootstrap, perhaps this is okay.

Test plan

1. Check out rebar3
2. Check out relx at rebar3/_checkouts/relx
3. Build rebar3 using ./bootstrap
4. Run ${path-to-rebar3-repo}/rebar3 release on another project
   • Before this fix, rebar3 crashes as reported in rebar3 release crashes when bootstrapped with relx as a checkout dependency #2600
5. Run ./escript-archive-list ${path-to-rebar3-repo}/rebar3 (see below)
   • Before this fix, relx/priv paths do not appear in the rebar3 escript archive

escript

#!/usr/bin/env escript
% escript-archive-list

main([EscriptPath]) ->
    {ok, Escript} = escript:extract(EscriptPath, []),
    [_Shebang, _Comment, _EmuArgs, Body] = Escript,
    {archive, ArchiveBin} = Body,
    zip:foldl(fun(Name, _, _, _) -> io:format("~s~n", [Name]) end,
                  undefined,
                  {EscriptPath, ArchiveBin}).

[mxxk]

As a bonus, these overrides for the prod profile can be dropped since the new implementation computes the output directory (using the current profiles in effect).

[ferd]

I don't think this works as a change. This is moving from a wildcard within any directory into something that becomes specific to OTP applications. I.e. an escript might want to bundle things unrelated to OTP apps, and there is no way to maintain that behaviour with the new configuration method.

[mxxk]

Good point @ferd. I can generalize the implementation of escript_incl_extra to not limit its use to OTP apps. To help steer this work, could you please clarify the following?

• escript_incl_extra is currently limited to rebar3-only internal use, but will it likely become available for other projects?
• What use cases are intended to be supported by escript_incl_extra? For instance:
  1. A directory generated by rebar3 (e.g., _build/<profile>/lib/)
  2. A directory under the project root (rebar.config), but not generated by rebar3 (e.g., priv/)
  3. A directory outside of the project root (e.g., /some/other/dir/). If any of these use cases are to be ruled out, I'd guess it's this one.

To support (2) and (3), it seems sufficient to allow static paths (as escript_incl_extra accepts today).

To support (1), are there additional use cases beyond an output directory of an OTP app?

[ferd]

We intended for it to only be used by rebar3, but as with all things, people who needed to bundle priv dirs in escripts started relying on it without us ever having improved the mechanism. So while its usage is discouraged, we're fully aware that changing behaviour will break random apps for people.

The use cases we have it for is just carrying priv/ dirs into apps, but not necessarily all of them. So we just did the quick workaround by adding a filepath with a regex. To carry the relx files, it had to be in _build since we don't have it from the root right away, but someone could use any path whatsoever, whether absolute or relative. It's messy, which is the reason we never really made the feature official.

{escript_priv_dir, [app | {app, Filter}]} would possibly be a nicer way to support this, that would partially overlap with the existing feature, but I don't know if it'd be cleaner.

A whole other approach would be the rebar.config.script trick doing detection of _checkouts and changing the paths accordingly, but that's extremely hackish.

[mxxk]

Just implemented. Do you think it is better to limit the new include wildcards to the app priv/ directory (as it is with this PR right now), or to keep it general, despite the rebar3 use case not needing it?

Also, the PR deviates somewhat from the app | {app, Filter} suggestion by only supporting {app, Filter}. The rationale here is to limit the different syntaxes which would need to be supported. Plus, {app, "*"} is arguably not that much worse to type out if needed. 😉 Of course, if there is a strong preference to retain it, it wouldn't be difficult to do so.

[ferd]

Yep, this looks solid! Thanks for the contribution there again. I believe the failing CI test is unrelated and is due to a third party plugin having a bad interaction since this feature logically has no overlap with it. This is going in.