Feature request: include priv in escript

[Zalastax]

I have an OTP application (Gradualizer) which I want to package as an escript.
The application has some application specific files and following the OTP directory structure guidelines we have placed these in the priv directory.
Unfortunately, the priv directory is not included in the escript.

I didn't find an official way to include additional directories in the escript and after reading rebar_prv_escriptize I know that it's not a bug that priv is not included.

Since priv is an official part of OTP I would very much want priv to be included in the escript. I suggest we add it to escript_incl_extra.

[OvermindDL1]

The big issue is extracting it from the escript though, you either have to access the escript directly or have to use a custom erlang build since it hardcodes the priv location. The normal Elixir style is to read the priv files into a module at compile-time then it can be accessed via a global non-gc binary from anywhere at any time (or if the file can be parsed out then parse it at compile-time to bake it into a better structure at runtime for even better speed). Doing the same in straight erlang is...not quite as easy but still doable (pregenerator to generate an appropriate erl file), and might be the best way to handle those files?

But yeah, in the official erlang distributions the code:priv_dir call is hardcoded to the application directory with /priv appended, it's not really possible to have it look 'in' to a compressed file like an escript.

[tsloughter]

What is wrong with including priv the same as rebar3 does with the escript_incl_extra option?

https://github.com/erlang/rebar3/blob/master/rebar.config#L30-L31

[OvermindDL1]

%% escript_incl_extra is for internal rebar-private use only.
%% Do not use outside rebar. Config interface is not stable.

^.^

What it looks like that does is what I described though, it reads in the files as a {filename, filecontents} tuple, then bakes that into the compressed file, then it looks like to access it then checks if the code:priv_dir(...) exists and if not then it looks if it's in a baked escript file and access that straight. That's a lot of work that something like Elixir just makes a single line of code def get_baked_file(), do: unquote(File.read!(:code.priv_dir(:gradualizer)<>"/filename")) or so.

Oh, and even the code all about that option in rebar is still full of scary warnings like %% For internal rebar-private use only. Do not use outside rebar..

[Zalastax]

For now I would be happy with any hack that makes it work both in the shell and as an escript.
In the long term I think this option should be documented, include priv by default, and has to work as if the file was in a normal file system, e.g. we use epp:parse_file and that just has to work.

[tsloughter]

@OvermindDL1 not sure what you mean about having to do anything different if it is baked in. The usual way of accessing it after using code:priv_dir like in your elixir file is all it is as far as I remember.

I agree we need to get rid of the warning, its been the same api for it for so long it should just be marked stable.

[tsloughter]

Bleh, nevermind, scratch my last response.

If that actually works in Elixir then let me know how and we could duplicate it. I have no idea how they achieve that.

[tsloughter]

@Zalastax yea, the only hack I know is what rebar3 does to extract the files:

find_escript_templates(Files) ->
    [{escript, Name}
     || {Name, _Bin} <- Files,
        re:run(Name, ?TEMPLATE_RE, [{capture, none}, unicode]) == match].

[OvermindDL1]

The usual way of accessing it after using code:priv_dir like in your elixir file is all it is as far as I remember.

Uh, it shouldn't? You have a modified OTP distribution if so? o.O

If that actually works in Elixir then let me know how and we could duplicate it. I have no idea how they achieve that.

Remember that Elixir is basically erlang with a ruby'ish (ugh) syntax but a fantastic compile-time code execution macro system, so literally the expression File.read!(:code.priv_dir(:gradualizer)<>"/filename") will read in that file and return it as a binary, and wrapping unquote() around it within a function definition means to execute that expression at compile-time instead of runtime, which will then bake the entire file contents as a binary into the module source directly, which since it is inlined means it gets loaded into the module code cache, thus meaning it is not GC'd and is the fastest way to encode a large chunk of data into the BEAM, thus the whole expression of def get_baked_file(), do: unquote(File.read!(:code.priv_dir(:gradualizer)<>"/filename")) just makes a function named get_baked_file of 0-arity that returns the binary of that file directly from inside the module code cache itself (no faster way). You can of course encode things in many ways, dynamic function definition based on files, return of a map of binaries, parse a file out into a deep tree structure or whatever and encode that in, you can unquote anything in elixir that is representable via the erlang AST, so no ref's, not pid's, not etc... but all the usual things people use are just fine.

Way way way back when I programmed mainly on erlang (rebar1 was but a new thing just being thought of) there was a useful parse transform that could grab a file contents and bake it into a module similar to what the above elixir code does, wouldn't be hard to recreate it, or just make a preprocessor script that reads in a file, converts it to a binary string and writes out a *.erl file of it similar to above.

All of the above patterns are quite a bit more efficient than what rebar is doing at the cost of slightly more ram being used (you obviously don't want to store a 100+ meg file in the module cache but a few megs here and there is perfectly fine). The elixir macro/unquote method works for escript's, libraries, any method at all with no code changes.

Baking it into the module cache is of course not existing as a file on the filesystem, but accepting file binaries is a better method for gradualizer to run anyway instead of accessing the filesystem directly, however if needed you can always write out a temp file and read it in...

[tsloughter]

Ah, ok, it does it at compile time.

Then yea, going to have to stick to finding it by extracting from the escript with escript:extract.

[kevinlang]

I believe this is the relevant upstream issue to keep an eye on:

erlang/otp#4476

[ferd]

#2602 will cover the basic priv dir stuff, I would however consider NIF resources to be distinct.

[uwiger]

The big issue is extracting it from the escript though, you either have to access the escript directly or have to use a custom erlang build since it hardcodes the priv location.

Picking up this thread, I've worked on this a bit, and have added a file access API to my setup application: setup_file.

Initially, I ran into the problem when trying to support OTP applications as 'plugins' packaged as .ez archives. Obviously, this failed, since many apps try to access the priv/ directory, and even if my own plugins could do it using the setup_file module, many other modules would crash.

In setup-2.2.0, I tweaked the support to also deal with escripts, which also use the zip archive support, but with an envelope that erl_prim_file_loader doesn't recognize.

Ironically, almost all the core logic in setup_file is either directly calling existing OTP code (zip, erl_prim_loader) or slightly tweaking code borrowed from OTP (file_io_server). I'm a little surprised that OTP hasn't already added this kind of support, but don't want to be too critical, as they do such excellent work.

The functions I've implemented so far are: read_file/1, open/2, close/1, consult/1, script/[1,2]
Not much effort has been put on handling weird corner cases.