[RFC] Add cmxs linkage for executables

[nojb]

This PR introduces a new linking mode for executables allowing to link them as .cmxs archives. Concretely,

(executable
 (name foo)
 (libraries bar)
 (modes cmxs)
 (modules foo))

will define a target foo.cmxs linking together bar.cmxa and foo.cmx.

Opinions appreciated (it has not seen much testing but it seems to work in simple cases). If there is agreement I will flesh out the patch (tests, doc, support for bytecode?)

[bobot]

I would prefer if we could solve this problem without, in spirit, bundling .cmxs together because we are not able to load them. I think a solution to this question is just a question of installation and localization #1185.

On the other hand, that doesn't add to much code. But still trying the right way would be better, so at least the feature should be added as experimental.

@nojb:

• Do you want a META to be generated for this cmxs?

• Where should it be installed by default?

• What about c shared libraries from which the linked libraries depends? Since it could be hard to localize them during execution.

[nojb]

I would prefer if we could solve this problem without, in spirit, bundling .cmxs together because we are not able to load them. I think a solution to this question is just a question of installation and localization #1185.

I am not sure I follow, but I don't see anything "hacky" about this PR. It exposes a feature of the OCaml compilation model: cmxs archives can bundle libraries (and individual object files) together, which is something not yet expressible in dune.

One easy use-case of this is linking a library together with a small "glue" module to make a plugin out of it. Yes, maybe this can also be solved by making the glue module into its own plugin and then loading both libraries in the right order, but it seems more complicated (and it is, on certain platforms 😄) than simply linking the library and the module together.

Do you want a META to be generated for this cmxs?

Personally I don't have a need for it (cf below, I would expect the resulting cmxs to link in all its dependencies), but maybe there is a use for it?

Where should it be installed by default?

Wherever the cmxs arising from libraries are installed, I guess.

What about c shared libraries from which the linked libraries depends? Since it could be hard to localize them during execution.

I haven't looked at this yet, but I guess it should link in everything it can inside the cmxs.

[bobot]

It exposes a feature of the OCaml compilation model: cmxs archives can bundle libraries (and individual object files) together, which is something not yet expressible in dune.

I think dune has not the goal of expressing everything the compilers allows. For example the linking trick #136 became virtual libraries.

but it seems more complicated (and it is, on certain platforms smile) than simply linking the library and the module together.

It is easier with the following restriction:

• No other .cmxs should link with the bundled libraries, even the main program
• C shared libraries should still be located.

What about c shared libraries from which the linked libraries depends? Since it could be hard to localize them during execution

I haven't looked at this yet, but I guess it should link in everything it can inside the cmxs.

The current patch doesn't do it, and I don't know if the compiler allows it.

How do you interact with the main program? The interaction library must be specified in the libraries field? But in that case it is statically linked with the .cmxs. How does that work?

[nojb]

How do you interact with the main program? The interaction library must be specified in the libraries field? But in that case it is statically linked with the .cmxs. How does that work?

Right now I am passing the -I flag by hand, but indeed this would need some additional work.

[ghost]

Regarding what features of the language Dune should provide: we should consider that Dune is a high-level interface for the OCaml language focused on the user experience. Invocations of the compiler is the assembly on which we build a consistent set of high-level and easy to use features.

When we can't find the right way to nicely abstract and expose a feature of the compiler such that it is easy to document and is consistent with rest of Dune, and such a feature is important for users of Dune, then it is nice if we can provide an escape hatch for expert users.

In the case of this PR, we have two choices:

• either we agree on a definition of the cmxs linking mode
• either we make sure there is a way for expert users to link such cmxs plugins

For the second option, it seems to me that the only thing that is difficult currently is getting the transitive list of cmxa files to link. So for instance if we added a way to get it, then an expert user might write:

(rule
 (targets (:p plugin.cmxs))
 (action (run ocamlopt -o %{p} %{transitive-cmxa:lwt,re})))

[bobot]

In the case of this PR, we have third:

• either, make easy to install all the needed cmxs and META needed for loading the plugin.

Something similar to transitive-cmxa, transitive-plugins could be used for knowing the needed files.

[ghost]

Indeed. You don't even need META files BTW, cmxs files already contain dependencies information. Given a set of cmxs files, it is possible to find out the order in which they should be loaded. We have some code for that somewhere at Jane Street.

[nojb]

Thanks for the comments; adding some expert-level variables seems a better way to go about it.