[RFC] Relocation library

[ghost]

It is common for programs to read configuration of data files installed on the system, for instance in /etc or /usr/share. In order for the program to find the files at runtime, it is common practice to capture an absolute path at configure time and store this path in the source code of the program.

There are two problems with this approach:

• it is not clear how the program should find such files inside the project tests, i.e. before it is installed on the system
• this doesn't play well when the binary artifacts are moved around. Systems that keep caches of built packages might do that for instance (esy, opam, ...)

The idea of this proposal is to provide a library so that programs can easily find such files without hassle. When running tests, dune would arrange things so that the paths reported by the library points to the local installation directory, i.e. _build/install. This library would also honor the environment variable from various systems (such as OPAM_SWITCH_PREFIX: ocaml/opam#3411) so that the binary artifacts can be easily relocated.

API

Here is a possible API for this library:

type path = string

val etc : unit -> path
val share : package:string -> path
...

[bobot]

Frama-C get the location during tests of share using the environment variable FRAMAC_SHARE. With #680 and #205 this variable would be set automatically to the local directory.

However having a common library with the API given would factorize the work done by each tool, which is great.

User configuration directory could also be a good candidate (c.f https://github.com/gildor478/ocaml-xdg-basedir)

[ejgallego]

👍 for this; in Coq we have to use a very ugly set of heuristics due to this, moreover because we cannot depend on non-installed targets we must take into account placement into the build dir context.

[emillon]

As discussed with @rgrinberg: it is possible to implement this using a global library, say dune.registry. This library embeds an absolute path that is generated at build time, and can be overridden at run time through environment variables.

The build time default corresponds to the dune install target directory. It will be possible to customize it by configuring --prefix through #1579, but this should already work without this.

At run time, we will be able to override this using the environment. The corresponding variables will be used by dune exec (and dune runtest) so that when running tests, the relocation directories will be resolved to local ones.

Variants might be helpful here, so that we can install a javascript version that points to local storage or similar instead of returning paths. Another interesting extension would be to add a crunch mode that returns the file contents instead of the path. It would need to read a string embedded into the resulting executable file.

[rgrinberg]

We've discussed this further with @bobot, and there's a couple of more requirements:

• When configuring to build dune itself, we want to be able to customize the relocation library. So for example, we want to be able to specify that the usr binary directory should be /usr/local/bin.

• We want to be able to build "relocatable" binaries. These are binaries that lookup, etc, share, etc. relative to the binary's path.

[emillon]

• for the "default" installation prefix, do we need to support anything else besides /usr/local? it seems that make, for example, always defaults to this. /usr is reserved for the package manager, so it would be dangerous to have a dune-install version that defaults to this.
• about relocatable binaries, I believe it's less important, but we should be able to solve that with a variant. I'm not sure how well it's possible to portably read the path of the binary. Also, in the context of a library that has data, should we detect the path of the binary or of the library?

[bobot]

for the "default" installation prefix, do we need to support anything else besides /usr/local?

The opam default.

[rgrinberg]

for the "default" installation prefix, do we need to support anything else besides /usr/local? it seems that make, for example, always defaults to this. /usr is reserved for the package manager, so it would be dangerous to have a dune-install version that defaults to this.

When we configure dune, the default for all of these should be the opam default I believe. So when we build dune with OPAM_SWITCH_PREFIX set, dune should prefer to install to that prefix.

[emillon]

OK, works for me! I think that we have reached a nice consensus here :)

[emillon]

I'll try to implement a first version of this.

[bobot]

Contrary to what I implied with the reference, we can't use for the share directory the technique used for the version, because it should also work for dynamic libraries.

[nojb]

What is the status of this PR ? Are there any blockers ?

[rgrinberg]

I don’t think there are any blockers, feel free to work on it.

…

On Sep 9, 2019, 1:12 AM +0900, Nicolás Ojeda Bär ***@***.***>, wrote: What is the status of this PR ? Are there any blockers ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jorsn]

TL;DR

The ability to set static paths at compile time is necessary, even if in some cases relocatability is desired:
This RFC provides no solution for systems like Nix where every package has its own prefix.
Also, it is probably easy to add this possibility, by adding an environment variable PREFIX to dune install and/or adding an option --prefix, env var PREFIX and dune var %{prefix} to the other dune commands, at least dune build.
Finally, this issue hasn't been resolved in two years, so there is probably the need for a simpler solution, instead of no solution at all.

Problem

Even if the proposed solution may be beneficial in many scenarios, it does not cover all that are relevant. The ability to set data or config paths at compile time is still necessary:

The solution still relies on the implicit information that there is one central prefix for everything. This is not always true. What about distros/package managers where every package has its own prefix, like Nix, Guix, GoboLinux, DJB's installation scheme, etc.?

It may be sometimes desirable to have relocatable executables, but not always. The mentioned package managers/distros (probably all, at least Nix, Guix) install immutable and immobile packages. Hence, relocatability provides no benefit, while the ability to hard-code directories at compile time makes things far easier: Everything is fixed at build time anyway, but runtime discoverability is really hard in these systems.

Of course one can wrap executables providing environment variables, but what if two used libs use two data/config files with the same name under two different prefixes? Then, even a *_PATH variable containing all data dirs in question as dir1:dir2:… wouldn't help.

By the way, e.g. the Nix community should be really relevant to dune: According to repology.org, it's the second largest linux distribution measured by the number of packages, and it's a community with a great affinity to functional programming.

Apart from the subject itself, AFAIK this RFC hasn't resulted in a pull request in two years, and the disability of reliably detecting any config/data paths resulted in multiple issues (at least #1253, #1534). Also, so far dune allows to detect and hardcode some installation paths, only not all consistently: You can e.g. execute opam config var prefix in your build, but you can't catch the --prefix option. According to the man page, you can set the DESTDIR as env var, but not the PREFIX.

Solution

I think, the simplest possible solution would be to make the inconsistent behavior consistent by adding a PREFIX environment variable which has the same meaning to dune install as the --prefix option (simple), and maybe by adding the --prefix option to all dune commands (kind of a general option of dune), together with a dune variable %{prefix}. Then, one can check PREFIX/%{prefix} in at least in every external command invoked by an action, e.g. in a configure script.

Alternative Solution

Implement config variables (#1579, #1674) and let prefix be such a config variable, respected by dune install.

[bobot]

The MR #3104 merged not a long time ago tried to help in that direction. The feature added is experimental so it is time to ask for modifications.

The ability to set static paths at compile time is necessary, even if in some cases relocatability is desired:
This RFC provides no solution for systems like Nix where every package has its own prefix.

We use Nix for our CI, so I kept it in mind at first but since I haven't yet tested it in this environment I perhaps made mistakes. So I'm very happy that you reached us with this problem.

I haven't understood all the roadblock you described, to help that I will describe the features from #3104 that would help:

• The currently undocumented environment variable DUNE_DIR_LOCATIONS, allows to give the site locations of a package PACKAGE:SITE:DIR:...
• --prefix will install all the files in the given directory even for files installed into the directory of another package (e.g plugins )

Do you think those steps are in the right direction?

[jorsn]

@bobot So --prefix overrides defined sites? Is it possible to define the trivial site .? Or how is the installation prefix determined if --prefix is not set? Can one also query the sites as variables to run e.g. in an action

(action (with-stdout-to config.ml
    (echo "let data = "%{site share}/file.data"\n")))

? If yes, then this is basically what I propose, I think. If the sites can only be defined in the env var and not be queried as dune variable, then the config script can query them. This is the important half of my proposal.

[bobot]

A better documentation and example is available in #3794.

So --prefix overrides defined sites? Or how is the installation prefix determined if --prefix is not set?

It does in the sense that when you install in the site of another package it will still be installed inside the --prefix provided. The prefix by default doesn't have any good value in Nix, except perhaps $out.

Can one also query the sites as variables to run e.g. in an action ?

Not in this form, because the sites is not the same between during the compilation/tests and after the installation, but the stanza generate_modules generate the code which provide this information.

This is the important half of my proposal.

Great. Can you test it in nix? One step I don't know exactly is if and which setupHook to use. Perhaps the nix dune package can create it so that the variable DUNE_DIR_LOCATIONS can be filled automatically. Perhaps dune install can install some specific files in $out that indicate what to append to DUNE_DIR_LOCATIONS in the nix setupHook.

[jorsn]

One question: How can a lib l reference its own site foo? By (site l foo)?

If I know this, I think I can try it out with Nix.

[bobot]

Yes it is, in the install stanza. For example in the new examples.

[bobot]

@jeremiedimino Do you think this feature is now fully implemented?

[ghost]

That's my understanding, yes. And if there are still details to sort out, let's rediscover them rather than keep this old issue.

I'm closing this issue then.