Formalise the management of versions

[ghost]

This PR formalises the management of versions in Dune. It explains how versions of libraries and executables are calculated and how they are written in library metadata files and embedded into executables.

Overview

The Goal of this PR is to provide a simple and clean way to expose the version of libraries and executables via ocamlfind and the --version command line options of executables.

Additionally, when the version is obtained from the version control system we make sure to preserve good incrementality in order to preserve a good development experience.

Calculation of versions

The version of a library or executable is calculated as follow:

• if the element is public the version is the one of the package the element is part of
• if the element is private the version is obtained from the version control system the element is part of

The version of a package is the version written in the p.opam or dune-project file. When no such version exist it is obtained from the version control system the package is part of.

When obtaining a version from a version control system, the root of the version control system does not need to be inside the workspace and can be in an ancestor directory of the root of the workspace.

Version obtained from version control systems

When the version is obtained from a git repository, the output of the following command is used:

$ git describe --always --dirty

When it is ontained from a mercurial repository, it uses an emulation to produce a result similar to git.

Accessing versions from OCaml code

One can use the special dune.build-info library to access the version of the executable as well as the list of statically linked libraries with their version. This library provides a Dune_build_info module with the following API:

(** Provide build information *)

module V1 : sig
  (** The version at which the current executable was built. *)
  val version : string

  module Statically_linked_library : sig
    type t

    (** The most visible name of the library. If it is a public
        library, this its public name otherwise it is its private name.
    *)
    val name : t -> string

    val version : t -> string
  end

  (** All the libraries that where statically linked in *)
  val statically_linked_libraries : Statically_linked_library.t list
end

Some parts of this library are generated at link time.

Version stored in package metadata files

Dune store the version of a package in META and dune-package files.

Version control system and incrementality

When the version is obtained from a version control system, it is not immediately stored in binaries and package metadata files as this would hurt the development experience by reducing incrementality.

Dune rewrites META and dune-package at installation time using the appropriate parser and printer. It makes sure to preserve comments in META files given that part of these might be written by the user. dune-package are always fully generated by Dune and contain no comments.

For executables that depend on dune.build-info, we leave placeholders that are rewritten at installation time or when promoting files to the source tree. Dune systematically scans and rewrites all files during promotion and installation. This allows the rewriting to be performed even if the .exe was copied via a rule.

It is expected to work with javasript files as well as long as the version numbers don't contain special characters that are not allowed in javascript string literals.

Usage in Dune itself

dune makes uses of dune.build-info for dune --version and has a new --build-info option printing the version of dune and the list of statically linked libraries:

$ dune --version
1.10.0-10-gc9cb8d4e-dirty
$ dune --build-info
version: 1.10.0-10-gc9cb8d4e-dirty
statically linked libraries:
- catapult           1.10.0-10-gc9cb8d4e-dirty
- cmdliner           1.10.0-10-gc9cb8d4e-dirty
- dag                1.10.0-10-gc9cb8d4e-dirty
- dune               1.10.0-10-gc9cb8d4e-dirty
- dune._caml         1.10.0-10-gc9cb8d4e-dirty
- dune._dune_lang    1.10.0-10-gc9cb8d4e-dirty
- dune._ocaml_config 1.10.0-10-gc9cb8d4e-dirty
- dune._result       1.10.0-10-gc9cb8d4e-dirty
- dune._stdune       1.10.0-10-gc9cb8d4e-dirty
- dune._wp.dune      1.10.0-10-gc9cb8d4e-dirty
- dune.build-info    1.10.0-10-gc9cb8d4e-dirty
- dune_re            1.10.0-10-gc9cb8d4e-dirty
- fiber              1.10.0-10-gc9cb8d4e-dirty
- incremental_cycles 1.10.0-10-gc9cb8d4e-dirty
- main               1.10.0-10-gc9cb8d4e-dirty
- memo               1.10.0-10-gc9cb8d4e-dirty
- opam_file_format   1.10.0-10-gc9cb8d4e-dirty
- threads.posix      [distributed with Ocaml]
- unix               [distributed with Ocaml]
- xdg                1.10.0-10-gc9cb8d4e-dirty

[ghost]

Let's include this in 2.0.0 rather than 1.x. Additionally, I'm removing the old ad-hoc rules we had for VERSION, package.version, ...

[ghost]

Well, that simplified the code quite a bit :)

[ghost]

Alright, I rebased this PR and formalised the versioning of the generated module so that the build-info library can be distributed separately. I prepared a package for it on my github: https://github.com/diml/build-info

This is now ready for review and inclusion in 1.11.

[rgrinberg]

I'm having some second thoughts on splitting off build-info into its own repository. All the same issues apply:

• Now we need to test all builds in a composed fashion
• Users don't know where to report bugs
• Making refactoring across both projects is a pain.

I understand that we'll need to support this library for 4.02+, but perhaps we should still consider a mono repo like this:

$ ls dune/
  .git
  build-info/
  dune/
  ..

In this structure, there's complete independence between the individual projects.

[ghost]

For the testing, it shouldn't be too much of an issue. In particular the test in this PR do not require the external library. They just test the generation feature directly by defining a library with (special_builtin_support (build_info ...)). Given that the API of the generated module is versioned we won't need to synchronise changes.

However, it's true that it becomes more confusing where to report errors. Including build-info in the dune repo works with a few caveats:

• it won't be possible to dev-pin build-info. It would require teaching opam that a package can live in a sub-directory of a repository
• given the versioning story precisely implemented by this PR, we will have to synchronise the versioning of build-info and dune

For the last point, we could either make a new release of build-info with every new dune release, however it means that we will have a lot of identical releases of build-info which is odd. Alternatively, we could skip versions but this means that the versions of build-info will be patchy, i.e we might jump directly from 1.11 to 2.5 for instance.

[ghost]

BTW, in practice this won't change the implementation. We do need the versioning of the generated module. In particular this decoupling is valuable for mono-repositories. Indeed, in a mono-repo one would embed a copy of build-info and its version wouldn't necessarily match the version of the dune binary currently being used.

[rgrinberg]

Alternatively, we could skip versions but this means that the versions of build-info will be patchy, i.e we might jump directly from 1.11 to 2.5 for instance.

Hmm, I suppose this is the most convincing point. We can't really have 2 versioning schemes inside a single git repo. I do think that it would be better to version build info independently of dune.

[rgrinberg]

I'm wondering if it makes sense to make version an abstract type?

It seems like it would be quite nice useful to include some basic operations on versions here that aren't as convenient to do as strings. For example

val major : version -> int
val minor : version -> int
val semver : version -> (int, int, int)
val commit : version -> string
val compare : version -> version -> int (* this one seems especially useful *)

I don't have such a strong opinion on this, but from experience this is the kind of thing people do get wrong because they are simply lazy about it.

Of course, this will make it more future proof as well.

[ghost]

Indeed, using abstract types will make the API more future proof. I'm doing that

[ghost]

I made the version abstract and moved build-info to its own package as we discussed on slack

[rgrinberg]

I made the version abstract and moved build-info to its own package as we discussed on slack

I think you might have forgotten to push this change. I've resolved conflicts and merged this PR. I'll make a ticket so that we don't forget this change before releasing 1.0.

EDIT: Nvm. I see that the public API is in `build_info.mli.

[ghost]

Thanks for rebasing and merging :)