Avoid unreleased versions of OCaml unless specified

[jberdine]

With opam 2, it is probably not desirable for the solver to choose the latest (and likely unreleased) version of ocaml unless specifically asked to:

$ opam switch create --empty ./
$ opam install ocaml
The following actions will be performed:
  ∗ install base-bigarray  base
  ∗ install base-threads   base
  ∗ install ocaml-variants 4.08.0+trunk+fp+flambda [required by ocaml]
  ∗ install base-unix      base
  ∗ install ocaml          4.08.0
===== ∗ 5 =====
Do you want to continue? [Y/n]

It is unlikely that I want 4.08.0+trunk+fp+flambda when just asking for ocaml. :-)

[AltGr]

It is planned to move most ocaml-variants packages to a specific repository (at least all the "branch" and "unreleased" ones), since they are not useful to the general public. People who need them would just need one opam repo add command, and that would reduce both noise and weight put on the solver for the others.

That would in this case solve the issue. Note that, however, there is no way to manually put different weight on different packages, and opam only optimises the overall freshness of the versions of packages it installs using its search criteria. So if let's say ocaml.4.07.0 is getting installed, the solver may choose ocaml-base-compiler.4.07.0, but also ocaml-variants.4.07.0+fp+flambda (which are the latest versions for either of the ocaml implementation packages), at its own discretion.

We haven't yet found a general solution to this problem, but in the case of creating new switches (which doesn't cover uses of --empty), we use a pre-defined, configurable formula to guide the choice of compiler. This is for example leveraged when running opam switch create . from a source tree: opam will use this formula to choose a compiler that is compatible with the local opam file, in case of ambiguity.

[gasche]

I just learned about this issue today (thanks to @rjbou) and I think it is a serious problem. We don't want to have opam2 users install unreleased versions of the compiler without noticing. The suggestion to move experimental compiler versions to a separate repository seems fine, has someone worked on it yet, is there a related PR or place to ping?

[samoht]

As far as I know, nobody has started to work on that. Any help to do that would be highly appreciated!

[avsm]

@gasche moving these to a new remote would be good. You may also want to look at the ocaml-version library -- my medium term plan is to try to autogenerate all the compiler variants from that library mechanically. The same library is used to generate (e.g.) the Dockerfile test matrix, so it's useful to have it programmatically expressed rather than cut-and-pasted across releases.

[gasche]

My time to work on this is very limited. What about a by-need approach? I propose to send a PR to remove all unreleased OCaml packages from opam-repository, and then wait and see who needs these in opam and kindly ask them to create (and maintain!) a new remote.

[gasche]

(Limited before October. But I think this should be solved sooner than that, because it sounds like a usability disaster.)

[avsm]

That sounds good. I have also not done the mechanical port for similar reasons :)

[avsm]

Actually, we do build 4.08 snapshots in the Dockerfiles, so this will break that CI. Cutting and pasting those into another remote would let me continue to build them...

[gasche]

@avsm, did you just volunteer? :-)

[avsm]

yeah ok, i'll take a look shortly

[avsm]

I just noticed that https://github.com/ocaml/ocaml-pr-repository (where we moved the opam1 PRs to) script has stopped updating, so it seems like that feature isn't particularly popular among our users. As far as I can tell to fix this PR we could:

• remove the 4.08 compiler directory only from the 2.0.0 branch on opam-repository
• add them to the ocaml-pr-repository, and update that one to be in opam 2.0 format and also add trunk compilers there.

Does this sound right ?

[gasche]

This sounds good to me, but I find it a bit odd that the 1.2 and 2.0 branches of opam-repository don't have the same compilers/switches available (I think we would have a simpler story if they exposed exactly the same installable things). I would be slightly in favor of also removing 4.08+trunk from the 1.2 branch, and advise trunk users to (migrate to opam2 and) add opam-pr-repository. But those who do the work decide.

[gasche]

P.S.: Thanks!

[avsm]

I've lost track of the rewriting a little bit, but my understanding is that the 1.2/2.0 divergence is deliberate due to some packaging changes between the two. @AltGr @rjbou may know more

[avsm]

I've switched the CI to using a refreshed https://github.com/ocaml/ocaml-pr-repository that is generated automatically. Docs and releases to follow, but this unblocks the 4.08 trunk builds for now.

[avsm]

This is quite a high priority bug to fix in a point release: it also affects RC versions of OCaml, which makes them difficult to stage in the mainline repository.

[rjbou]

As described in ocaml/opam-repository#13302 (comment), @avsm set up a new repo https://github.com/ocaml/ocaml-beta-repository. to host beta ocaml compilers packages.

[kit-ty-kate]

I believe this was fixed by the introduction of the avoid-version flag in opam 2.1.0 (via #4281 and #4527). Closing. If I made a mistake feel free to ask to reopen.