Allow libraries without archives

[rgrinberg]

Before we fix this bug, we need proper tests.

@kit-ty-kate are there any other cases?

[kit-ty-kate]

Apart from adding (used in digestif.0.7.1, see #3904 (comment)):

(library
  (public_name foo)
  (wrapped false)
  (modules_without_implementation foo))

I don't see any other tests to add to that for now.

Thanks a lot for doing that!

[rgrinberg]

Apart from adding (used in digestif.0.7.1, see #3904 (comment)):

I'm not yet confident this is the same issue. Let me investigate this separately.

[kit-ty-kate]

I'm not yet confident this is the same issue. Let me investigate this separately.

It does look like the same category of issue to me, as with master this test-case works with OCaml 4.11 but fails on OCaml 4.12

[rgrinberg]

It does look like the same category of issue to me, as with master this test-case works with OCaml 4.11 but fails on OCaml 4.12

Indeed. I've added it as well.

[dra27]

One more case - an externally built library (so providing a META file) may have a .cmxa with no .a/.lib

[rgrinberg]

@dra27 do you mind going over the test cases + review the checks in dune_file.ml?

@kit-ty-kate could you please test?

@nojb would you be able to review the rest of the PR?

[rgrinberg]

I ended up discarding the dummy module idea - it doesn't make anything easier after all. If we decide to use a dummy module, we must create an Obj_dir.t that accommodates private modules (or pay a huge performance price to always use the inefficient layout)., but we already need an instance of Obj_dir.t to load the rules and to initialize the library database. Hence we are back at the same cycle that we wanted to avoid in the first place.

In the end, I had to bite the bullet and workaround this cycle. I chose to do it in a truly gruesome manner (my apologies to reviewers) to avoid making this PR even huger and more prone to mistakes. I will make another attempt to cut this gordian cycle between dir_contents, lib, and super_context in another PR.

[rgrinberg]

@dra27 this the sensitive bit.

[kit-ty-kate]

@rgrinberg Thanks a lot! I tried this branch locally and even though the great majority of packages compiled fine, some failed:

#=== ERROR while compiling uuuu.0.2.0 =========================================#
# context     2.1.0~beta4 | linux/x86_64 | ocaml-variants.4.12.0+trunk | git://github.com/ocaml/opam-repository.git
# path        ~/.opam/4.12/.opam-switch/build/uuuu.0.2.0
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p uuuu -j 3
# exit-code   1
# env-file    ~/.opam/log/uuuu-1304759-16c4d6.env
# output-file ~/.opam/log/uuuu-1304759-16c4d6.out
### output ###
#     ocamlopt gen/generate.exe (exit 2)
# (cd _build/default && /home/kit_ty_kate/.opam/4.12/bin/ocamlopt.opt -w -40 -g -o gen/generate.exe /home/kit_ty_kate/.opam/4.12/lib/bigarray-compat/bigarray_compat.cmxa /home/kit_ty_kate/.opam/4.12/lib/bigstringaf/bigstringaf.cmxa -I /home/kit_ty_kate/.opam/4.12/lib/bigstringaf /home/kit_ty_kate/.opam/4.12/lib/angstrom/angstrom.cmxa gen/.generate.eobjs/native/source.cmx gen/.generate.eobjs/nat[...]
# gcc: error: /home/kit_ty_kate/.opam/4.12/lib/bigarray-compat/bigarray_compat.a: No such file or directory
# File "caml_startup", line 1:
# Error: Error during linking (exit code 1)

#=== ERROR while compiling uri.4.0.0 ==========================================#
# context     2.1.0~beta4 | linux/x86_64 | ocaml-variants.4.12.0+trunk | git://github.com/ocaml/opam-repository.git
# path        ~/.opam/4.12/.opam-switch/build/uri.4.0.0
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p uri -j 3
# exit-code   1
# env-file    ~/.opam/log/uri-1304759-db578e.env
# output-file ~/.opam/log/uri-1304759-db578e.out
### output ###
#     ocamlopt config/gen_services.exe (exit 2)
# (cd _build/default && /home/kit_ty_kate/.opam/4.12/bin/ocamlopt.opt -w -40 -g -o config/gen_services.exe /home/kit_ty_kate/.opam/4.12/lib/stringext/stringext.cmxa config/.gen_services.eobjs/native/gen_services.cmx)
# gcc: error: /home/kit_ty_kate/.opam/4.12/lib/stringext/stringext.a: No such file or directory
# File "caml_startup", line 1:
# Error: Error during linking (exit code 1)

#=== ERROR while compiling tls.0.12.8 =========================================#
# context     2.1.0~beta4 | linux/x86_64 | ocaml-variants.4.12.0+trunk | git://github.com/ocaml/opam-repository.git
# path        ~/.opam/4.12/.opam-switch/build/tls.0.12.8
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p tls -j 3
# exit-code   1
# env-file    ~/.opam/log/tls-1304759-5cb026.env
# output-file ~/.opam/log/tls-1304759-5cb026.out
### output ###
#     ocamlopt .ppx/d824e2e5ac48a1dc0db041713fc7f187/ppx.exe (exit 2)
# (cd _build/default && /home/kit_ty_kate/.opam/4.12/bin/ocamlopt.opt -g -w -24 -o .ppx/d824e2e5ac48a1dc0db041713fc7f187/ppx.exe /home/kit_ty_kate/.opam/4.12/lib/ocaml/unix.cmxa -I /home/kit_ty_kate/.opam/4.12/lib/ocaml /home/kit_ty_kate/.opam/4.12/lib/ocaml/bigarray.cmxa -I /home/kit_ty_kate/.opam/4.12/lib/ocaml /home/kit_ty_kate/.opam/4.12/lib/sexplib0/sexplib0.cmxa /home/kit_ty_kate/.opam/4.[...]
# gcc: error: /home/kit_ty_kate/.opam/4.12/lib/ppx_cstruct/ppx_cstruct.a: No such file or directory
# File "caml_startup", line 1:
# Error: Error during linking (exit code 1)

[dra27]

I haven't reviewed the code, but I can confirm that with 4.12.0~alpha3 on msvc64, dune master can build stdlib-shims but not run the tests and this PR can both build stdlib-shims and run the tests, thanks! 🎉

[kit-ty-kate]

After double-checking using opam-health-check, I can confirm this PR is all good to go. None of the packages in opam-repository are now broken.

Thanks a lot! 🎉

[Octachron]

Asking the question on the right PR this time, what is the status of this PR? From the compiler point of view, the beta release of OCaml 4.12.0 is looming on the horizon, and #3766 is the major remaining blocker. Would it be easier from the point of view of dune team to backport this fix to a hypothetical dune 2.7.2?

[rgrinberg]

Would it be easier from the point of view of dune team to backport this fix to a hypothetical dune 2.7.2?

Releasing 2.8 would be easier as backporting this PR to the 2.7 branch might not be trivial. Once this PR is reviewed & merged, we'll release 2.8 ASAP.

[ghost]

LGTM