Strange issue with virtual libraries, non-virtual module ends up with missing implementation

[Lupus]

I'm running into a weird issue when trying to get rid of functors for platform-dependent parts of my library in favor of virtual libraries. I've distilled down my codebase down to this project to illustrate the problem: https://github.com/Lupus/virtual-lib-issue

lib folder has 2 libraries, a and b, a has one virtual module, and one ordinary module, b depends on a and calls function from ordinary module.

lib-unix folder includes a, which is an implementation for a virtual library from lib folder.

examples contains an executable which calls function from b, which in turn calls function from a.

When I try to build the executable, I get the following error:

File "_none_", line 1:
Error: No implementations provided for the following modules:
         A__Real_module referenced from lib/b/b.cmxa(B__Foo)

I'm a bit lost on why this happens.

OCaml version is 4.06.1, dune 1.11.0.

Any help on this is greatly appreciated.

[Lupus]

I removed usage of variants (sample repo is updated to reflect this), but it does not help. Upgrade of dune to 1.11.4 helps neither.

Looking at final linking command (listed below), there's no -I lib/a and no lib/a/a.cmxa (this file does not even exist), is that expected?

$ (cd _build/default && /home/vagrant/.opam/4.06.1/bin/ocamlopt.opt \
    -w @[email protected]@30..39@[email protected]@[email protected] \
    -strict-sequence \
    -strict-formats \
    -short-paths \
    -keep-locs \
    -g \
    -o examples/Example.exe \
    -I lib \
    -I lib-unix \
    -I lib-unix/a \
    -I lib/b \
    lib/b/b.cmxa \
    lib/the_lib.cmxa \
    lib-unix/a/a_unix.cmxa \
    lib-unix/the_lib_unix.cmxa \
    examples/.Example.eobjs/native/Example.cmx\
)

Inspecting other build commands it seems that Real_module is getting included only into lib-unix/a/a_unix.cma

Running[37]: (cd _build/default && /home/vagrant/.opam/4.06.1/bin/ocamlc.opt <....> -a -o lib-unix/a/a_unix.cma <...> lib-unix/a/.a_unix.objs/byte/a__Real_module.cmo)                                                                                                                                    
Running[39]: (cd _build/default && /home/vagrant/.opam/4.06.1/bin/ocamlopt.opt <....> -a -o lib-unix/a/a_unix.cmxa <...> lib-unix/a/.a_unix.objs/native/a__Real_module.cmx)   

Documentation on virtual libraries states that implementations are possible in them, but what will be the proper way to use those implementations in other libraries, depending on virtual library? Is that expected to work?

I "fixed" the build by running a couple of manual commands:

$ dune clean
$ dune build
File "_none_", line 1: 
Error: No implementations provided for the following modules:
         A__Real_module referenced from lib/b/b.cmxa(B__Foo)

$ cd _build/default/
$ /home/vagrant/.opam/4.06.1/bin/ocamlc.opt -w @[email protected]@30..39@[email protected]@[email protected] -strict-sequence -strict-formats -short-paths -keep-locs -g -a -o lib/a/a.cma lib/a/.a.objs/byte/a__Real_module.cmo

$ /home/vagrant/.opam/4.06.1/bin/ocamlopt.opt -w @[email protected]@30..39@[email protected]@[email protected] -strict-sequence -strict-formats -short-paths -keep-locs -g -a -o lib/a/a.cmxa lib/a/.a.objs/native/a__Real_module.cmx

$ /home/vagrant/.opam/4.06.1/bin/ocamlopt.opt -w @[email protected]@30..39@[email protected]@[email protected] -strict-sequence -strict-formats -short-paths -keep-locs -g -o examples/Example.exe -I lib -I lib-unix -I lib-unix/a -I lib/a -I lib/b lib/a/a.cmxa lib/b/b.cmxa lib/the_lib.cmxa lib-unix/a/a_unix.cmxa lib-unix/the_lib_unix.cmxa examples/.Example.eobjs/native/Example.cmx

$ ./examples/Example.exe 
x: 42

Is there a way I could make dune run these for me?

[Lupus]

If I just remove the virtual libraries related stanzas, everything links just fine.

diff --git a/lib-unix/a/dune b/lib-unix/a/dune
index 36c2e13..2fc0b5a 100644
--- a/lib-unix/a/dune
+++ b/lib-unix/a/dune
@@ -1,5 +1,4 @@
 (library
  (name a_unix)
  (public_name the-lib-unix.a)
- (implements the-lib.a)
  (libraries the-lib))
diff --git a/lib/a/dune b/lib/a/dune
index ff45c45..968fb32 100644
--- a/lib/a/dune
+++ b/lib/a/dune
@@ -1,4 +1,4 @@
 (library
  (name a)
  (public_name the-lib.a)
- (virtual_modules Virt_module))
+ (modules_without_implementation Virt_module))

[rgrinberg]

Could you try pinning this commit to see if it fixes your issue: rgrinberg@56ef00a

[Lupus]

I'll try in a moment. So far I managed to reduce it even further to this branch: https://github.com/Lupus/virtual-lib-issue/tree/pure-interface-lib

Only 3 libraries, virtual (even interface-only) a_i with corresponding a_i_impl, a which calls a function from a_i and an executable.

This patch makes it work (it makes executable call a_i directly, as opposed to via intermediate library):

diff --git a/examples/Example.re b/examples/Example.re
index 3eecde8..f4747e9 100644
--- a/examples/Example.re
+++ b/examples/Example.re
@@ -1,6 +1,6 @@
 
 let x =
-  A.Real_module.zzz();
+  A_i.Virt_module.virt_zzz();
 
 Printf.printf("x: %d\n%!", x);

[Lupus]

$ opam pin
dune.1.11.4               git  git+https://github.com/rgrinberg/jbuilder.git#56ef00a56067e3b04f1033a6e0fcf3334f6c2133
dune-build-info.1.11.4    git  git+https://github.com/rgrinberg/jbuilder.git#56ef00a56067e3b04f1033a6e0fcf3334f6c2133

Looks like it fixes the build in pure-interface-lib branch of the sample repository, but still does not build the master branch, which has virtual lib with implementation, but I'm not sure if the implementation bits of a virtual library are allowed to be used by other libraries directly?

[rgrinberg]

which has virtual lib with implementation, but I'm not sure if the implementation bits of a virtual library are allowed to be used by other libraries directly?

They are. So it looks like my fix wasn't completely sufficient. Thanks for the reproduction case, I've been trying to isolate this issue for quite a while now. Hopefully will have a full fix ready soon.

[Lupus]

Glad to help! Thanks for looking into this. Let me know if more testing is required. master branch should match the real library I have pretty close, but testing on a larger thing as well is always a good thing.

[jordwalke]

I wonder if we could start by merging the partial fix (it addresses my similar issue last time I checked).

[jordwalke]

I'm glad @Lupus has a publicly released repo that repros the issue though. That was something I wasn't able to share previously. Thank you @Lupus

[Lupus]

Same for me, I can't share the original repository where I ran into this. Luckily it was relatively small, so I was able to remove the noise and leave only the structure of libraries that reproduce the issue. Chopping it down to even smaller subset took more time actually, that's the branch pure-interface-lib, and it gets fixed by the commit Rudi shared. Is that commit the partial fix you talk about Jordan? Looks like it ought to fix transitive dependencies which end up in a virtual library?

[Lupus]

Correction: I meant not the exact commit itself, but the state of the branch at that commit, I glanced through it and it's not immediately obvious to me what exactly does it fix :)

[rgrinberg]

Hmm, in the end I'm actually not sure if the test case can be supported at all. The issue here is that we have a library that depends on the virtual parts, and then is used in the implementation itself. What should be the correct linking order?

We can express the dependencies between archives as follows:

impl.cma: lib.cma # the virtual lib implementation uses lib
lib.cma: $impl.cma # where impl is selected once we selected the implementation

This means that the implementation modules of impl.cma will depend on modules in lib, while the virtual modules are themselves required for lib. This clearly cannot work.

To fix this issue we need to enforce the following restriction: if a library depends on a virtual library. Then it cannot be used by the implementation of the same virtual library.

[Lupus]

If I'm not mistaken with the hierarchy, and assuming we're speaking about master branch of test repo, it looks like this:

I'm not exactly sure what's wrong with such dependency graph. a_unix which is the virtual library implementation, does not depend on anything (at least explicitly)?

P.S.: Graphviz code in case we'll need to modify the graph:

digraph G {
  a [shape=diamond];
  "the_lib_unix" -> "the_lib";
  "the_lib" -> a;
  "the_lib" -> b;
  b -> a;
  Example -> "the_lib_unix";
  Example -> "a_unix";
  "a_unix" -> a [style=dashed];
}

[Lupus]

Let me explain what I'm trying to achieve, may be that helps. the_lib is some platform agnostic library, which consists of sub-libraries, some of which require some platform features. I wanted to make those virtual libraries with platform features represented as virtual modules. the_lib_unix is the overlay library, which exposes the_lib with some platform-specific top-level modules for unix platform, and also drags in unix implementations of virtual libraries under the_lib, so that the final executable can depend only on the_lib_unix and get everything up and running. Currently this is done using functors. I've added direct dependecy from a_unix to the executable for troubleshooting this issue.

[rgrinberg]

I'm looking at your pure-interface-lib branch in fact. Since this was the simplified case, I chose to analyze it first. Here are the dune files you posted:

(library
 (name a_i)
 (virtual_modules Virt_module))
(library
 (name a_i_impl)
 (implements a_i)
 (libraries a))
(library
 (name a)
 (libraries a_i))

The definition for a_i_impl here is invalid. Because a depends on a_i, it cannot be used the implementation.

Cool graphics btw, I need to pick up some graphviz one day :)

[Lupus]

Okay, pure-interface-lib visualizes like this:

My previous graph was wrong, I've just concatenated all dune files like you did above and started converting that to graphviz iteratively and got the following result this time:

It has the same pattern, as demonstrated by the smaller test case actually...

To fix this issue we need to enforce the following restriction: if a library depends on a virtual library. Then it cannot be used by the implementation of the same virtual library.

Looks like now I understand this clearly. Probably this restriction will lead to better error message explaining what's happening?

Meanwhile I will try to apply this knowledge to my original library, hopefully I know what to fix now...

@jordwalke does this explanation help with the case you observed?

Cool graphics btw, I need to pick up some graphviz one day :)

May be teach dune to generate such fancy graphs of library/executable dependencies? Offending dependency path can be highlighted in red :)

[rgrinberg]

Fixed by #5050