Private modules and scope

[bobot]

In the documentation for private modules it is written:

Private modules are inaccessible from outside the libraries they are defined in

In the documentation for scope it is written:

a scope determines where private items are visible.

Is a library foo.b able to access the private libraries of foo.a (given that foo.opam exists)? The too quotes seems to imply different answers.

Currently it seems foo.b can't access them. In #106 the contrary was tried and refined multiple times.

EDIT: I mean't private modules of `foo.a` when I wrote private libraries of `foo.a`

[rgrinberg]

Is a library foo.b able to access the private libraries of foo.a (given that foo.opam exists)? The too quotes seems to imply different answers.

Private modules were intended to be accessible only from a single library. In the context of a scope, private and public refer to libraries so there's no contradiction.

I recall two ideas being thrown around for sharing code that is visible only inside a scope:

• Fixing private libraries to be usable by public libraries. This probably needs to be fixed anyway so it's good to start with this.

• Adding another visibility type for modules. protected modules would be visible to libraries in the same scope. This is sometimes necessary as sometimes factoring out the protected set of modules into its own private library can be awkward.

IIRC @nojb had some ideas regarding this as well in case I'm missing something.

[bobot]

Any directory containing at least one .opam file defines a scope.

I think I don't get clearly what "determine" in a scope determine means, so that's why perhaps I don't understand the same thing than you.

But I was clearly not clear in my first message, I edited it.

Indeed protected_modules could be added, I'm wondering what would be a good default in order to not have the same problem than private_modules. All the modules that are not in public_modules field (#1476)? A module is private if it is not in public_modules and protected_modules? Should we ensure that at most 2 fields of the three public_modules, protected_modules, private_modules are defined?

[bobot]

Or at most 3 fields of the four public_modules, protected_modules, private_modules and modules?

[rgrinberg]

Indeed protected_modules could be added, I'm wondering what would be a good default in order to not have the same problem than private_modules. All the modules that are not in public_modules field (#1476)? A module is private if it is not in public_modules and protected_modules? Should we ensure that at most 2 fields of the three public_modules, protected_modules, private_modules are defined?

I'd prefer that modules were private by default (except for the lib interface module). Protected is not a very good default because most users do not have a good intuition for what a scope is, and the use case for protected modules is narrow enough that we're not yet convinced if the feature should be added at all.

[bobot]

use case for protected modules is narrow enough that we're not yet convinced if the feature should be added at all

FWIW the current design for the migration of Frama-C needs it:

• Each plugin is a separate package: $plugin.opam
• Each plugin as a non-gui part and a gui part which are two libraries : $plugin.main and $plugin.gui.
• The gui part depends on the non-gui library and on the general gui framework library.
• So the non-gui part has an interface (the lib interface module) that define what other plugins could use from it (which could be nothing), but we still want the gui part to have access to everything from non-gui part.

Protected is not a very good default because most users do not have a good intuition for what a scope is

I agree it is not simple and scopes are not the good level for protected modules because the packages of the same scope can be installed at different time. So I though that protected would allow visibility only to the library of the same package.

I'd prefer that modules were private by default (except for the lib interface module).

What do you prefer if there is no lib interface module? The default module alias defines nothing if the other modules are private.

EDIT: remove opam in opam package because it is more ocamlfind package.

[rgrinberg]

FWIW the current design for the migration of Frama-C needs it:

If I understand it correctly, you would like to define protected modules in the $plugin.main? Would it work for you to move all the protected modules into a separate library? Say, $plugin.shared? This library would be kept private and would be covered by #1017

Otherwise, you would indeed something like protected modules

I agree it is not simple and scopes are not the good level for protected modules because the packages of the same scope can be installed at different time. So I though that protected would allow visibility only to the library of the same package.

Your suggestion of doing protected at the package level is definitely something to consider. I like that it's basically analogous to java's definition of protected, therefore should be more intuitive to users.

Another idea to consider would be to make the scope context itself first class to the user. For example, a user would be able to define a scope, and then attach modules to a scope in library definitions.

;; any stanza defined in this dir (recursively) will now have access to the plugin scope
(scope (name plugin))

(library
 (name plugin.main)
 (scoped_modules
  ;; these module are visible to libraries/exes defined in this scope
  (plugin mod1 mod2)))

(library
 (name plugin.gui)
 (libraries plugin.main))

In this scheme, we could provide some default scopes such as :package for example. This idea isn't well baked yet, but I do wonder if hard coding a set of visibility levels is always going to be too rigid.

What do you prefer if there is no lib interface module? The default module alias defines nothing if the other modules are private.

Isn't this what we want? The modules should default to private rather than public. Those who don't like this default can always invert it with (public_modules :all) or (private_modules ()).

[bobot]

Would it work for you to move all the protected modules into a separate library? Say, $plugin.shared? This library would be kept private and would be covered by #1017

It is possible but not just by moving modules, because some shared private module could depend on public module. So some re-export would be needed. However I think #1017 is a good thing to have (except for the third restriction).

This idea isn't well baked yet, but I do wonder if hard coding a set of visibility levels is always going to be too rigid.

We have the question of the -p (-for-release-of-packages) that allows to install package separately, which currently force to work at the level of packages. If we accept user defined scope we have to force some combination of -p so that you can't compile and install only part of the user-defined scope.

Isn't this what we want? The modules should default to private rather than public.

That would mean that if one define a library stanza with minimal fields, they can't use it. I prefer the default of an .ml without .mli or the default of -pack: without restriction everything is visible.

[emillon]

To expand on the potential use cases of protected (I'm not too fond of the name which is more tied with how inheritance works in Java - the feature is closer to package-private, ie what happens with no modifier), one thing that I'd like to handle is tests for non-public API.

For example, consider a library:

• there are several modules L_a and L_b
• L just reexports some parts of L_a and L_b
• there are some unit tests for L_a and L_b
• L is marked as public API in the docs. The rest (L_a and L_b) are just implementation details as far as dependencies are concerned.

At the moment, I don't think it's possible to make L_a and L_b private since it would make it impossible to link the unit tests against them.

I'd like a way to either force linking (libraries l :with-private-modules), or somehow use the fact that unit tests are in the same dune-project to make private modules visible.

[rgrinberg]

@emillon I think I just confused the package private and the protected modifiers in java. In which case I agree that protected is the wrong name. We can think of something else.

At the moment, I don't think it's possible to make L_a and L_b private since it would make it impossible to link the unit tests against them.

Don't you mean compile? Linking against the private modules is always possible as the archives will stay there. It's the cmi's that are hidden.

I'd like a way to either force linking (libraries l :with-private-modules), or somehow use the fact that unit tests are in the same dune-project to make private modules visible.

So this is a 3rd access scope. We now want modules that are visible to tests, but are still going to be invisible to other libraries in the package. I guess this is why I'm inclined to make something more flexible right off the bat.

[emillon]

Don't you mean compile?

Yes, I was thinking more generally that it's impossible to create an executable referring to these modules.

So this is a 3rd access scope

I haven't thought of that, but yes this seems to be something separate. Though the most important part of private modules is that they're not visible to the exterior. I think it could be acceptable if they were visible inside the project, but if we can find a flexible solution this is even better.

[bobot]

So this is a 3rd access scope. We now want modules that are visible to tests, but are still going to be invisible to other libraries in the package.

In @Emilion case we just want them to not be visible from the exterior, they could be visible inside the package. So since test stanzas could be part of a package the same rule for protected modules is usable.

[rgrinberg]

You don't think it's useful to have levels of visibilities within a package? I don't follow that a module that is visible to the tests should be visible to all libraries within the same package as well. But it's not such a bad approximation either.

[emillon]

If this was the only visibility knob we had, I think that it would be manageable. The number one thing I'd like to avoid as a package maintainer is other packages depending on non-public API. Having properly isolated opam packages within a module is a nice property, but it's easier to check "by hand" since all the source code is in the same repository. That could be a lint rule for example.

That being said, I'm fine if there's finer control, but I'm worried it will make the semantics and configuration more complicated later on.

[bobot]

So after discussion we agreed on, for a library L inside package P:

• modules is the set of modules of L
• package is the set of modules that other libraries and executable of P could access
• public the set of modules accessible by any libraries and executables that depend on L
• public is included in package which is included in modules
• In the stanza as a syntactic sugar the fields modules, package and public doesn't have to be included, modules is the union of the three fields.
• if private is defined and not package and public, they are set to modules without private
• the cmi of the modules are stored in three different directories for the different visibility
• foo.ml_gen_modules is an alias file with all the modules
• if foo.ml doesn't exists it is generated as an alias file with all the modules of public and package. The later are hidden from the documentation.
• virtual modules should be in package or public

[Alizter]

Closing this as there doesn't appear to be anything actionable.