Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Idea/Proposal: Allow for public libraries to depend on private libraries #1017

Closed
rgrinberg opened this issue Jul 13, 2018 · 31 comments · Fixed by ocaml/opam-repository#17967
Labels
accepted accepted proposals proposal RFC's that are awaiting discussion to be accepted or rejected
Milestone

Comments

@rgrinberg
Copy link
Member

Abstract

Currently, private libs may depend on public libs but the reverse isn't allowed. Note that an analogous situation exists with executables, but this restriction is absent. This proposal seeks to harmonize the situation between public libs and public exes. Both should be allowed to consume private libs as dependencies.

Issues to address

To solve this problem, three issues must be addressed:

  • Private libs should work the same way as public libs. Regardless of whether they are consumed from findlib or from a local workspace, the handling of deps should be the same.

  • Private libraries should be hidden from users of the public libs

  • There should be a guarantee that public libs do not leak the types of private libraries in their interface.

Proposed solution

tl;dr is we'll install the private libs alongside the public libs in findlib and differentiate the two with a naming convention. Private libs will be guarded from downstream users by hiding the cmi's. To guard private types escaping into public interfaces, we won't allow the user to compile the public interfaces against the private lib's cmi's.

The first issue is really an issue when the public libs (with private deps) are installed as findlib libs. The private libs must also be installed alongside. This can be accomplished with some naming convention for private libs. Note that we cannot just "inline" the object files of a private lib with the public lib because multiple public libs can share the same private libs. I understand this is a huge hack, though hopefully real namespaces would allow for a simpler solution.

The second issue is also resolved by not installing the .cmi's of the private libs. This is the same mechanism that we'll use for implementing private modules.

The last issue is resolved by making sure that the interfaces of public libs will not be compiled against the interfaces of private libs. Hence, we'll exclude paths of the .cmi's of private deps when building the .cmi's of the public libs. This should make sure that no types defined in a private lib can end up in the interface of a public lib.

Possible issues

There's a bit of a complication with private modules. Private modules should really be able to export the types of private libraries.

@rgrinberg rgrinberg changed the title Idea/Proposal: Allow for public libraries to depend on public libraries Idea/Proposal: Allow for public libraries to depend on private libraries Jul 16, 2018
@nojb
Copy link
Collaborator

nojb commented Aug 14, 2018

There should be a guarantee that public libs do not leak the types of private libraries in their interface.

Isn't this too strong? In my opinion, "private" should mean ".cmi not installed". This still allows to expose types from private libraries/modules in the public interface. In our experience it is useful to do this. Let me explain what I mean in a toy explample.

Suppose priv.ml contains

type t = int
let x : t = 1

pub.mli contains

val x : Priv.t
val f : Priv.t -> unit

and pub.ml

let x = Priv.x
let f = print_int

Then we can compile modules in our "library" with

ocamlc -c priv.ml pub.mli pub.ml

but afterwards we get rid of priv.cmi:

rm priv.cmi

Now suppose we have main.ml

let () = Pub.f Pub.x

Then we can still compile

ocamlc priv.cmo pub.cmo main.ml

but in this compilation Priv.t is abstract. Note that even if you cannot write Priv.t in Main (because the .cmi is not available), you could have re-exported it from Pub by adding type t = Priv.t in pub.mli and then you could use the abstract manifest type Pub.t = Priv.t in Main.

At LexiFi we ship a plugin editor that allows to write OCaml plugins against a public OCaml API and we enforce this by using the above method. Not being able to expose "private" types as abstract would be too restrictive.

So, in a nutshell, I think not requiring that types from private libraries not be exposed in public libraries will simplify implementation and make it more powerful.

@rgrinberg
Copy link
Member Author

At LexiFi we ship a plugin editor that allows to write OCaml plugins against a public OCaml API and we enforce this by using the above method. Not being able to expose "private" types as abstract would be too restrictive.

I see. I would have thought that explicitly re-exporting the private types as abstract in the public library would be the recommended practice here, but maybe this is unwelcome boilerplate.

I still think that having a mode that prevents you from leaking private types is useful, but I tend to agree with you that it's probably an unnecessary restriction.

@ghost
Copy link

ghost commented Aug 14, 2018

Note that it would also be hard to enforce this property.

@rgrinberg
Copy link
Member Author

Why is it hard? Isn't it just a matter of not including the private lib's cmi's when compiling public libs mlis? The only annoyance is that the error messages aren't going to be so good.

@ghost
Copy link

ghost commented Aug 15, 2018

That's true, I was thinking of enforcing it afterwards. FTR, when I used private modules in the past, I was also referring to types from private modules in public mli files.

@jaredly
Copy link

jaredly commented Sep 25, 2018

tbh I think it makes sense to require that types from private libraries be re-exported

@rgrinberg rgrinberg added proposal RFC's that are awaiting discussion to be accepted or rejected accepted accepted proposals labels Oct 22, 2018
@bobot
Copy link
Collaborator

bobot commented Oct 23, 2018

Which public libraries could depend on a private libraries?

I believe only the one of the same package because compilation and installation of different package could be separated with -p. Or library of the same scope if some combination of -p are mandatory.

@rgrinberg
Copy link
Member Author

I see, that complicated things a little. Perhaps for a v1, we'll restrict private library to be accessible only within a single package.

Note that the plan was to always install the private libraries along with the public libraries. So, the private library would always be present either in the source or as an installed artifact. The only problem separate installation brings is that when a public library specifies a private library as a dependency. The name should be specified in a way that can't be overriden by a public library with the same name.

@SaswatPadhi
Copy link
Contributor

Hello,

For our project, we are facing this issue -- we are creating subdirectories with dune files without a public_name for the library, so those are sort of like private libraries. We would like to use these private libraries in the dune file in the parent directory which builds a public library.
Does the current version of dune provide another way of achieving similar functionality?

Thanks.

@Lupus
Copy link

Lupus commented Mar 5, 2020

Would be awesome to have such feature supported by dune! We currently have some libraries that are built from a bunch of sub-libraries. Naively naming private sub-libraries something like "lib" we later ran into issues that "lib" ends up being top level library when original library is linked in.

It's sometimes convenient to have unwrapped library, for example to bundle all atdgen specs, link this lib and have all of atdgen produced modules available, but as this lib is used by public library, it also has to be public, and it pollutes the namespace with those atdgen modules.

We'll have to mangle private library names ourselves manually to somewhat hide them, and have atdgen bundle lib be wrapped. Would be cool if dune could do that for us.

@yawaramin
Copy link
Contributor

@Lupus interesting that you ran into this ... I did too recently and sent a doc PR #3111 to warn about it. I think the Dune folks want to add a config option to solve the nested libs issue.

@rgrinberg
Copy link
Member Author

but as this lib is used by public library, it also has to be public, and it pollutes the namespace with those atdgen modules.

Note that you will still need to attach this library to a package. Otherwise you have two libraries in different packages that use the private library. In this case, which package should install the private lib?

@ghost
Copy link

ghost commented Mar 5, 2020

BTW, from the other discussion and this one, something is not clear to me. Are you interested in:

  1. the ability to have a public library depend on a private library
  2. the ability to have a library be split across several directories, with the directory names reflected in the module system. i.e. foo/bar.ml seen as Bar in foo/x.ml and as Foo.Bar in y.ml

?

@Lupus
Copy link

Lupus commented Mar 5, 2020

Note that you will still need to attach this library to a package. Otherwise you have two libraries in different packages that use the private library. In this case, which package should install the private lib?

That's not a problem, all of the to-be-private sub-libraries are in one package. And in general it makes sense that public library from a package can depend only on private libraries from the same package.

the ability to have a library be split across several directories, with the directory names reflected in the module system. i.e. foo/bar.ml seen as Bar in foo/x.ml and as Foo.Bar in y.ml

The above was actually my goal. Somewhere in dune documentation (if I'm not mistaken) there was a recommendation to make more libraries to organize the codebase as creating those is cheap. It indeed helps in organizing large collections of files into clusters by their relevance. Having the ability to split library into a bunch of subfolders such that hierarchy gets reflected in module paths also solves this, especially if top level folder could provide a .mli/.rei for the whole library to filter out what's not supposed to be public.

@rgrinberg
Copy link
Member Author

rgrinberg commented Mar 5, 2020 via email

@Lupus
Copy link

Lupus commented Mar 5, 2020

Just checked #1084, looks awesome! Given that module organization within one library can be solved with that, just curious, what would be the use case for depending on a private library in a public one? Share some code between executable and public library within one project?

@rgrinberg
Copy link
Member Author

rgrinberg commented Mar 5, 2020 via email

@emillon
Copy link
Collaborator

emillon commented Jun 29, 2020

I was about to write a proposal for this, but I had forgotten about this issue... Anyway here's my suggestion:

Sub-libraries that start with an underscore character, such as (public_name a._b) get special treatment by dune:

  • they can only appear in the (libraries) field of a, its sub-libraries (public or private), or private executables.
  • its cmi files are not installed

I believe that this is enough for many use cases, such as wanting to test internals of a library and defining a public API at the same time.

If this works, we can change how findlib deals with this as well, so that for example ocamlfind list does not show these private sublibraries.

@rgrinberg
Copy link
Member Author

rgrinberg commented Jun 29, 2020 via email

@ghost
Copy link

ghost commented Jun 30, 2020

@rgrinberg, with this proposal, what happens in this case:

  • public library p1.a depends on private library x
  • public library p1.b depends on private library x
  • executable prog that is part of another p2 package depends on both p1.a and p1.b

How do we ensure that x is linked only once in prog?

@rgrinberg
Copy link
Member Author

My proposal still requires we specify the package of x. So x will be installed as part of that package.

@ghost
Copy link

ghost commented Jun 30, 2020

Indeed. In which directory would such a private library be installed?

@rgrinberg
Copy link
Member Author

Indeed. In which directory would such a private library be installed?

We could use a naming convention like suggester earlier. A private library foo in a package bar would be foo._bar. I'm not even sure if we need the obfuscation even.

@ghost
Copy link

ghost commented Jul 1, 2020

I see, so it's really just a different presentation. Yes, that makes sense to me and I indeed like it better, this way the user doesn't have to come up with a public name.

Regarding the obfuscation, if we don't do it we might have clashes. For instance If a public library is already called foo.bar. We also don't want such libraries to appear in the listing when doing ocamlfind list or dune installed-libraries.

@rgrinberg
Copy link
Member Author

rgrinberg commented Jul 1, 2020 via email

@Lupus
Copy link

Lupus commented Jul 2, 2020

Regarding the obfuscation, if we don't do it we might have clashes.

+1 for obfuscation! We tend to have atd specifications piled into separate library because dune flags are different for those (does not work with -open Base), and just name those libs atd_specs. When some public library depends on such "sub-library", we have to manually prefix the private name with public name so that it does not clash with atd_specs from another library... That results in a bunch of ugly module opens/aliases on top like module Atd_specs = My_public_lib__atd_specs.

@ghost
Copy link

ghost commented Jul 2, 2020

Another possible naming convention is to reserve the foo.private prefix for private libraries. So, foo.private.* would all be private libraries. That seems a little more obvious to me.

What about foo.__private__? We are pretty sure this won't clash with other names and it seems even more private.

@Lupus we were not talking about this kind of obfuscation here, though we could imagine it. In any case, it seems that what you really want is the ability to specify flags per file, which is another feature we have been talking about adding and should happen at some point.

@Lupus
Copy link

Lupus commented Jul 2, 2020

Oh, sorry for confusion. Flags per file sound nice, in tandem with multi-folder libraries probably should be sufficient to structure large library codebases without splitting in separate libs due to artificial reasons like compilation flags.

@ghost
Copy link

ghost commented Jul 2, 2020

Just to sum up, the proposal becomes:

the user writes:

(library
 (name foo)
 (package p))

and it gets installed as p.__private__.foo without the cmi files. @emillon what do you think?

@emillon
Copy link
Collaborator

emillon commented Jul 3, 2020

That seems good, yes.

I'm not 100% sure about the __private__ since the name is already overloaded with different terms and it'd be nice to have something that can be easily searched for if somebody discovers this kind of libraries on their system. But the semantics are in line with what I was suggesting, yes!

@rgrinberg rgrinberg added this to the 2.8 milestone Aug 7, 2020
@rgrinberg
Copy link
Member Author

Ready in master :)

rgrinberg added a commit to rgrinberg/opam-repository that referenced this issue Jan 13, 2021
…ne-action-plugin, dune-private-libs and dune-glob (2.8.0)

CHANGES:

- `dune rules` accepts aliases and other non-path rules (ocaml/dune#4063, @mrmr1993)

- Action `(diff reference test_result)` now accept `reference` to be absent and
  in that case consider that the reference is empty. Then running `dune promote`
  will create the reference file. (ocaml/dune#3795, @bobot)

- Ignore special files (BLK, CHR, FIFO, SOCKET), (ocaml/dune#3570, fixes ocaml/dune#3124, ocaml/dune#3546,
  @ejgallego)

- Experimental: Simplify loading of additional files (data or code) at runtime
  in programs by introducing specific installation sites. In particular it allow
  to define plugins to be installed in these sites. (ocaml/dune#3104, ocaml/dune#3794, fixes ocaml/dune#1185,
  @bobot)

- Move all temporary files created by dune to run actions to a single directory
  and make sure that actions executed by dune also use this directory by setting
  `TMPDIR` (or `TEMP` on Windows). (ocaml/dune#3691, fixes ocaml/dune#3422, @rgrinberg)

- Fix bootstrap script with custom configuration. (ocaml/dune#3757, fixes ocaml/dune#3774, @marsam)

- Add the `executable` field to `inline_tests` to customize the compilation
  flags of the test runner executable (ocaml/dune#3747, fixes ocaml/dune#3679, @lubegasimon)

- Add `(enabled_if ...)` to `(copy_files ...)` (ocaml/dune#3756, @nojb)

- Make sure Dune cleans up the status line before exiting (ocaml/dune#3767,
  fixes ocaml/dune#3737, @alan-j-hu)

- Add `{gitlab,bitbucket}` as options for defining project sources with `source`
  stanza `(source (<host> user/repo))` in the `dune-project` file.  (ocaml/dune#3813,
  @rgrinberg)

- Fix generation of `META` and `dune-package` files when some targets (byte,
  native, dynlink) are disabled. Previously, dune would generate all archives
  for regardless of settings. (ocaml/dune#3829, ocaml/dune#4041, @rgrinberg)

- Do not run ocamldep to for single module executables & libraries. The
  dependency graph for such artifacts is trivial (ocaml/dune#3847, @rgrinberg)

- Fix cram tests inside vendored directories not being interpreted correctly.
  (ocaml/dune#3860, fixes ocaml/dune#3843, @rgrinberg)

- Add `package` field to private libraries. This allows such libraries to be
  installed and to be usable by other public libraries in the same project
  (ocaml/dune#3655, fixes ocaml/dune#1017, @rgrinberg)

- Fix the `%{make}` variable on Windows by only checking for a `gmake` binary
  on UNIX-like systems as a unrelated `gmake` binary might exist on Windows.
  (ocaml/dune#3853, @kit-ty-kate)

- Fix `$ dune install` modifying the build directory. This made the build
  directory unusable when `$ sudo dune install` modified permissions. (fix
  ocaml/dune#3857, @rgrinberg)

- Fix handling of aliases given on the command line (using the `@` and `@@`
  syntax) so as to correctly handle relative paths. (ocaml/dune#3874, fixes ocaml/dune#3850, @nojb)

- Allow link time code generation to be used in preprocessing executable. This
  makes it possible to use the build info module inside the preprocessor.
  (ocaml/dune#3848, fix ocaml/dune#3848, @rgrinberg)

- Correctly call `git ls-tree` so unicode files are not quoted, this fixes
  problems with `dune subst` in the presence of unicode files. Fixes ocaml/dune#3219
  (ocaml/dune#3879, @ejgallego)

- `dune subst` now accepts common command-line arguments such as
  `--debug-backtraces` (ocaml/dune#3878, @ejgallego)

- `dune describe` now also includes information about executables in addition to
  that of libraries. (ocaml/dune#3892, ocaml/dune#3895, @nojb)

- instrumentation backends can now receive arguments via `(instrumentation
  (backend <name> <args>))`. (ocaml/dune#3906, ocaml/dune#3932, @nojb)

- Tweak auto-formatting of `dune` files to improve readability. (ocaml/dune#3928, @nojb)

- Add a switch argument to opam when context is not default. (ocaml/dune#3951, @tmattio)

- Avoid pager when running `$ git diff` (ocaml/dune#3912, @AltGr)

- Add `(root_module ..)` field to libraries & executables. This makes it
  possible to use library dependencies shadowed by local modules (ocaml/dune#3825,
  @rgrinberg)

- Allow `(formatting ...)` field in `(env ...)` stanza to set per-directory
  formatting specification. (ocaml/dune#3942, @nojb)

- [coq] In `coq.theory`, `:standard` for the `flags` field now uses the
  flags set in `env` profile flags (ocaml/dune#3931 , @ejgallego @rgrinberg)

- [coq] Add `-q` flag to `:standard` `coqc` flags , fixes ocaml/dune#3924, (ocaml/dune#3931 , @ejgallego)

- Add support for Coq's native compute compilation mode (@ejgallego, ocaml/dune#3210)

- Add a `SUFFIX` directive in `.merlin` files for each dialect with no
  preprocessing, to let merlin know of additional file extensions (ocaml/dune#3977,
  @vouillon)

- Stop promoting `.merlin` files. Write per-stanza Merlin configurations in
  binary form. Add a new subcommand `dune ocaml-merlin` that Merlin can use to
  query the configuration files. The `allow_approximate_merlin` option is now
  useless and deprecated. Dune now conflicts with `merlin < 3.4.0` and
  `ocaml-lsp-server < 1.3.0` (ocaml/dune#3554, @voodoos)

- Configurator: fix a bug introduced in 2.6.0 where the configurator V1 API
  doesn't work at all when used outside of dune. (ocaml/dune#4046, @aalekseyev)

- Fix `libexec` and `libexec-private` variables. In cross-compilation settings,
  they now point to the file in the host context. (ocaml/dune#4058, fixes ocaml/dune#4057,
  @TheLortex)

- When running `$ dune subst`, use project metadata as a fallback when package
  metadata is missing. We also generate a warning when `(name ..)` is missing in
  `dune-project` files to avoid failures in production builds.

- Remove support for passing `-nodynlink` for executables. It was bypassed in
  most cases and not correct in other cases in particular on arm32.
  (ocaml/dune#4085, fixes ocaml/dune#4069, fixes ocaml/dune#2527, @emillon)

- Generate archive rules compatible with 4.12. Dune longer attempt to generate
  an archive file if it's unnecessary (ocaml/dune#3973, fixes ocaml/dune#3766, @rgrinberg)

- Fix generated Merlin configurations when multiple preprocessors are defined
  for different modules in the same folder. (ocaml/dune#4092, fixes ocaml/dune#2596, ocaml/dune#1212 and
  ocaml/dune#3409, @voodoos)

- Add the option `use_standard_c_and_cxx_flags` to `dune-project` that 1.
  disables the unconditional use of the `ocamlc_cflags` and `ocamlc_cppflags`
  from `ocamlc -config` in C compiler calls, these flags will be present in the
  `:standard` set instead; and 2. enables the detection of the C compiler family
  and populates the `:standard` set of flags with common default values when
  building CXX stubs. (ocaml/dune#3875, ocaml/dune#3802, fix ocaml/dune#3718 and ocaml/dune#3528, @voodoos)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted accepted proposals proposal RFC's that are awaiting discussion to be accepted or rejected
Projects
None yet
8 participants