-
Notifications
You must be signed in to change notification settings - Fork 410
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Idea/Proposal: Allow for public libraries to depend on private libraries #1017
Comments
Isn't this too strong? In my opinion, "private" should mean ".cmi not installed". This still allows to expose types from private libraries/modules in the public interface. In our experience it is useful to do this. Let me explain what I mean in a toy explample. Suppose type t = int
let x : t = 1
val x : Priv.t
val f : Priv.t -> unit and let x = Priv.x
let f = print_int Then we can compile modules in our "library" with
but afterwards we get rid of
Now suppose we have let () = Pub.f Pub.x Then we can still compile
but in this compilation At LexiFi we ship a plugin editor that allows to write OCaml plugins against a public OCaml API and we enforce this by using the above method. Not being able to expose "private" types as abstract would be too restrictive. So, in a nutshell, I think not requiring that types from private libraries not be exposed in public libraries will simplify implementation and make it more powerful. |
I see. I would have thought that explicitly re-exporting the private types as abstract in the public library would be the recommended practice here, but maybe this is unwelcome boilerplate. I still think that having a mode that prevents you from leaking private types is useful, but I tend to agree with you that it's probably an unnecessary restriction. |
Note that it would also be hard to enforce this property. |
Why is it hard? Isn't it just a matter of not including the private lib's cmi's when compiling public libs mlis? The only annoyance is that the error messages aren't going to be so good. |
That's true, I was thinking of enforcing it afterwards. FTR, when I used private modules in the past, I was also referring to types from private modules in public mli files. |
tbh I think it makes sense to require that types from private libraries be re-exported |
Which public libraries could depend on a private libraries? I believe only the one of the same package because compilation and installation of different package could be separated with |
I see, that complicated things a little. Perhaps for a v1, we'll restrict private library to be accessible only within a single package. Note that the plan was to always install the private libraries along with the public libraries. So, the private library would always be present either in the source or as an installed artifact. The only problem separate installation brings is that when a public library specifies a private library as a dependency. The name should be specified in a way that can't be overriden by a public library with the same name. |
Hello, For our project, we are facing this issue -- we are creating subdirectories with Thanks. |
Would be awesome to have such feature supported by dune! We currently have some libraries that are built from a bunch of sub-libraries. Naively naming private sub-libraries something like "lib" we later ran into issues that "lib" ends up being top level library when original library is linked in. It's sometimes convenient to have unwrapped library, for example to bundle all atdgen specs, link this lib and have all of atdgen produced modules available, but as this lib is used by public library, it also has to be public, and it pollutes the namespace with those atdgen modules. We'll have to mangle private library names ourselves manually to somewhat hide them, and have atdgen bundle lib be wrapped. Would be cool if dune could do that for us. |
Note that you will still need to attach this library to a package. Otherwise you have two libraries in different packages that use the private library. In this case, which package should install the private lib? |
BTW, from the other discussion and this one, something is not clear to me. Are you interested in:
? |
That's not a problem, all of the to-be-private sub-libraries are in one package. And in general it makes sense that public library from a package can depend only on private libraries from the same package.
The above was actually my goal. Somewhere in dune documentation (if I'm not mistaken) there was a recommendation to make more libraries to organize the codebase as creating those is cheap. It indeed helps in organizing large collections of files into clusters by their relevance. Having the ability to split library into a bunch of subfolders such that hierarchy gets reflected in module paths also solves this, especially if top level folder could provide a .mli/.rei for the whole library to filter out what's not supposed to be public. |
Check out the (include_subdirs qualified) proposal then. We’re actually not too far from implementing anymore as all the prerequisite refactoring has been done.
…On Mar 5, 2020, 10:49 AM +0000, Konstantin A. Olkhovskiy ***@***.***>, wrote:
> Note that you will still need to attach this library to a package. Otherwise you have two libraries in different packages that use the private library. In this case, which package should install the private lib?
That's not a problem, all of the to-be-private sub-libraries are in one package. And in general it makes sense that public library from a package can depend only on private libraries from the same package.
> the ability to have a library be split across several directories, with the directory names reflected in the module system. i.e. foo/bar.ml seen as Bar in foo/x.ml and as Foo.Bar in y.ml
The above was actually my goal. Somewhere in dune documentation (if I'm not mistaken) there was a recommendation to make more libraries to organize the codebase as creating those is cheap. It indeed helps in organizing large collections of files into clusters by their relevance. Having the ability to split library into a bunch of subfolders such that hierarchy gets reflected in module paths also solves this, especially if top level folder could provide a .mli/.rei for the whole library to filter out what's not supposed to be public.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Just checked #1084, looks awesome! Given that module organization within one library can be solved with that, just curious, what would be the use case for depending on a private library in a public one? Share some code between executable and public library within one project? |
That’s one use case. Another one is when you can partition a library so that they have different dependencies. Think foo and foo_lwt for example. In this case, it’s likely that you’ll want to share some code between these two libraries but maintain them in separate packages.
…On Mar 5, 2020, 11:14 AM +0000, Konstantin A. Olkhovskiy ***@***.***>, wrote:
Just checked #1084, looks awesome! Given that module organization within one library can be solved with that, just curious, what would be the use case for depending on a private library in a public one? Share some code between executable and public library within one project?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
I was about to write a proposal for this, but I had forgotten about this issue... Anyway here's my suggestion: Sub-libraries that start with an underscore character, such as
I believe that this is enough for many use cases, such as wanting to test internals of a library and defining a public API at the same time. If this works, we can change how findlib deals with this as well, so that for example |
Here’s a simpler version of this proposal that was discussed before:
There’s no need to add a public_name for such libraries. They can be referred by their private names and we can specify the package using the package field like we do for tests, executables.
…On Jun 29, 2020, 5:59 AM -0700, Etienne Millon ***@***.***>, wrote:
I was about to write a proposal for this, but I had forgotten about this issue... Anyway here's my suggestion:
Sub-libraries that start with an underscore character, such as (public_name a._b) get special treatment by dune:
• they can only appear in the (libraries) field of a, its sub-libraries (public or private), or private executables.
• its cmi files are not installed
I believe that this is enough for many use cases, such as wanting to test internals of a library and defining a public API at the same time.
If this works, we can change how findlib deals with this as well, so that for example ocamlfind list does not show these private sublibraries.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
@rgrinberg, with this proposal, what happens in this case:
How do we ensure that |
My proposal still requires we specify the package of x. So x will be installed as part of that package. |
Indeed. In which directory would such a private library be installed? |
We could use a naming convention like suggester earlier. A private library |
I see, so it's really just a different presentation. Yes, that makes sense to me and I indeed like it better, this way the user doesn't have to come up with a public name. Regarding the obfuscation, if we don't do it we might have clashes. For instance If a public library is already called |
Regarding the obfuscation, if we don't do it we might have clashes. For instance If a public library is already called foo.bar
That’s right, but I was hoping the underscore would help prevent that. So we’d have foo._bar for the private library. That’s still not bullet proof, but I think we do need a convention of sorts to solve the next problem:
We also don't want such libraries to appear in the listing when doing ocamlfind list or dune installed-libraries.
ocamlfind and dune would both respect the naming convention above.
Another possible naming convention is to reserve the `foo.private` prefix for private libraries. So, `foo.private.*` would all be private libraries. That seems a little more obvious to me.
…On Jul 1, 2020, 1:12 AM -0700, Jérémie Dimino ***@***.***>, wrote:
I see, so it's really just a different presentation. Yes, that makes sense to me and I indeed like it better, this way the user doesn't have to come up with a public name.
Regarding the obfuscation, if we don't do it we might have clashes. For instance If a public library is already called foo.bar. We also don't want such libraries to appear in the listing when doing ocamlfind list or dune installed-libraries.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
+1 for obfuscation! We tend to have atd specifications piled into separate library because dune flags are different for those (does not work with |
What about @Lupus we were not talking about this kind of obfuscation here, though we could imagine it. In any case, it seems that what you really want is the ability to specify flags per file, which is another feature we have been talking about adding and should happen at some point. |
Oh, sorry for confusion. Flags per file sound nice, in tandem with multi-folder libraries probably should be sufficient to structure large library codebases without splitting in separate libs due to artificial reasons like compilation flags. |
Just to sum up, the proposal becomes: the user writes:
and it gets installed as |
That seems good, yes. I'm not 100% sure about the |
Ready in master :) |
…ne-action-plugin, dune-private-libs and dune-glob (2.8.0) CHANGES: - `dune rules` accepts aliases and other non-path rules (ocaml/dune#4063, @mrmr1993) - Action `(diff reference test_result)` now accept `reference` to be absent and in that case consider that the reference is empty. Then running `dune promote` will create the reference file. (ocaml/dune#3795, @bobot) - Ignore special files (BLK, CHR, FIFO, SOCKET), (ocaml/dune#3570, fixes ocaml/dune#3124, ocaml/dune#3546, @ejgallego) - Experimental: Simplify loading of additional files (data or code) at runtime in programs by introducing specific installation sites. In particular it allow to define plugins to be installed in these sites. (ocaml/dune#3104, ocaml/dune#3794, fixes ocaml/dune#1185, @bobot) - Move all temporary files created by dune to run actions to a single directory and make sure that actions executed by dune also use this directory by setting `TMPDIR` (or `TEMP` on Windows). (ocaml/dune#3691, fixes ocaml/dune#3422, @rgrinberg) - Fix bootstrap script with custom configuration. (ocaml/dune#3757, fixes ocaml/dune#3774, @marsam) - Add the `executable` field to `inline_tests` to customize the compilation flags of the test runner executable (ocaml/dune#3747, fixes ocaml/dune#3679, @lubegasimon) - Add `(enabled_if ...)` to `(copy_files ...)` (ocaml/dune#3756, @nojb) - Make sure Dune cleans up the status line before exiting (ocaml/dune#3767, fixes ocaml/dune#3737, @alan-j-hu) - Add `{gitlab,bitbucket}` as options for defining project sources with `source` stanza `(source (<host> user/repo))` in the `dune-project` file. (ocaml/dune#3813, @rgrinberg) - Fix generation of `META` and `dune-package` files when some targets (byte, native, dynlink) are disabled. Previously, dune would generate all archives for regardless of settings. (ocaml/dune#3829, ocaml/dune#4041, @rgrinberg) - Do not run ocamldep to for single module executables & libraries. The dependency graph for such artifacts is trivial (ocaml/dune#3847, @rgrinberg) - Fix cram tests inside vendored directories not being interpreted correctly. (ocaml/dune#3860, fixes ocaml/dune#3843, @rgrinberg) - Add `package` field to private libraries. This allows such libraries to be installed and to be usable by other public libraries in the same project (ocaml/dune#3655, fixes ocaml/dune#1017, @rgrinberg) - Fix the `%{make}` variable on Windows by only checking for a `gmake` binary on UNIX-like systems as a unrelated `gmake` binary might exist on Windows. (ocaml/dune#3853, @kit-ty-kate) - Fix `$ dune install` modifying the build directory. This made the build directory unusable when `$ sudo dune install` modified permissions. (fix ocaml/dune#3857, @rgrinberg) - Fix handling of aliases given on the command line (using the `@` and `@@` syntax) so as to correctly handle relative paths. (ocaml/dune#3874, fixes ocaml/dune#3850, @nojb) - Allow link time code generation to be used in preprocessing executable. This makes it possible to use the build info module inside the preprocessor. (ocaml/dune#3848, fix ocaml/dune#3848, @rgrinberg) - Correctly call `git ls-tree` so unicode files are not quoted, this fixes problems with `dune subst` in the presence of unicode files. Fixes ocaml/dune#3219 (ocaml/dune#3879, @ejgallego) - `dune subst` now accepts common command-line arguments such as `--debug-backtraces` (ocaml/dune#3878, @ejgallego) - `dune describe` now also includes information about executables in addition to that of libraries. (ocaml/dune#3892, ocaml/dune#3895, @nojb) - instrumentation backends can now receive arguments via `(instrumentation (backend <name> <args>))`. (ocaml/dune#3906, ocaml/dune#3932, @nojb) - Tweak auto-formatting of `dune` files to improve readability. (ocaml/dune#3928, @nojb) - Add a switch argument to opam when context is not default. (ocaml/dune#3951, @tmattio) - Avoid pager when running `$ git diff` (ocaml/dune#3912, @AltGr) - Add `(root_module ..)` field to libraries & executables. This makes it possible to use library dependencies shadowed by local modules (ocaml/dune#3825, @rgrinberg) - Allow `(formatting ...)` field in `(env ...)` stanza to set per-directory formatting specification. (ocaml/dune#3942, @nojb) - [coq] In `coq.theory`, `:standard` for the `flags` field now uses the flags set in `env` profile flags (ocaml/dune#3931 , @ejgallego @rgrinberg) - [coq] Add `-q` flag to `:standard` `coqc` flags , fixes ocaml/dune#3924, (ocaml/dune#3931 , @ejgallego) - Add support for Coq's native compute compilation mode (@ejgallego, ocaml/dune#3210) - Add a `SUFFIX` directive in `.merlin` files for each dialect with no preprocessing, to let merlin know of additional file extensions (ocaml/dune#3977, @vouillon) - Stop promoting `.merlin` files. Write per-stanza Merlin configurations in binary form. Add a new subcommand `dune ocaml-merlin` that Merlin can use to query the configuration files. The `allow_approximate_merlin` option is now useless and deprecated. Dune now conflicts with `merlin < 3.4.0` and `ocaml-lsp-server < 1.3.0` (ocaml/dune#3554, @voodoos) - Configurator: fix a bug introduced in 2.6.0 where the configurator V1 API doesn't work at all when used outside of dune. (ocaml/dune#4046, @aalekseyev) - Fix `libexec` and `libexec-private` variables. In cross-compilation settings, they now point to the file in the host context. (ocaml/dune#4058, fixes ocaml/dune#4057, @TheLortex) - When running `$ dune subst`, use project metadata as a fallback when package metadata is missing. We also generate a warning when `(name ..)` is missing in `dune-project` files to avoid failures in production builds. - Remove support for passing `-nodynlink` for executables. It was bypassed in most cases and not correct in other cases in particular on arm32. (ocaml/dune#4085, fixes ocaml/dune#4069, fixes ocaml/dune#2527, @emillon) - Generate archive rules compatible with 4.12. Dune longer attempt to generate an archive file if it's unnecessary (ocaml/dune#3973, fixes ocaml/dune#3766, @rgrinberg) - Fix generated Merlin configurations when multiple preprocessors are defined for different modules in the same folder. (ocaml/dune#4092, fixes ocaml/dune#2596, ocaml/dune#1212 and ocaml/dune#3409, @voodoos) - Add the option `use_standard_c_and_cxx_flags` to `dune-project` that 1. disables the unconditional use of the `ocamlc_cflags` and `ocamlc_cppflags` from `ocamlc -config` in C compiler calls, these flags will be present in the `:standard` set instead; and 2. enables the detection of the C compiler family and populates the `:standard` set of flags with common default values when building CXX stubs. (ocaml/dune#3875, ocaml/dune#3802, fix ocaml/dune#3718 and ocaml/dune#3528, @voodoos)
Abstract
Currently, private libs may depend on public libs but the reverse isn't allowed. Note that an analogous situation exists with executables, but this restriction is absent. This proposal seeks to harmonize the situation between public libs and public exes. Both should be allowed to consume private libs as dependencies.
Issues to address
To solve this problem, three issues must be addressed:
Private libs should work the same way as public libs. Regardless of whether they are consumed from findlib or from a local workspace, the handling of deps should be the same.
Private libraries should be hidden from users of the public libs
There should be a guarantee that public libs do not leak the types of private libraries in their interface.
Proposed solution
tl;dr is we'll install the private libs alongside the public libs in findlib and differentiate the two with a naming convention. Private libs will be guarded from downstream users by hiding the cmi's. To guard private types escaping into public interfaces, we won't allow the user to compile the public interfaces against the private lib's cmi's.
The first issue is really an issue when the public libs (with private deps) are installed as findlib libs. The private libs must also be installed alongside. This can be accomplished with some naming convention for private libs. Note that we cannot just "inline" the object files of a private lib with the public lib because multiple public libs can share the same private libs. I understand this is a huge hack, though hopefully real namespaces would allow for a simpler solution.
The second issue is also resolved by not installing the .cmi's of the private libs. This is the same mechanism that we'll use for implementing private modules.
The last issue is resolved by making sure that the interfaces of public libs will not be compiled against the interfaces of private libs. Hence, we'll exclude paths of the .cmi's of private deps when building the .cmi's of the public libs. This should make sure that no types defined in a private lib can end up in the interface of a public lib.
Possible issues
There's a bit of a complication with private modules. Private modules should really be able to export the types of private libraries.
The text was updated successfully, but these errors were encountered: