Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not run ocamldep on single module buildables #3847

Merged
merged 5 commits into from
Oct 9, 2020

Conversation

rgrinberg
Copy link
Member

For executables & libraries with only a single module, we know the
dependency graph without running ocamldep.

@rgrinberg rgrinberg requested review from nojb, aalekseyev and a user October 3, 2020 19:45
@rgrinberg rgrinberg force-pushed the remove-ocamldep-single-module branch 3 times, most recently from 87870f8 to 9a6f99b Compare October 3, 2020 22:13
Copy link
Collaborator

@nojb nojb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. LGTM.

src/dune_rules/menhir.ml Show resolved Hide resolved
@rgrinberg rgrinberg force-pushed the remove-ocamldep-single-module branch from 9a6f99b to 3d2a02c Compare October 4, 2020 02:31
@rgrinberg
Copy link
Member Author

Let's hear from @aalekseyev if he has experience with such an optimization on the jenga side.

And from @jeremiedimino since he is more familiar with the menhir code.

@aalekseyev
Copy link
Collaborator

I think we never even thought of this optimization, so we don't have any experience.

A more general thing we can do is to never run ocamldep on the main module of the library or executable because we know it's going to be the terminal node in the dependency graph.

Two conceivable problems with this general approach:

  1. Error messages may be different between ocamldep+cycle_detection and compilation.
  2. Maybe calling ocamldep will let you reject some pathological dependency. If doing the compilation skipping ocamldep, you should be prepared to the compiler reading an arbitrary .cmi.

(1) can be confusing to the user because the same error will manifest differently. We should try as best as we can to make them look similar.
(2) is probably relevant for the binaries when multiple binaries are compiled by the same executables stanza (I think we use ocamldep to forbid the binaries' main modules depending on each other). I'm not sure if ocamldep is avoidable here, but maybe.

@ghost
Copy link

ghost commented Oct 5, 2020

FTR, there is currently a custom error message when Dune detects (via ocamldep) that something depends on the main module:

User_error.raise

@rgrinberg rgrinberg force-pushed the remove-ocamldep-single-module branch 2 times, most recently from 804a9a3 to 8e9bfd5 Compare October 5, 2020 17:06
@rgrinberg
Copy link
Member Author

A more general thing we can do is to never run ocamldep on the main module of the library or executable because we know it's going to be the terminal node in the dependency graph.

That seems to be like a nice optimization as well. Do you have special a special check for when other modules depend on the terminal module? E.g.

$ cat dune
(executable (name foo))
$ cat foo.ml
let x = 42
let () = print_int Bar.x
$ cat bar.ml
let x = Foo.x

Error messages may be different between ocamldep+cycle_detection and compilation.

Maybe, but I don't think there's any different in behavior introduced in this PR.

is probably relevant for the binaries when multiple binaries are compiled by the same executables stanza (I think we use ocamldep to forbid the binaries' main modules depending on each other). I'm not sure if ocamldep is avoidable here, but maybe.

This optimization does not apply to such executables stanza. It only applies to an executable stanza if there's a single module in the group.

@rgrinberg
Copy link
Member Author

FTR, there is currently a custom error message when Dune detects (via ocamldep) that something depends on the main module:

I don't think that ever worked for single module libraries and executables. Moreover, it would be annoying if it did. It would fail on some valid programs like:

(executable
 (name foo)
 (libraries foo))

Foo should be visible foo.ml from the foo library.

It's a good think to keep in mind though, so I've added tests for all of this stuff in this PR. They confirm that that this optimization doesn't change the behavior.

@rgrinberg rgrinberg force-pushed the remove-ocamldep-single-module branch from 8e9bfd5 to 88ef9b6 Compare October 8, 2020 16:25
@rgrinberg
Copy link
Member Author

@aalekseyev gentle ping. Did I address your concerns? There's now tests to show that no error messages changed.

@aalekseyev
Copy link
Collaborator

Sorry, I haven't had much time this week.
I haven't read the code, but from what you say it sounds like this PR is benign and I have nothing against it.
I think, given review from @nojb, you should feel free to merge.

For executables & libraries with only a single module, we know the
dependency graph without running ocamldep.

Signed-off-by: Rudi Grinberg <[email protected]>
Use a simpler pipeline for compiling a single file

Signed-off-by: Rudi Grinberg <[email protected]>
@rgrinberg rgrinberg force-pushed the remove-ocamldep-single-module branch from 88ef9b6 to 175af78 Compare October 9, 2020 20:39
@rgrinberg
Copy link
Member Author

OK. Will merge then. Thanks for the feedback.

@rgrinberg rgrinberg merged commit f967df6 into ocaml:master Oct 9, 2020
@rgrinberg rgrinberg deleted the remove-ocamldep-single-module branch October 11, 2020 19:30
rgrinberg added a commit to rgrinberg/opam-repository that referenced this pull request Jan 13, 2021
…ne-action-plugin, dune-private-libs and dune-glob (2.8.0)

CHANGES:

- `dune rules` accepts aliases and other non-path rules (ocaml/dune#4063, @mrmr1993)

- Action `(diff reference test_result)` now accept `reference` to be absent and
  in that case consider that the reference is empty. Then running `dune promote`
  will create the reference file. (ocaml/dune#3795, @bobot)

- Ignore special files (BLK, CHR, FIFO, SOCKET), (ocaml/dune#3570, fixes ocaml/dune#3124, ocaml/dune#3546,
  @ejgallego)

- Experimental: Simplify loading of additional files (data or code) at runtime
  in programs by introducing specific installation sites. In particular it allow
  to define plugins to be installed in these sites. (ocaml/dune#3104, ocaml/dune#3794, fixes ocaml/dune#1185,
  @bobot)

- Move all temporary files created by dune to run actions to a single directory
  and make sure that actions executed by dune also use this directory by setting
  `TMPDIR` (or `TEMP` on Windows). (ocaml/dune#3691, fixes ocaml/dune#3422, @rgrinberg)

- Fix bootstrap script with custom configuration. (ocaml/dune#3757, fixes ocaml/dune#3774, @marsam)

- Add the `executable` field to `inline_tests` to customize the compilation
  flags of the test runner executable (ocaml/dune#3747, fixes ocaml/dune#3679, @lubegasimon)

- Add `(enabled_if ...)` to `(copy_files ...)` (ocaml/dune#3756, @nojb)

- Make sure Dune cleans up the status line before exiting (ocaml/dune#3767,
  fixes ocaml/dune#3737, @alan-j-hu)

- Add `{gitlab,bitbucket}` as options for defining project sources with `source`
  stanza `(source (<host> user/repo))` in the `dune-project` file.  (ocaml/dune#3813,
  @rgrinberg)

- Fix generation of `META` and `dune-package` files when some targets (byte,
  native, dynlink) are disabled. Previously, dune would generate all archives
  for regardless of settings. (ocaml/dune#3829, ocaml/dune#4041, @rgrinberg)

- Do not run ocamldep to for single module executables & libraries. The
  dependency graph for such artifacts is trivial (ocaml/dune#3847, @rgrinberg)

- Fix cram tests inside vendored directories not being interpreted correctly.
  (ocaml/dune#3860, fixes ocaml/dune#3843, @rgrinberg)

- Add `package` field to private libraries. This allows such libraries to be
  installed and to be usable by other public libraries in the same project
  (ocaml/dune#3655, fixes ocaml/dune#1017, @rgrinberg)

- Fix the `%{make}` variable on Windows by only checking for a `gmake` binary
  on UNIX-like systems as a unrelated `gmake` binary might exist on Windows.
  (ocaml/dune#3853, @kit-ty-kate)

- Fix `$ dune install` modifying the build directory. This made the build
  directory unusable when `$ sudo dune install` modified permissions. (fix
  ocaml/dune#3857, @rgrinberg)

- Fix handling of aliases given on the command line (using the `@` and `@@`
  syntax) so as to correctly handle relative paths. (ocaml/dune#3874, fixes ocaml/dune#3850, @nojb)

- Allow link time code generation to be used in preprocessing executable. This
  makes it possible to use the build info module inside the preprocessor.
  (ocaml/dune#3848, fix ocaml/dune#3848, @rgrinberg)

- Correctly call `git ls-tree` so unicode files are not quoted, this fixes
  problems with `dune subst` in the presence of unicode files. Fixes ocaml/dune#3219
  (ocaml/dune#3879, @ejgallego)

- `dune subst` now accepts common command-line arguments such as
  `--debug-backtraces` (ocaml/dune#3878, @ejgallego)

- `dune describe` now also includes information about executables in addition to
  that of libraries. (ocaml/dune#3892, ocaml/dune#3895, @nojb)

- instrumentation backends can now receive arguments via `(instrumentation
  (backend <name> <args>))`. (ocaml/dune#3906, ocaml/dune#3932, @nojb)

- Tweak auto-formatting of `dune` files to improve readability. (ocaml/dune#3928, @nojb)

- Add a switch argument to opam when context is not default. (ocaml/dune#3951, @tmattio)

- Avoid pager when running `$ git diff` (ocaml/dune#3912, @AltGr)

- Add `(root_module ..)` field to libraries & executables. This makes it
  possible to use library dependencies shadowed by local modules (ocaml/dune#3825,
  @rgrinberg)

- Allow `(formatting ...)` field in `(env ...)` stanza to set per-directory
  formatting specification. (ocaml/dune#3942, @nojb)

- [coq] In `coq.theory`, `:standard` for the `flags` field now uses the
  flags set in `env` profile flags (ocaml/dune#3931 , @ejgallego @rgrinberg)

- [coq] Add `-q` flag to `:standard` `coqc` flags , fixes ocaml/dune#3924, (ocaml/dune#3931 , @ejgallego)

- Add support for Coq's native compute compilation mode (@ejgallego, ocaml/dune#3210)

- Add a `SUFFIX` directive in `.merlin` files for each dialect with no
  preprocessing, to let merlin know of additional file extensions (ocaml/dune#3977,
  @vouillon)

- Stop promoting `.merlin` files. Write per-stanza Merlin configurations in
  binary form. Add a new subcommand `dune ocaml-merlin` that Merlin can use to
  query the configuration files. The `allow_approximate_merlin` option is now
  useless and deprecated. Dune now conflicts with `merlin < 3.4.0` and
  `ocaml-lsp-server < 1.3.0` (ocaml/dune#3554, @voodoos)

- Configurator: fix a bug introduced in 2.6.0 where the configurator V1 API
  doesn't work at all when used outside of dune. (ocaml/dune#4046, @aalekseyev)

- Fix `libexec` and `libexec-private` variables. In cross-compilation settings,
  they now point to the file in the host context. (ocaml/dune#4058, fixes ocaml/dune#4057,
  @TheLortex)

- When running `$ dune subst`, use project metadata as a fallback when package
  metadata is missing. We also generate a warning when `(name ..)` is missing in
  `dune-project` files to avoid failures in production builds.

- Remove support for passing `-nodynlink` for executables. It was bypassed in
  most cases and not correct in other cases in particular on arm32.
  (ocaml/dune#4085, fixes ocaml/dune#4069, fixes ocaml/dune#2527, @emillon)

- Generate archive rules compatible with 4.12. Dune longer attempt to generate
  an archive file if it's unnecessary (ocaml/dune#3973, fixes ocaml/dune#3766, @rgrinberg)

- Fix generated Merlin configurations when multiple preprocessors are defined
  for different modules in the same folder. (ocaml/dune#4092, fixes ocaml/dune#2596, ocaml/dune#1212 and
  ocaml/dune#3409, @voodoos)

- Add the option `use_standard_c_and_cxx_flags` to `dune-project` that 1.
  disables the unconditional use of the `ocamlc_cflags` and `ocamlc_cppflags`
  from `ocamlc -config` in C compiler calls, these flags will be present in the
  `:standard` set instead; and 2. enables the detection of the C compiler family
  and populates the `:standard` set of flags with common default values when
  building CXX stubs. (ocaml/dune#3875, ocaml/dune#3802, fix ocaml/dune#3718 and ocaml/dune#3528, @voodoos)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants