Remove the external-lib-deps command

[ghost]

As previously discussed, this PR removes the external-lib-deps subcommand.

The user experience is slightly less good as we no longer print a hint suggesting to run dune external-lib-deps --missing, however as explained in #4297 it would be possible to recover a similarly useful hint at a smaller cost.

We are currently using this command inside Jane Street to generate opam files for the packages we publish on github. So to cope with this removal, we implemented a small tool doing a loose analysis of dune files to collect such library dependencies. It works pretty well in practice. We could share it if it would be useful for someone else.

[samoht]

Have you looked at https://github.com/ocurrent/opam-dune-lint ? That tools works the other way round (e.g. it inspects the installed opam packages to find where to the libraries comes from -- so it won't help to replace the external-lib-deps command) but maybe it might be useful somehow...

[snowleopard]

Looks good to me! The diff is quite a bit bigger than I expected -- it's great to lose so much complexity.

[snowleopard]

@samoht Looks like that tool actually uses external-lib-deps under the hood.

[rgrinberg]

That tools works the other way round (e.g. it inspects the installed opam packages to find where to the libraries comes from -- so it won't help to replace the external-lib-deps command) but maybe it might be useful somehow...

Yeah, this is going in the right direction. However, I think that we need to completely move away from the idea that package dependencies can be inferred from library dependencies. First of all, it's completely broken in the presence of executables, and second of all, we can do much better if we go in the other direction and make sure that when a user specifies library dependencies, they respect the already specified package deps. Finally, the user needs to write package deps to specify complex constraints anyways.

[ghost]

Indeed. I'm going to merge this now as this will make our work on the core of Dune easier, and we have until August to think about alternative solutions if this removal is an issue.

/cc @talex5 who maintains opam-lint.

[avsm]

The features in opam-dune-lint might make sense to directly integrate into dune itself, as that's the observable feature that is useful to developers rather than external-lib-deps. We could extend the dune-project format to specifically encode the package->library mapping somehow to address @rgrinberg's point. That mapping database really needs to exist somewhere.

[talex5]

The main problem I see is that dune would need to be able to parse opam files:

• Read *.opam files, for projects not generating them with dune.
• Read *.changes files to discover the mapping from packages to libraries.

So this would mean having dune depend on opam, which I guess we don't want?

[ghost]

Dune already reads opam file; we vendor opam-file-format. What are *.changes files?

[talex5]

The changes files record which package installed which files. opam-dune-lint uses them to find out which package installed each library's META file. e.g.

$ grep META ~/.opam/4.11.1/.opam-switch/install/ocurl.changes 
  "lib/curl/META" {"F:S334T1614952102.19"}

This shows that the ocurl opam package installed the curl ocamlfind library.

It's probably not a supported API.

[ghost]

Oh I see. Well, instead of giving you the external library dependencies, dune could give you the whole set of external dependencies that were accessed during the build. With that info and the changes files, you could figure out all the packages that were used.

However, that would include the transitive dependencies as well. It should be possible to tag direct and transitive dependencies differently, but that would be a bit more involved.

[talex5]

Yes, it has to exclude transitive dependencies. It would also be nice to know which directory each dependency came from, as it shows that in the output, e.g.

ocluster.opam: changes needed:
  "capnp-rpc-lwt" {>= 0.8.0}               [from scheduler, worker]
  "capnp-rpc-net" {>= 0.8.0}               [from scheduler]
  "ppx_sexp_conv" {>= v0.14.1}             [from (ppx)]
  "prometheus" {>= 0.7}                    [from scheduler]
  "alcotest-lwt" {with-test}               [from test] (missing {with-test} annotation)

The rule it's trying to enforce is: every library mentioned in a dune file has the corresponding package mentioned in the corresponding opam file.

[avsm]

Dune would not need to depend on opam to parse *.opam files, only the opam-file-format library which is standalone and could be vendored straightforwardly. What are the .changes files you are referring to, @talex5?

[talex5]

What are the .changes files you are referring to

@avsm: see #4298 (comment)

[ghost]

Should we organise a brainstorming session about this?

[talex5]

Should we organise a brainstorming session about this?

I'm not sure who "we" is referring to, but I'd be interested in any discussion about it.

[ghost]

you, me and maybe Anil. And anyone else interested. On the Jane Street side, we are in a code sprint until the end of the month, but could do early April. Can do Monday the 5th of April at 11am UK time if that works for you?

[talex5]

5th Apr is OK for me, though it is a public holiday, so other people may be busy then.

[ghost]

Indeed, I forgot about that. Probably not good for me either actually. What about Wednesday 7th same time?

[talex5]

OK, Wed 7th is good for me.

[ghost]

It's in the calendar