Port build system to jbuilder + configurator

[rgrinberg]

This has a number of benefits:

• Much faster builds
• Builds just fine on 4.06.0
• Able to build docs using odoc ($ make doc)
• No need to check in generated code.

The only disadvantage here is that we're bringing in some extra build time dependencies from configurator. However, the dependency profile of configurator will be reduced. And of course build time dependencies shouldn't matter to the vast majority of users.

[dmentre]

Hello @rgrinberg . I don't know jbuilder very much. Does jbuilder allow to compile with any version of OCaml compiler? Do we absolutely need jbuilder to build binary from source or is it possible to release a tarball containing cryptokit sources plus whatever is produced by jbuilder to build from source?

@xavierleroy : I don't have much opinion on this pull request. From a SW engineering point of view it adds one or more dependencies which always make the software more complex. But on the other side it is only a build dependency. And jbuilder might be the way to go, I don't follow OCaml ecosystem closely enough to have a meaningful opinion. I haven't tried the patch, let me know if you want me to do some tests.

Best regards,
david

[rgrinberg]

Hello @rgrinberg . I don't know jbuilder very much. Does jbuilder allow to compile with any version of OCaml compiler

Jbuilder officially supports OCaml >= 4.02.3. I'm pretty sure you could use jbuilder to compile against older versions of OCaml, but you'd need a precompiled version of jbuilder. I think it's pretty difficult to support anything older and -safe-string at the same time, so it's a valid lower bound IMO.

Do we absolutely need jbuilder to build binary from source or is it possible to release a tarball containing cryptokit sources plus whatever is produced by jbuilder to build from source?

This isn't possible unfortunately. But note that using jbuilder doesn't increase the number of build tools - it decreases them. Currently, cryptokit needs both ocamlfind and ocamlbuild to build, and jbuilder makes both of those redundant.

It's true that configurator brings in a bigger build time profile, but it's an orthogonal change that can be avoided. I'll simply have to parse ocamlc -config myself. It's not a big deal, but configurator is already used by many C bindings, so I just didn't bother avoiding it. However, if that is what is required to get this PR accepted, I can do that change.

[xavierleroy]

I agree to get rid of Oasis for Cryptokit, but replacing it with jbuilder feels like going from an already complicated system that I don't fully understand to a very complicated system that I don't understand at all. For small libraries such as Cryptokit, what's wrong with a Makefile and a configure shell script?

[rgrinberg]

You can get started with a makefile quite easily, but I've never seen a makefile that covers all the needs of packaging an OCaml library. Here's a non exhaustive list of common omissions:

• Forgetting to make the library buildable on systems without ocamlopt
• Not building shared libraries when dynlink is available
• Non functional builds on Windows due to reliance on Unixms
• Over reliance on Bashisms that breaks compat on other Unix systems
• Omitting the building and installation of cmt's and cmti's which handicaps IDE tools
• Not respecting configuration in ocamlc -config
• Not building documentation (ocamldoc is the minimum, odoc is preferred)
• Incorrect, manually written META files. This is especially true for libraries with configure scripts.
• Being completely hostile to cross compilation (jbuilder packages will get cross compilation for free)
• Not generating .install files for opam. This makes opam rely on hacks like diffing the opam switch dir to find out which files where installed. Also, some people forget to include a proper --prefix option in their configure script.

All in all, you can see that I'm skeptical of just using Makefiles. To the point where I'd even suggest to stick to the oasis status quo which at least addresses some of my points above.

However, while jbuilder itself might not be so simple internally, it does free up quite a bit of complexity when defining the build for a package - just look at how much auto generated crap as been deleted in this PR. So if you'd like to maintain a Makefile based build system as well, I think there's nothing wrong with that. But for installing cryptokit through opam, jbuilder will give users a much better user experience.

Site note: jbuilder is already the most popular build system by a fairly wide margin on opam today. Despite being so much younger than other contenders. Obviously I don't have an unbiased opinion here, and this isn't a popularity contest, but there are many others who are convinced that jbuilder solves many of these problems for them.

[UnixJunkie]

I have tried many build systems for OCaml and it is true that jbuilder is not bad. Plus, at least there is some well funded manpower behinds it. Which is quite good for a component so critical to the whole OCaml user community... Plus, its development looks more open than other things from Janestreet.

[rgrinberg]

Note that the latest version of configurator gets rid of all the ppx build time dependencies. I don't think there's a reason to get rid of configurator anymore.

[avsm]

One thing I've noticed in RWO's builds (that use Cryptokit as an example library) is that:

$ jbuilder build basic_md5.exe
    ocamldep basic_md5.depends.ocamldep-output
      ocamlc basic_md5.{cmi,cmo,cmt}
    ocamlopt basic_md5.{cmx,o}
    ocamlopt basic_md5.exe
ld: warning: directory not found for option '-L/opt/local/lib'

The incorrect /opt/local/lib warning possibly comes from the existing Makefile doing something wrong on OSX.

https://dev.realworldocaml.org/14-command-line-parsing.html#running-basic-commands

[UnixJunkie]

Why was this closed?
jbuilder should be renamed to dune in the issue title.