Per-component interface for Setup

[ezyang]

This ticket details a design for a per-component interface for Setup. Such an interface would fix #3049 (nix-local-build cabal-install would be able to request recompilation of only components that changed, rather than all components), #2802 (you could configure an individual component), #2780 (you would have per-component copy), #2775 (you can just build one component at a time), #2725 (you can configure each component with different dependencies), #2623 (you would parallelize over calls to build components, not packages), #2071 (you would have per-component test) and #1575 (you could just configure one component). I also happen to need it for Backpack.

The existing package interface

The interface for Setup was historically defined by the Cabal specification: https://www.haskell.org/cabal/proposal/pkg-spec.pdf It defined that a Setup script must satisfy the following command line interface:

./Setup configure [flags]
./Setup build
./Setup copy # not in the original spec but what we use today
./Setup register
./Setup clean
./Setup test

These commands are package oriented and, left to their own devices, will build your entire package.

Building components using the existing interface

Over time, we've grown more flags and modes, e.g. cabal build foo which builds just the foo component. Thus, to a certain degree, per-component builds can be simulated in the following way:

• ./Setup configure must still be run once, with ALL components you may be interested in building enabled at this time.
• ./Setup build foo then builds just the foo component
• ./Setup register if you built the library component (because that's the only component that can get registered, pre Convenience libraries #3022 )
• ./Setup copy could be extended to copy just individual components, but it doesn't really support it, see copy command fails when building only some components #2780. Teaching copy to copy individual components should be relatively simple, but nobody has done it.

But there are a few deep assumptions which are a bit harder to deal with in the current world:

• ./Setup configure must configure all the components you want to build in a single dist at once. If you decide to enable another component, Cabal redos the configuration step. If an executable and a test-suite share a dependency, it must be picked consistently between the two.
• ./Setup scripts DO support building to different dist directories, so you might imagine making a separate dist directory per component. But you quickly run into a different problem: many components in a package may depend on the library, which you would like to share between compilations. If you are able to install the library, no problem, but if you are build-rebuilding inplace you need to make sure the separate components can see the inplace library you built in a different dist-directory. (Also, there is no way to not build the library, c.f. Disable library building, only build other components #2775).

Proposal 1: A new interface

In this proposal, we propose to add component-ized modes to all existing commands. Thus, we have the following API:

./Setup configure-component cname [flags]
./Setup build-component cname
./Setup copy-component cname
./Setup register-component cname
./Setup clean-component cname
./Setup test-component cname

./Setup configure-component UID cname is responsible for configuring a build for the component named cname (using the existing convention from cabal build). Every other command takes an explicit cname to specify which component the command should apply to.

Proposal 2: Use build directories to partition component builds

In the previous proposal, you may have noticed that the proposed interface is extremely similar to what we have today, except for the fact that (1) configuration is per component, and (2) some otherwise missing parameters have been added. So another possibility is to just add this functionality piecemeal. The key idea for supporting configuration per component is to just create a separate dist dir for each component. Then the API looks like this:

./Setup configure --component cname --builddir=distdir
./Setup build --builddir=distdir
./Setup copy --builddir=distdir
./Setup register --builddir=distdir
./Setup clean --builddir=distdir
./Setup test --builddir=distdir

In this case, the only new flag is --component, which enables ONLY the component specified. Thus, if you say --component pkg-tests, the resulting configuration will only build the test suite (so the library that it is being built against must be installed in whatever database you are configuring to build against. This is NOT user friendly, but the intent never was for users to use this interface.) To build multiple components from the same package, you simply have to specify distinct dist directories. Essentially, the per-component interface is precisely the existing one, but there is only ever one component enabled.

Changes to cabal-install

In both cases, the intention is to switch cabal-install install plans from operating on packages to operating on components.

Unresolved questions

1. How should multi-component builds be treated inside the Cabal library? At the moment, they are a source of a considerable amount of complexity, and it would be nice if we could simply drop it (leaving the smarts in tools like Stack and cabal-install). But to maintain compatibility with the old Setup interface, this needs to be maintained. Should we try to factor out the multi-component logic into its own set of modules? This makes Proposal 2 a little goofy, since the most straightforward way to implement it is to just reuse the existing multi-component infrastructure. (By the way, if you don't restrict to enabling only one component at a time, then this proposal is essentially Specify components when configuring, not building #2802 )
2. How to deal with data files? At the moment, data files are shared across all components, so you need to install them once for everyone. Probably the correct resolution is to install a copy for each component, and not attempt to share them. (Then we should also add the ability to specify data files differently per component.)
3. How to deal with hooks? In fact, this is already causing people grief (LocalBuildInfo should change at build time when components are selected #2910). The approach I suggested was to introduce a parallel set of per-component hooks.

CC @23Skidoo @dcoutts @hvr @mgsloan @ttuegel @kosmikus @snoyberg

[dcoutts]

It's not at all clear to me that we want to configure each component with different dependencies, or if so, not by default. Having consistent deps within a package I think is a feature not a bug. That said, if we're just making it possible and not doing it by default then that's not so bad.

Re proposal 2, with different dist dirs and only one component at a time enabled. I don't see how that works given that components depend on each other within a package.

[ezyang]

Definitely not by default; the solver can't deal with it. But it is certainly possible that we should configure each package once, and then just call build/copy/register multiple times per component.

Re proposal 2, it would work the same way as dependencies on packages work: the component dependency would be fed into the configure script and must be available in the package database stack. Of course, you need to be able to uniquely identify these components but a ComponentId exactly is adequate for this purpose.

[ezyang]

Discussing with @dcoutts, a more backwards compatible interface would be to just fix #2802 and #2775 and then teach cabal-install to use these flags.