Implementation and purpose of Show-Build-Info

[fendor]

EDIT:

Final implementation

Finally the chosen implementation has been the third ("Dump build info as side effect of Setup.hs build") with the pr #7498

---

Nomenclature (uses recursive name bindings)

Cabal = `lib:Cabal`
cabal = `exe:cabal`
v2-build = nix-style build command for project building (happens in exe:cabal)
Setup.hs build = Cabal build system, e.g. what happens under the hood (happens in lib:Cabal)
v2-show-build-info = nix-style show-build-info command, cli interface for users (happens in exe:cabal)
Setup.hs show-build-info = Cabal show-build-info system, performs actual work (happens in lib:Cabal)

---

The command show-build-info is intended to help tooling based on Cabal to find detailed information on how to build each component individually. It is not intended to expose Cabal details, such as what preprocessors are executed, etc... It also does not subsume Cabal's build process, i.e. having the output of show-build-info does not suffice to build a cabal component in certain situations. For example it is still Cabal's task to generate Paths_*.hs. Therefore, show-build-info only works to compile a project, if Cabal has basically built the project, already.

But then tooling such as hie-bios and hsinspect can use the output of show-build-info to feed them into a GHC session and provide IDE functionality.

Implementation

Extracting compiler information is theoretically not a problem for cabal. Since we parse the .cabal file description and already solve for which dependencies to use, basically all information should be present, right? However, Cabal has been designed with extensibility in mind and achieved that by allowing developers to overwrite certain cycles of a build via custom Setup.hs hooks. These hooks are basically a way for a developer of a Cabal package to overwrite or redefine various build-steps in Cabal.
This means that while exe:cabal knows everything about the package to build, lib:Cabal can change compilation flags when building the project.
Therefore, exe:cabal alone can not provide us with the low-level details on how to compile a component, as only lib:Cabal has the full build information (when using build-type: Custom). Thus, show-build-info must be a part of lib:Cabal, as long as we have build hooks in order to generate the correct build-info reliably.

For the implementation itself, we present three ideas:

Dump build info as side effect in v2-build

Idea: Add an option to exe:cabal v2-build to dump the build information during the build. We would explicitly invoke Setup.hs show-build-info from the v2-build command.

Pros:

• No extra command
• Keeps build-info up-to-date automatically
• Can be easily used by editors without extra invocation

Con:

• Performs more work than necessary
  • build will compile a component, although we might not need to (e.g. build a component of which we only need the ghc options)
  • Could be mitigated by additional flag, such as --only-prebuild
• Maintaining a Cabal command is a lot of work and error-prone.
• Handling build hooks correctly is non-trivial.

Issues:

• What components are available?
  • Solvable via plan.json
    • Doesn't include disabled components right now.
• Which ghc is going to be used?
• How to tell whether build-info is outdated?
  • --dry-run to the rescue! Tells you when a component needs to be rebuilt.

New command, v2-show-build-info, which calls Setup.hs show-build-info

Idea: manually trigger the generation of build information.

Pros:

• Manual control
  • Can be explicitly instructed to give possibly outdated information
• Minimal amount of work possible
  • Don't build a component if you don't have to.
• Extensible
  • New flags might be added, such as --buildinfo-project that prints project global information (quickly).

Con:

• Preprocessing of relevant files might be missing
• Maintaining a cabal command is a lot of work and error-prone
• Handling build hooks correctly is non-trivial.

Issues:

• What components are available?
  • Solvable via plan.json
    • Doesn't include disabled components right now.

Dump build info as side effect of Setup.hs build

Idea: Generate build info whenever Cabal builds a component and put it into the "old-style" builddir of Setup.hs build, i.e. next to setup-config.

Pros:

• Probably the quickest solution as we already parsed LocalBuildInfo/setup-config. Standalone s-b-i command has to redo that work.
• Keeps build-info up-to-date automatically
• Can be easily used by editors without extra invocation
• Best way to avoid missing bugs and corner-cases
  • Handles custom hooks correctly

Con:

• Discovering build-info files is not as trivial as possible.

Issues:

• What components are available?
  • Solvable via plan.json
    • Doesn't include disabled components right now.
• How to tell whether build-info is outdated?
  • --dry-run to the rescue! Tells you when a component needs to be rebuilt.

Implementation details

We can generate the build-info and write it to the build directory of each component (e.g. dist-newstyle/build/.../<package-name>).

After performing a v2-build, exe:cabal writes out build-info.list file to dist-newstyle/cache, containing the list of all build-info files which might exist, i.e. all enabled components. This file can be parsed by external tools to discover all build-infos of the project.

Optionally, we can add a command v2-ide that does:

• Give us the location of each individual build info file
• Give us the build-information for a given filepath/target.
• List all components in the project (enabled and disabled)
• Give us project wide information (such as compiler version) without running the solver (really fast!)

Summary

I discussed show-build-info quite thoroughly with @DanielG, and we concluded that the third approach is probably the cleanest solution.

Current PR: #7478 which basically implements approach 2.

[fendor]

This issue serves as a discussion point that was missing so far.

cc @jneira, @emilypi, @mpickering, @fgaz, @Mikolaj and everyone else who wants to give input!

[jneira]

Each technical doc you write is better than the previous one, congrats!

The discover of the config file is the worse caveat of 1 and 3 i can think of. What happened if the user changes the cabal build dir? We would need just s-b-i to know where s-b-i is placed! When cabal will change the default build dir all clients will have to change as well.

What about doing 1 or 3 and add a flag to the build command to output the content of the file (or at least its location)
Sorry if I am missing something obvious.

[jneira]

Another option could be let the caller decide the location of the generated file: --dump-buildinfo=path and fallback inside the build dir if not set (or create two flags)

[fendor]

I think the latter is a good, lightweight option that we can add to cabal v2-build

[jneira]

@emilypi @Mikolaj @phadej it would be great unblock the feature, choosing one of both open alternatives (dedicated command or v2-build flag), thoughts?

[Mikolaj]

If there is no strong preference either way from the feature's users nor implementors, then an extra flag sounds less intrusive for users not interested in the feature, doesn't it? Such users just see one more line deep down in the help output and the manual (as opposed to a more prominent mention for a command).

Regarding codebase intrusiveness, I guess it's similar, given that the flag needs a counterpart in cabal.project, etc. (unless it's reflected fully automatically; I don't know).

How would the flag interact with cabal install and other commands?

[fendor]

I think the flag --dump-buildinfo=<path> should have no effect on other commands and is therefore only part of BuildFlags.

However, a flag such as --enable-dumpbuildinfo (name subject to bikeshed) should be part of ConfigFlags (Thus part of NixStyleFlags), and interact "as expected" in the sense it generates the build artefacts.

[DanielG]

--enable-dumpbuildinfo

So is the idea for this new flag that this is what controls the generation of buldinfo.json by Setup.hs build and --dump-buildinfo= writes the list of paths for each component's buildinfo.json into the given file?

[fendor]

@DanielG Yes exactly.

Maybe even one step further and combine the results into one build-info.json (e.g. <> all components), since we can (for performance reason) parse the plan.json ourselves and get the location of each build-info.json quite easily.

[fendor]

I add here an overview of the current PRs, their relation, which I care about and which need to be reviewed. They are sorted by relevance, so the first PR is the most important for this issue.

• Remove show-build-info command and generate build-info as a side-effect of 'build' #7498 Implements approach three from the above list quite completely.
  • Main PR, once this is merged we will have a workable s-b-i
• Json bytestring #7477 Cabal Json Lib rewrite
  • Absolutely required for Remove show-build-info command and generate build-info as a side-effect of 'build' #7498
• Add status command to cabal #7500 Adds a user-frontend for plan.json and build-info.json essentially
  • Optional: might be a good idea, might be a terrible idea, reviews and opinions desired. Implemented second part of approach three from above.
• Add compilerLocation to Compiler datatype #7507 Code refactor, maybe improves s-b-i.
  • Optional
• Add AmbiguityResolver to decide how to resolve ambiguity #7482 Adds convenience
  • Optional and not required, extracted from show-build-info (exe:cabal part plus tests) #6241
• Don't output haddock stdout if verbosity is silent #7483 fixes weird haddock behaviour.
  • Completely optional, but might be interesting to merge nevertheless.

Outdated PRs:

• Implement show-build-info for cabal-install #7478 Rebase and improvement of show-build-info (exe:cabal part plus tests) #6241. Basically approach 1 from the above list.
• show-build-info (exe:cabal part plus tests) #6241 Mother of all efforts, only still exists for legacy reasons

cc @emilypi

[fendor]

@tseenshe Sorry for pinging, but I think hsinspect can benefit from approach 3, too.

[emilypi]

@fendor Tzeenshe no longer works in Haskell, and has iceboxed the project until further notice

[masaeedu]

I'm not sure where to post this exactly, but I would greatly appreciate a sensible decomposition of the tasks that cabal performs, so that some sum of parts does in fact equal cabal build. In this part you mention:

It also does not subsume Cabal's build process, i.e. having the output of show-build-info does not suffice to build a cabal component in certain situations. For example it is still Cabal's task to generate Paths_*.hs. Therefore, show-build-info only works to compile a project, if Cabal has basically built the project, already.

I am not looking forward to a cabal show-build-info that does not work correctly unless I've already run cabal build. If there are some prerequisite stages of work, it would be useful if these were be exposed as separate commands, or (lesser of two evils) implicitly carried out in cabal show-build-info.

It seems like it would be useful to describe the possible states that a cabal project can be in with respect to the presence/absence of generated build products/metadata, and put the commands in their proper context as transitions in this state space. This way, we would at least have some specification for an initial state in which it is valid to execute cabal show-build-info, and we would be able to see what side effects if any it introduces.

[fendor]

I am not looking forward to a cabal show-build-info that does not work correctly unless I've already run cabal build

Indeed. That's one reason why we no longer have a show-build-info command, we have a flag for v2-build to generate the build-information: cabal v2-build --enable-build-info.

To be clear, the current implementation follows the third idea from the top-post for show-build-info, which is dumping build-info as a side effect of cabal v2-build.

[jneira]

Maybe this could be closed as #7498 was already merged?

This still needs #7500 to be finished, as stated in the pr

[kokobd]

Not sure if it's appropriate to discuss here, but if a standalone tool calls tryGetPersistBuildConfig, there are problems with packages that have custom Setup.hs.

A custom Setup.hs makes the generated setup-config file using the Cabal comes with itself, not the one baked in cabal-install tool. So, in a multi-package project, this can leads to a situation where some packages have Cabal version A, other packages have Cabal version B. One of them will be different from my standalone tool's Cabal version.

My question is, currently, is it possible to take Setup.hs into account when calling tryGetPersistBuildConfig? If not, is this a use case of --enable-build-info?

[fendor]

Yeah, that sounds like a use-case for --enable-build-info to me.