Engine: Contexts

[rgrinberg]

This is another idea from a discussion with @snowleopard. I thought I would write it down for the record and to give an opportunity for other curious souls to participate. The end goal of this proposal is to demonstrate why contexts can, and should be implemented outside of the engine, with a plan on how to get there.

Past

Contexts have a long history and have existed since the earliest versions of jbuilder. Therefore it will be helpful to talk about why they were introduced and what were the problems they were meant to solve.

Multi contexts builds were introduced to solve the simple problem of testing dune against multiple versions of the compiler. As such, they were nothing more than parallel builds of the same rules with different configurations, where each configuration was an opam switch. Shortly after, it was realized that they also neatly solved the problem of cross compilation. In a cross compilation setup, the "target" context would delegate the running of all build tools defined in the workspace to the "host" context automatically.

Eventually, a bunch of other things were grafted on. Everything from setting profiles per context, designating a merlin context, setting instrumentation. Fast forward to today, there's demand for customizing ignored directories per context, dune automatically constructing contexts using opam, overriding compiler configuration, and a dozen of other features I might have missed.

Present

The problem with contexts is that it's one of the only features that partially exists in both the rules in the engine. One must understand how contexts work to make sense of the engine, but all the powerful features offered by contexts such as cross compilation are actually provided by the rules. Worst of all, it's not possible to opt out of contexts at all, even though that truthfully, they're a rarely used feature that consumes a lot of memory when used to its full power.

Future

Our goal should be to make the engine context agnostic and have the context implementation live in one place (the rules, or a separate library). To do that, we need to consider the different ways the engine is aware of different contexts today.

Rule Loading

Our rule loading scheme has special context aware implicit rules for copying from the source directory. However, there's no reason why the build configuration callback can't setup the equivalent copying rules.

Promotions, Error Messages

When registering promotions, the engine derives the source path of the file from the build path by stripping the build context from the path. Similarly, when reporting errors, the engine strips the context to translate build paths to source paths to simplify error messages. If the engine were to be context oblivious, we would need to provide a callback to translate build paths to source paths when appropriate.

Directory Targets

Currently, it's not possible for dune to depend on directories that were copied from the source even though it's possible to depend on directories generated by rules. One of the reasons why this is hard to fix is all the special casing of source directories in the rule loading. If this special casing was to be removed, I imagine it would be much simpler to fix this bug.

Progress

• host contexts removed from engine
• knowledge of the install context has been removed
• source rules can be setup in the rules

[snowleopard]

Many thanks for writing this up, Rudi! I'm looking forward to fully untangling the Dune engine from rule definitions, including the contexts.

[rgrinberg]

This proposal is quite aggressive and we're unlikely to get all of it done. However, a much simpler subset of this proposal is to at least get rid of the special _build/install handling in the engine. This is a step in the right direction and will be useful for package management.

[Alizter]

@rgrinberg How do you propose to modify _build/install? Are you going to distribute it per context?

[rgrinberg]

I don't propose to modify it at all, but implement it purely outside of the engine. Rudi.

…

On Aug 23, 2023, 3:56 PM +0100, Ali Caglayan ***@***.***>, wrote: @rgrinberg How do you propose to modify _build/install? Are you going to distribute it per context? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>