Move /src to /subprojects

[Ericson2314]

Motivation

Get rid of the current symlink soup we have, as part of switching to Meson only.

The plan is to backport these reorgs just to 2.24, so backports of actual changes to the Nix release in Nixpkgs until 25.05 will be easy.

Context

In c7ec336 I had to stop using src as a custom subproject dir, because Meson was getting confused with src being a subdir of the docs dir.

This is a bug on Meson's part, but I think the simplest fix is just for us to do the rename; hence I am reopening this PR.

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[edolstra]

Why not just use "src" as the subprojects directory? Way less churn that way.

[Ericson2314]

@edolstra Per https://mesonbuild.com/Subprojects, the directory must be named subprojects. Sorry.

(They are very opinionated about this, also about there being only one level of subprojects. Basically they want to avoid any need/desire for something like follows / more complicated dependency structures in general, vs seeing subprojects as a bunch of "flat" mix-ins, and as long as they are searched for in dependency order things just work.)

[inclyc]

(They are very opinionated about this,

I don't think we should follow meson's opinion because semantically our /src directory is not "subprojects" at all.

[Ericson2314]

I don't think we should follow meson's opinion because semantically our /src directory is not "subprojects" at all.

But I want to make it be that. It also gives us a way to port things over little by little and have smaller derivations from the get-go, which makes the conversion more productive.

[inclyc]

But I want to make it be that. It also gives us a way to port things over little by little and have smaller derivations from the get-go, which makes the conversion more productive.

Breaking a project into multiple projects with meson does not require moving all stuff into "subprojects". Instead, there is an alternative to achieve this:

Firstly, suppose there are some source files under projA/src/a.cpp, projB/src/b.cpp, and we want to build them separately. And also it may be formed as a monorepo.

Things to do:

1. create a meson.build under projA/src and projB/src. Then write build scripts for it, but do not invoke project()
2. For separated builds, create meson.build files under projA, and projB, with project() invoked.
3. For monorepo (development) builds, create a meson.build under top-level directory + monorepo project() and invoke subdir('projA/src'), subdir('projB/src').

the directory must be named subprojects. Sorry.

After all I think hardcoded subprojects is a funny bad design in meson and we should NOT follow that.

As per my understanding, meson documentation says

Some platforms do not provide a native packaging system. In these cases it is common to bundle all third party libraries in your source tree. This is usually frowned upon because it makes it hard to add these kinds of projects into e.g. those Linux distributions that forbid bundled libraries.

Frankly this project is a "packaging system". Meson authors may designed to use subprojects as an ungly workaround to bundle all dependencies. I think things under /src is not our dependencies, right?
IMHO, we are nix, the most advanced package manager, we should handle things elegantly.

[Ericson2314]

@inclyc So regardless of intent, what I think submodules are really good at as making dev shells for multiple packages.

I really care about Nix Layering: in my view libstore and libexpr are separate packages, never built in the same derivation. If the interface between them becomes table enough, I would even consider moving them into separate repos. (Stuff like Josh which Tvix uses to merge/split repos could help with this.)

On the other hand, with most build systems, it sucks to simultaneously work on upstream and downstream packages. LLVM for example has a huge file of hacks (I've helped maintain, hehe) to do this with CMake. With Meson, however, it works out of the box.

Maybe subprojects were created for the "wrap db", but I don't care. What the give me is incremental-cross project builds, and that's amazing! No more tradeoff between proper layering and development productivity!

See (the second commit of) #10855 for this in action.

[p01arst0rm]

But I want to make it be that. It also gives us a way to port things over little by little and have smaller derivations from the get-go, which makes the conversion more productive.

Breaking a project into multiple projects with meson does not require moving all stuff into "subprojects". Instead, there is an alternative to achieve this:

The motive behind this is to allow out of order includes that are required currently under the Nix code docgen process.

Firstly, suppose there are some source files under projA/src/a.cpp, projB/src/b.cpp, and we want to build them separately. And also it may be formed as a monorepo.

Things to do:

1. create a `meson.build` under `projA/src` and `projB/src`. Then write build scripts for it, but do not invoke `project()`

2. For separated builds, create `meson.build` files under `projA`, and `projB`, with `project()` invoked.

3. For monorepo (development) builds, create a `meson.build` under top-level directory + monorepo `project()` and invoke `subdir('projA/src')`, `subdir('projB/src')`.

This breaks the meson spec.

the directory must be named subprojects. Sorry.
After all I think hardcoded subprojects is a funny bad design in meson and we should NOT follow that.

The subprojects directory is created for a reason to force meson code to not index or modify data inside it with subdir()

As per my understanding, meson documentation says

Some platforms do not provide a native packaging system. In these cases it is common to bundle all third party libraries in your source tree. This is usually frowned upon because it makes it hard to add these kinds of projects into e.g. those Linux distributions that forbid bundled libraries.

Frankly this project is a "packaging system". Meson authors may designed to use subprojects as an ungly workaround to bundle all dependencies. I think things under /src is not our dependencies, right? IMHO, we are nix, the most advanced package manager, we should handle things elegantly.

/src currently includes vendored code, which breaks the spec. it also references files outside of itself out of order, which also breaks the spec.

[p01arst0rm]

@inclyc see mesonbuild/meson#6404

@eli-schwartz may also be able to weigh in

[eli-schwartz]

Meson explicitly allows you to use whatever subprojects directory you wish.

Doing so requires defining the subprojects_dir in your project declaration, and also means you cannot use that directory for non subprojects.

Subprojects aren't just directories of source code. They are sandboxed from each other, and accessing the contents of a subproject needs to be done via the subproject exporting an interface (override find_program, override dependency). Subprojects have their own names. They have their own version numbers. You can build them standalone. You can release them separately and upload release tarballs of just the subproject.

In short, subprojects are in fact their own derivations.

Meson isn't opinionated about whether they are their own derivations because you vendored a third-party project or because you develop both of them in one repository. It is simply opinionated about the fact that if you want to treat them as their own derivations, then they are in fact separate derivations and need to e.g. be sandboxed that way.

This will facilitate breaking up Nix into multiple packages for each component with Meson.

It is a project specific decision whether that project wishes to be able to release its code as multiple derivations.

I don't think we should follow meson's opinion because semantically our /src directory is not "subprojects" at all.

... then you are at odds with the motivation of this PR, which was to make your src directory into multiple derivations.

If parts of nix are split out into their own derivations then they are, per definition, a subproject.

[p01arst0rm]

even if we didnt call it subprojects, it would still be more semantically accurate to call the folder Nix or cppnix, because the directory does not contain only source filles. having directory structures such as /src/libutil/tests/.. /src/libutil/src/.. isnt very clear and leaves the door open to a confusing codebase

[Ericson2314]

Thanks @eli-schwartz, that matches my understanding, except I didn't know about subprojects_dir!

[eli-schwartz]

Overriding the name of the subprojects directory was in fact added as a feature specifically because it allows people with existing code in e.g. a third-party/ directory to avoid the churn of moving files around. :D

This does assume that your choice of subprojects directory does itself consist of e.g. third-party/zlib/ rather than yeeting all your source files directly into third-party/*.c

Or as noted,

Why not just use "src" as the subprojects directory? Way less churn that way.

[Ericson2314]

Closed because #10855 uses src as the subproject directory now, and so this is not needed :).

[eli-schwartz]

The good news is if you ever decide to rename it to cppnix/ after all, you can just move it and all the work you're about to do, just magically still works. :)

Since it's a subproject and looked up by name, you don't even need to update file contents to match the rename (other than changing the subprojects_dir at the top level).

In other words: that decision is something which can be trivially deferred.

[inclyc]

I think it is definitely acceptable to have #10855, just kindly ask @eli-schwartz

Meson isn't opinionated about whether they are their own derivations because you vendored a third-party project or because you develop both of them in one repository. It is simply opinionated about the fact that if you want to treat them as their own derivations, then they are in fact separate derivations and need to e.g. be sandboxed that way.

Does this mean vendored third-party projects code and 'our own codes' but separated derivations, these two kinds of code directory, should be treated as the same thing in meson?

I'm a little confused about this because they are actually different. For the second case (separated derivations), we may have some unstable API/ABI in the project thus it must be built in a monorepo.

But for the first case, (vendored 3rd-party codes) we do expect it have some stable interface (right?)

I do think this have real-world needs. For example in llvm-project there are some separated components, say clang, llvm, ... and llvm's C++ API is not very stable but people do need to build the monorepo in development + build each subdir separately. If someday llvm wants to use meson, clang, llvm, ... all the components should be moved to subproject_dir cluttered with zlib-like 3rd-deps, this is rather bad.

Is there some approach to clearly keep the semantics? Not just specifying one subproject_dir?

[eli-schwartz]

• a subproject can also have its own subprojects (meson will process the subprojects on a first-come first-served basis to avoid diamond dependencies)
• a subproject can be a small INI file called zlib.wrap, which means the zlib source code isn't actually checked into your codebase but is downloaded on the fly, on demand, when zlib isn't available as a system package (or derivation, if you like). (The environment $MESON_PACKAGE_CACHE_DIR can be used to provide offline tarball sources, to avoid breaking a network sandbox.) This means it doesn't clutter things up by default.

[inclyc]

This means it doesn't clutter things up by default.

Could you elaborate how to handle llvm structure(https://github.com/llvm/llvm-project) i.e.

clang, llvm, as subprojects and llvm depedency, zlib?

As per my understanding currently we must move all sources into subproject_dir, right?

Is there some approach to clearly keep the semantics?

To clarify, my question is how to put things we want to build separately + with a monorepo style for ABI reason and things vendored from 3rd party in different dirs.

[fricklerhandwerk]

Tracking #2503

[p01arst0rm]

This means it doesn't clutter things up by default.

Could you elaborate how to handle llvm structure(https://github.com/llvm/llvm-project) i.e.

clang, llvm, as subprojects and llvm depedency, zlib?

As per my understanding currently we must move all sources into subproject_dir, right?

Is there some approach to clearly keep the semantics?

To clarify, my question is how to put things we want to build separately + with a monorepo style for ABI reason and things vendored from 3rd party in different dirs.

no, we technically dont have to put all code into the subproject dir, but we should put code relating to each library into the subdir related to that library. the end goal is to remove all vendored code anyway :),

[Ericson2314]

In c7ec336 I had to stop using src as a custom subproject dir, because Meson was getting confused with src being a subdir of the docs dir.

This is a bug on Meson's part, but I think the simplest fix is just for us to do the rename; hence I am reopening this PR.

[edolstra]

I really don't want to move everything around just to placate the build system.

Apart from invalidating 20 years of muscle memory about where files reside, this makes every patch/backport harder, probably causes issues for numerous existing PRs, etc. Let's just not.

[Ericson2314]

We have an alternative rename we agreed upon to work around the Meson bug

[Ericson2314]

mesonbuild/meson#13774 is the meson bug btw