Dub support in Meson is broken

[rtbo]

Change #2589 has broken Meson support for Dub packages.
Meson will not try to build Dub packages directly but rely instead on the Dub cache to find a package built by Dub.
The way it is done is to find the relevant info in the cache dir name.
For example library-debug-linux.posix-x86_64-ldc_2081-EF934983A3319F8F8FF2F0E107A363BA.
Now in the new shorter cache dir name, it is not possible anymore.

If short names are that important, can I suggest that the dir names is composed of two hashes:

1. one hash that contains infos meson needs and that can be reliably computed by meson as well
   a. compiler
   b. compiler version
   c. arch
   d. platform
2. a second hash for all info dub needs and meson has no clue about

E.g.

library-debug-bUZ3sLErP9iQlvgdxNNYQg

would become

library-debug-xXxXx-yYyYyYyYyYyYyYyYyYyYyY
              ~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
                1            2

5 chars in base64 is more than a billion possibilities. Even if not perfectly distributed, it is more than enough to prevent accidental collisions.

[WebFreak001]

how is this important for meson? The data is still in the build hash, it's just not written out for humans anymore. You can still check if filenames match exactly to see if things need to change. (which dub checks)

Since the information for each build tool will vary between tools and maybe even projects, you probably more likely want to store the information how things were built somewhere on your own, every time when doing a build, and associate that with the cache file name.

The cache would for example also need to depend on things like dflags that change linker names, etc., which were never exposed before.

[rtbo]

Maybe I'm missing a point but I don't see how to associate the Dub's build hash to a specific build configuration.

By design, Meson will not call dub build, as there is a strong policy to not make any change outside the build directory.

When a meson recipe expresses a dependency to a dub package, Meson must find a compatible library already built by Dub.
Compatibility requirements are:

• same architecture
• same platform
• same compiler and compiler version
• same configuration
• if the same build type is available, we use, otherwise we issue a warning

As Meson scans Dub's build cache, it must be able to check for these compatiblity requirements in order to pick the right build artifact to link to.

Now if dub describe can report the location of the artifact in the cache, that would also solve the problem, but IFAIK it doesn't.

[rikkimax]

What we could do relatively easily, is write a json file that contains this information inside the individual cache directory to allow final narrowing down by other package managers.

That'll get you to the end goal, however, anything else would be an initial filter to cut down the files that need processing (and hence improving performance).

[WebFreak001]

making dub describe emit the build path is much more what we would like to have in DUB I think. However since you want to ensure it's built, you most likely just want to call dub build and dub will trigger a build if needed, otherwise not doing anything except outputting where it's built.

[eli-schwartz]

By design, Meson will not call dub build, as there is a strong policy to not make any change outside the build directory.

Meson developer here. Just want to confirm this. Meson is not going to "trigger a build if needed", because:

• meson will not invoke processes that write files outside of its own build artifacts directory, with the exception of things like ccache that is an optional drop-in wrapper for the real compiler plus only writes additional copies for caching purposes while still producing files where meson tells it to. All build artifacts must be under meson's control
• meson will not compile anything during configuration, only during the build stage, and must know during configuration, where the outputs will be at build time

(I am not a D user. I know that meson's current usage of D doesn't violate this, but I have no way of knowing whether it is optimal or could be done better. @rtbo has contributed quite a bit to meson's D support, though, so I assume it is accurate that meson doesn't have better options than whst was broken by this filename change.)

Users can and do inject their own prebuilt dependencies for a variety of languages, including for the D language. Meson aims to be able to recognize those prebuilt dependencies and link to them.

There are undoubtedly tools that build multiple packages in turn, some with dub build and some with meson. Like Linux package managers, or like Conan/vcpkg.

[WebFreak001]

we can introduce a short hash like rtbo has suggested, but note that it may change in the future, especially if we start to add more things that affect the build output (filtering dflags for related stuff, etc.) and is not gonna be able to be generated manually / outside dub due to being an implementation detail.

@rtbo how would you use the short hash in meson? would an opaque string like this that may change between dub versions be fine for you?

Currently I think if we want a full hash that reliably gives build information this would contain things like:

• target triple (arch, os, abi)
• compiler
• compiler version
• enabled previews / reverts (dips)
• flags affecting linker names (shared/static, defaultlib, lto I think?)
• compiler-specific optimization flags

maybe it would make sense just hashing the entire compiler CLI flags?

[rtbo]

The purpose of the short hash is to encode only the few identifiers known by Meson to locate a build.
Meson will also compute the hash with the same algorithm that must not change across versions.
The spec of the algorithm should Dub source code with a comment PLEASE DON'T BREAK ME !
For all settings that will evolve across versions, Dub can make the longer hash evolve (see the initial comment).

But after second thoughts, I'm more in favor of @rikkimax proposal.
Dub issuing a Json description of the cache that can be read by 3rd party software is a much better solution.
It would allow to list the builds available in the cache with path to the built artifact and the associated build settings.

[ljmf00]

Dub is fundamentally broken in meson for various reasons, and the lack of pkg-config makes it harder to integrate not only on meson but on every decent build system.

[eli-schwartz]

So if I understand correctly the original implementation of json descriptions is probably not going to be the final released dub interface, and this ticket is actually going to be closed by #2644 instead?

[rtbo]

Very likely so yes. I'm preparing also the change corresponding to #2644 in Meson.