[proposal] Alternative build-packaging approach, less duplication and less manual work

[ezaiakinsc]

Everything described here - is not a theory, it is a description of actual solution (we are calling it Native Libraries Orchestration, NLO for short), based on Conan, that works for 5+ platforms, including dual runtime platforms like Android and iOS. Also it is a lot of text, sorry.

We are describing our approach to deal with native libraries for several reasons:

• We believe we found interesting alternative way to package libraries using conan, than one is currently used by respective conan community. Our proposal describes approach that involves significantly less manual work, and once and forever solves problems with include/link/etc path. We do understand our approach requires serious re-work of existing packages, but the result is significant enough to start this conversation.
• Some recent changes of conan (packages in editable mode etc) seems to investing a lot of effort into manual management of things we completely automatized already. We see that as an opportunity to help conan community back.
• Conan is a great tool and we see its greatness comes from its flexibility. Latest changes to workspaces are a little bit too “hard-coded”, and we are afraid that future versions of conan might impose something, that will be incompatible with our approach. We believe forking conan is not healthy idea, so we declare our usages - even if conan community will not find a use for our ideas, conan can still stay compatible with our NLO.

Basic ideas in our approach are: avoid any duplication, and use only high-level concepts like CMake’s targets. Yes, its kinda same idea actually - if one is consuming native library by appending some ABSTRACT_LIB_PATH to the include path, and some linking ABSTRACT_LIB_BINS to the application - it is duplication already. Find_package is a duplication as well - dependency is already mentioned in requirements in conanfile, that’s enough for computers to do the rest of the job. Non-transitive dependencies - duplication. Every duplication is bad. You got the idea :-)

A bit of terminology: we are dividing usages of libraries into 2 categories: intermediate build nodes (dependencies, if any, are delivered by conan, result is packed back to conan), and terminating build nodes - result goes out of C++/C/ObjC compilation, and becomes consumer executable, or iOS .ipa package, or Android .apk or .aar package. Intermediate build nodes of the buildgraph do have strict rules to follow, they are like LEGO bricks - must have unified connectors to each other. Terminating build nodes have almost no rules to follow, just “best NLO consumption practices”. In this message we will focus on intermediate build nodes, and if there is community interest - we can describe our “Terminating build nodes practices”, for example, how to efficiently use conan from Gradle to build combo native-java Android applications.

For intermediate NLO build node libraries, we never using/preparing cpp_info of the conan recipe. Instead, we are forcing underlying build system (CMake, mostly) to calculate and export all necessary information as part of the build step. That means, the result of the build is not a static or dynamic library, it is generated (by conan) targets.yaml file, that explicitly describes every high-level build target, with all includes, public transitive defines etc, that build system has built and exported. When we are going to consume such a conan package, NLO generator reads all targets.yaml files from conan packages and makes all listed targets available for transitive targets linkage. Let’s do some examples:

Commented CMakeLists.txt of some intermediate node library:

cmake_minimum_required(VERSION 3.10)
project(AsyncTaskFlow LANGUAGES CXX)
include(${CMAKE_CURRENT_BINARY_DIR}/nlo_link.cmake) #connecting all dependencies from conanfile 

file(GLOB_RECURSE sources src-core/*.cpp src-core/*.h)
add_library(Core STATIC ${sources})
target_include_directories(Core PUBLIC src-core)

# activating coroutines support on all compilers
if(MSVC)
    target_compile_options(Core
    PUBLIC     # public options are transitive and exported by NLO to conan
        /await                   # coroutines support 
        /GT                      # fiber-safe thread_local is coroutine-safe
    PRIVATE 
        /W4)                     # stricter warnings
else()
    target_compile_options(Core
    PUBLIC 
        -fcoroutines-ts          # coroutines support
    PRIVATE 
        -Wall -Wextra -pedantic) # stricter warnings
endif()

target_link_libraries(Core ConcurrentQueue::ConcurrentQueue) # connecting transitive dependency, delivered by conan. Fact of linkage is automatically exported to later stages

nlo_export(Core) #exporting resulting AsyncTaskFlow::Core to conan

In this example we see that CMake build script is building a library and assigns proper options and flags to it. Final nlo_export() command does 2 things: for conan in-cache package build mode, it emits targetname_local_properties.yaml for every exported target, and for workspace-editing mode (“editable package”) it just creates longer alias (projectname::targetname) for every listed target. As one can see, any NLO-oriented CMakeLists.txt is ideally consumable via add_subdirectory command, without any path adjustments etc, that is by design.

Please note - there is no INSTALL() command or step (that would be pure duplication), all we need to comply NLO rules - accurately and honestly separate include directories/flags/etc into PRIVATE and PUBLIC/INTERFACE ones. That’s all. That is our “universal brick connector”. Also please note, that build system will calculate include directories and all other flags anyway - it is necessary to build the code, so why doing it twice?

What happens next, after build system has calculated all the properties and built the libraries?
The next stage is conan picking all *_local_properties.yaml from the build directory, analyzes all of them, and calculates which files should be copied into the packages, with all relative path conversion etc. This stage is automatic, and resulting output is targets.yaml like this one:

AsyncTaskFlow::Core:
  compile_options:
  - -fcoroutines-ts
  include_dirs:
  - ~#package#~/include
  link_binary: ~#package#~/lib/libCore.a
  link_libraries:
  - ConcurrentQueue::ConcurrentQueue
  type: static

Why conan does that? Thanks to beautiful recipe reuse feature, we can just inherit our recipes from “NloBasicRecipe”, that provides prepareCmakeExports() method - it package-localizes every transitive public include/bin directory, for every target. Obviously, custom generator later can read those targets.yaml files and emit something like this:

# target for AsyncTaskFlow::Core
if(NOT TARGET AsyncTaskFlow::Core)
  add_library(AsyncTaskFlow::Core STATIC IMPORTED)
  set_target_properties(AsyncTaskFlow::Core PROPERTIES
    IMPORTED_LOCATION "/home/anon/.conan/data/AsyncTaskFlow/1.0/snap/stable/package/ff399343ed77f827b2e22b962d324d87777f148d/lib/libCore.a"
    # no importlib for AsyncTaskFlow::Core
    INTERFACE_INCLUDE_DIRECTORIES "/home/anon/.conan/data/AsyncTaskFlow/1.0/snap/stable/package/ff399343ed77f827b2e22b962d324d87777f148d/include"
    INTERFACE_LINK_LIBRARIES "ConcurrentQueue::ConcurrentQueue"
    INTERFACE_COMPILE_OPTIONS "-fcoroutines-ts"
  )
endif()

As one can see, our scheme does not require manual include dirs/bins/flags manipulation (in fact, in NLO it is not allowed, since it breaks idea of transitive dependencies). Also library is reusable “out of the box” as part of workspace build - all we need is to emit add_directory(localtion_of_cmakelists.txt) instead of generating code above. Every intermediate library in our NLO is self-consistent, no external tweaks like layout files etc are needed.

Obviously, we can dig into all catchy details, but let us finalize this message with short summary:
we are proposing an alternative packages build approach, which dramatically reduces duplication, and delegates a lot of work to systems that are doing this work already anyway. We are doing such an optimization by using most powerful high-level concepts build system provides - abstract build “targets”. Those targets are automatically exported by the build system. We never “synchronize” cmake’s INSTALL behaviour and conanfile cpp_info properties - we never using them at all. We never worry about include directories or library naming on disk - with high-level targets, one needs to know only high-level target name, and not filenames of intermediate static libraries. Just be honest with PUBLIC properties of your target, that’s all.
And this proposed scheme is out-of-the-box 100% workspace-editing ready, no layouts ever.

Yes, as a side effect - we have to port (well, simplify) ALL third-party packages we are using in our company, to make them NLO-compatible - our management considered this as an acceptable price for universal cross-platform “C++ modules”. We are ready to share our results, and we are hoping conan community will either adopt some of our findings, or at least will consider some compatibility with ideas like ours.

If you are interested in details, please answer here or contact me.

[Johnnyxy]

Hi there,
what you try to do essentially is to avoid duplication in a build process. This is good.

Introduction

What you are doing is to introduce an external description (your .yaml file) of a package. It could be any type of description format too, e.g. JSON, INI, ... . The advantage of this is that one can produce such a machine readable description with the build system of manually or by any other means and parse it later on.
While you are building projects this could be very helpful to unify the package description.

simple packages

For simple dependency packages this could be used out-of-the-box with only a .yaml file that describes the package and a simple conanfile.[txt,py] to integrate it into the Conan system.

complex packages

For complex packages that need some logic at Conan time there will be duplication of some sort.
If I need to do some logic for inclusion of dependencies or setting options etc. a static description file always has no such capabilities. Hence one has to introduce some form of "scripting" in the description file or use another form like Python (.py file) to make decisions at Conan time.

Conan logic and description

Historically Conan contains a logic part and a description part. If one has a simple package the conanfile.txt is enough that describes the dependencies if the package follows the common use definitions for libraries of C/C++ (bin/include/lib/...). The logic part does not need to be customized and one can rely on the default logic from Conan.

For complex packages the conanfile.txt is not sufficient as one sometimes needs additional logic to perform. Hence the conanfile.py is used.
Now what Conan does is to combine a logic and description part in the conanfile.py file together (with python code and cpp_info attribute).

There is the point what is an advantage with your approach. CMake or any other build system (normally) cannot access the settings of cpp_info. So now one has to duplicate the package description to be included in Conan as well as in the recipe of the build system that creates the package (e.g. CMake).

Thinking about package informations

class Recipe(Conanfile):
    settings = "os", "arch", "compiler"
    ...

Conan creates a unique hash value from a recipe's settings, options, ... attributes. With that information Conan identifies a dependency package that needs to be pulled for a consuming package.
Those Conan informations not only contain the target system and architecture of the dependency package they also contain some critical informations like for example the C/C++ runtime it has been linked to (Visual Studio: MT[d]/MD[d]; GCC: libcxx; ...). Now following your maxim to not duplicate informations you would need to specify this only in one place too.

How would you integrate this into your approach that I do not need to set the package settings in Conan separately?

The other way around "Conan-style"

Conan already provides with the CMake-generators all dependency parameters a consuming package needs to consume a dependency.

Additionally if this is not enough Conan has a feature that qualifies it to produce a JSON file with all the informations of a dependency what a build system can parse to get all the parameters for the build (like defines and command line parameters).
For consuming there would not be an advantage with your approach. As the build system has to parse a separate file anyway (.yaml or .json or ...).

But at build time there can be some advantages with you approach. The build system would produce a complete file that describes the package layout and content. Those informations will be later used to import the package as dependency (like include paths, etc.).
The reason for this was/is that Conan has no informations about where the build system puts the built files, etc.

Partially Conan already addresses this already that you do not have to specify the package subfolders and compiler parameters (redundantly) in CMake again. Have a look at the CMake build helper and its defintions (see here) e.g. CMAKE_INSTALL_BINDIR, CMAKE_INSTALL_LIBDIR, ... .

Conclusion

Overall the Conan system is not perfect. Conan is still heavily in development but already has so many stable features that it can be used in production toolchains.

There is still plenty of room for new approaches and ways to solve common software development problems and unify ways to get the software package one wants.
It is admirable that your company is willing to finance such an effort.
I do not know if it is advisable to reformat/restructure/rework ALL (external) recipes that your company uses (like maybe boost, zlib, ...) as for those recipes there is already done much work to make them usable with many build systems and targets. You would introduce more work in the future as you remove the OSS strategy that many many people can contribute and test software/code from the Conan usage. Not limited to those in one company :).

Your approach does do a step into the unifying direction as generally only the build system really knows that it does and where it puts what. I would tend to an approach that Conan provides some means to parse such a build-description after the build to integrate it into its cpp_info to avoid duplication.

Anyway your work can be a good template for further considerations.

[ezaiakinsc]

Hi Johnnyxy,

First, thank you for your time - you provided nicely detailed answer! It is pleasure to have this discussion with you.

Sorry if that sounds defensive, but we are not trying to avoid duplication, we actually managed to do that - that's why we explicitly mentioned our results are not theoretical. It works as described. It is not an attempt, it is success.

As a generalization, let's call "connectivity information" between include path, library path, compiler flags and definitions as a "meta-information", that is absolutely necessary for correct (transitive) library consumption. This information is definitely part of the library itself - yes, sometimes it is expressed in documentation as "include this, and link that, and define that preprocessor flag", sometimes it is manually hardcoded into conanfile.py etc. Lack of standardization (or even acceptance of its importance) for this metainfo is one of original causes of C++ in 2019 still has worse modules/units/packages structure, than many other technologies around.

Sure, you are right, there are a lot of different approaches to deliver/restore such a metainfo - for example, you've mentioned one with CMAKE_INSTALL_LIBDIR - this one tries to do it in reverse, by forcing all libraries to have same metainfo. And you are very right - many existing approaches are struggling when it comes to simple libraries (say, libpng), or, as opposite, to a complex libraries, like abseil/boost/opencv.

Say, with current conan community approach, this metainformation is manually hardcoded 3 times:

• in original build script - build system has to know those to build code
• then in cpp_info of the conanfile
• and then again, in layout files for workspaces.

That is one of the reasons we had to avoid packages, provided by conan community - long-time costs of supporting and synchronizing 3 duplication points for us is more expensive, than porting libraries to automated system once. If there will be forth/fifth build scenario - conan community will have to hardcode metainfo for those again, for every library. And we will have to duplicate that for our libraries, as long we are using cpp_info. With our approach, library metainfo is not lost, no need to re-restore it, it is full and self-contained, thus can be automatically reused in (any?) scenario.

Another disadvantage of the current cpp_info/CMAKE_INSTALL_LIBDIR approaches - inability to correctly handle complex libraries in general case, because it is impossible to have different targets with different include directories or/and flags. It's hard to say, if and when exactly that might become a problem, but it is just a fact - even in case of relatively simple libraries, like GoogleTest, multiple targets support is a necessity (targets are GTest and Main). Any library, that will require differentiation of includes or build defines/flags per target, will be impossible to use with "community conan" and any existing OSS generators, since attribute cpp_info is global for the package, and cannot be used on per-target basis in non-trivial case.

You are right, within our approach, "library consumption metainformation" is made explicit and reusable, with a help of conan. Conan has beautiful "options + settings -> hash -> separate binary" architecture, which naturally fits for "calculate and bake targets metainfo once per combination of os/compiler/options/etc". Build-generated targets.yaml/json/whatever - is ideal fit into conan's versioning model.

Sure, it is not the only change we did while designing our Native Libraries Orchestration approach - for example, correct workspaces support is impossible with community packages, while community is using "anonymous" BUILD_SHARED_LIBS-alike defines. For NLO packages, all defines must be package-specific (that is also automatized btw), say, NLO_OPTION_GTEST_SHARED is workspace-compatible, opposite to "just" BUILD_SHARED_LIBS. So, speaking about us "removing OSS strategy" is not true - we are reusing everything that is not proven to be suboptimal in something, that is important for our cross-platform development flow, and we are happy to contribute to fix things for community. Today conan community is investing a lot of resources in cpp_info infrastructure, and for us it seems suboptimal - we know, it can be automatized once and forever, with better "detail level" or "precision", than manual version.

We believe it is cheaper to aid community approach right now, when there are no hundreds thousands OSS packages with manually hardcoded metainfo, and maybe aim for Conan 2.0 as target-aware system with minimal/no duplication. If community is interested in such an improvements - we are ready to cooperate. If our ideas are not valid for conan developers and community - we hope, at least, conan will stay flexible enough to be used in all scenarios without mandatory cpp_info/layout files etc, with recent changes in editable packages and workspaces those are harder and harder to avoid.

[Johnnyxy]

Hi there,
first ... I did not see the workspaces / layouts as third section where one has to describe something redundantly. I think I missed this point :). Regarding this it would be 3 places where one has to specify the structure and with 3+ it starts to be too much.

Targets

If I did misunderstood you please elaborate :).

I am a little confused what you mean with a "target". You said:

Another disadvantage of the current cpp_info/CMAKE_INSTALL_LIBDIR approaches - inability to correctly handle complex libraries in general case, because it is impossible to have different targets with different include directories or/and flags.

If you mean a target in terms of CMake I do not see the problem as one can build serval CMake targets with one Conen recipe.
I think we have to distinguish between build-targets and the resulting package. One package consists of a build-step and package-step. Ignoring the Conan specific source, export, etc.
In the build-step one has at least one target that has to be built. If you have serval targets in CMake (e.g. add_library (TARGET_NAME)) you can build them independently in your def build(self) function. Actually I have done this while working on a project that consists of one target and multiple targets that depend on that.
One way was to create a CMake-helper for every target and build it. This is one way to build multiple targets in CMake with different arguments for CMake/Compiler/... . Not the prettiest but it does the work :).

def build(self):
    cmake = CMake(self, source_folder="/PROJECT1")
    cmake.configure()
    cmake.build(target="MYTARGET1_1", definitions={arg: "value1_1"})
    cmake.build(target="MYTARGET1_2", definitions={arg: "value1_2"})
    ...
    cmake = CMake(self, source_folder="/PROJECT2")
    cmake.configure()
    cmake.build(definitions={arg: "value2"})
    ...
    cmake = CMake(self, source_folder="/PROJECT3")
    cmake.configure()
    cmake.build(target="MYTARGET3", definitions={arg: "value3"})
    ...

But if different (CMake) targets have different dependencies and depending on them different build attributes it is quite difficult to achieve this with Conan in one recipe.
Conan goes the simple way here and "says": "Use one recipe for one target.". It is easier to maintain and I think was a first good step to simplify things in the history of this relatively young project.

Conanfile settings

Can you elaborate how you define the settings in a Conanfile with a separate description file?

Thinking about it, with python it is possible to execute a function that returns the value for a setting or the whole settings object

settings = {"os": get_os_from_yaml_file()}

But as the YAML file is generated at build time this information does not exists when running Conan to build a package.

A proposal proposal

Now innovation / development is never wrong or futile. I generally would like to know more about the technical design behind this.

You are using a static .yaml file to retrieve all informations about your packages. A proposal I would go for is for "executable" files. As you already use Conan and Conan depends on Python it is possible to use Python for that as it normally has to be available already.

class MyPackage(self):
    def get_compile_options(self):
        return ["W4", "WX"] # for MSVC
    def get_include_dirs(self):
        return ["/abc/def", "ghi/klm"]

As you already have serval functions implemented in CMake for your approach it would be possible to port this to retrieve the informations from the Python script file.
As a template I am thinking about the following implementation:

function (GET_INFO)
    set (KEY_VALUE "PATH" "RESULT")
    cmake_parse_arguments (PARSE_ARGV 0 "PARAM" "" "${KEY_VALUE}" "")

    set (COMMAND "python -c 'import YOUR_YAML_FILE; print(MyPackage().get_compile_options())'"

    execute_process(
            COMMAND             ${COMMAND}
            OUTPUT_VARIABLE MY_INFORMATIONS
            RESULT_VARIABLE RESULT)
    if (NOT RESULT EQUAL 0)
        message (FATAL_ERROR "retrieving informations ... failed ({error: ${RESULT}})")
    endif()
    set (${RESULT} "${MY_INFORMATIONS}" PARENT_SCOPE)
endfunction()

With that you would have the possibilities to have flexible code (Python) and integration into other build systems.
Even if you have a build system that does not support such thing as execute_process in CMake you could still generate a YAML file out of the Python script before build system invocation.

[ezaiakinsc]

Hi Johnnyxy!

We definitely should elaborate what "NLO target" means, from the very beginning :-)

In our terminology, target is "atomic (as in "minimal possible") self-sufficient piece of functionality, that can be expressed as set of include directories, and/or link binaries, and/or compilation flags and defines (and nested sub-dependencies targets, for sure)". When one is using NLO target, it is always consumed by target_link_libraries() call, and only in transitive manner.

For us, self-sufficiency of a consumable target is one of the keys features (distributed teams) - target must be configured properly, by activating all necessary corresponding compiler features with PUBLIC and INTERFACE properties - it is cheaper to do that once on "producer" side, rather than compensate lack of this precise metainfo in every "consumer". Your assumption is really close - NLO targets are almost equivalent to "Modern CMake" targets, if you think about them in "generic manner". For example, Eigen, header-only library, in NLO is exported as target with include directory and transitive defines, stipulated by conan package options. And within CMakeLists.txt Eigen is represented as INTERFACE target, created with add_library(eigen INTERFACE) - so, it is a valid target-link participant without build procedure - it has only nlo_export() stage. Another representation of the NLO target is cpp_info entry - since cpp_info defines exactly one atomic "piece of the program", it is almost 1:1 to a single entry in targets.yaml file. So, NLO target is a logic concept, but it is representable in real world (mostly as sub-project).

Within NLO consumers do not care if some library is header-only or not, does it requires some defines to function/comply with conan options or not - C++ artifacts are automatically connected properly to one's program, like imaginary "C++ modules", one just adds necessary requirement to conanfile, and then target-links necessary targets subset from this requirement.

Say, imagine opencv library in a conan package. Then opencv::calib3d and opencv::videoio - are examples of targets, provided by this package. Unfortunately, since cpp_info is a "singleton" for an OSS conan package, there are no theoretical way to separate public-transitive defines for opencv::calib3d from defines for opencv::videoio - if you touching one of them, you got the whole set. No way to add include directories only for opencv::videoio, so at least one line of "code validation defense" is lost - one getting waaay more includes than he or she explicitly linked to the program. And way more compilation flags than actually requested.

You are right, it is possible to build different targets in some kind of iteration, but the problem of "singleton cpp_info" is that it will be single consumption target in the end, regardless how complicated was the build procedure. Imagine you have some kind of highly-optimized C++/C/asm library with 2 targets: 1st is library itself - super-minimalistic, emdedded-systems oriented core, probably with pure C api. And second target of the same library - is a nicely crafted C++ wrapper, for those who love templates and exceptions. Since second target uses exceptions as part of its interface, it must declare -fcxx-exceptions as its public transitive compilation flag - otherwise that library is not self-sufficient anymore. And here comes "singleton" problem - since information about "who actually needs -fcxx-exceptions flag" is completely lost by merging both targets into single cpp_info - consumers will get exception-targeted function prologs and epilogs even if they are requesting only "C core" part of the library. So, result of this mind experiment with OSS targets approach is bloated binary in cases when only part of the library is used.

For us, binary size is one of important metrics, our users are not happy to download hundreds of megabytes of bloatware, so we cannot afford "just link the whole boost, who cares" approach. We also cannot afford manual approach - if one has to read some kind of documentation and add flags/directories/defines to one's program just to consume a library - it is a failure of build automatization, transitiveness, as well as self-sufficiency. And supporting bunch of non-self-sufficient libraries for 5+ platforms is a huge pain in the apple.

OSS conan packages do not have targets separation, and can be affected by "just link everything" approach. Of course, one can say "if there are different inlcude dirs/flags/etc - put them into separate conan packages", but idea of splitting GoogleTest into 2 conan packages does not seems like an improvement or duplication reduction. Some libraries do contain dozens of targets (abseil, boost, BulletPhysics, opencv etc) - atomizing those into separate conan packages each sounds horrifying.

PS: After re-reading your answers, I think I might explained roles and phases of NLO package not good enough. Yes, generated targets.yaml file is static, but it is (potentially) different for every conan binary package, and correlates to exact settings and options set, used for generation of this targets.yaml file (that's why it is static). Let's do this with examples:

Example recipe with options:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from conans import CMake, python_requires, tools
import os
nlo = python_requires("nlo-cmake-pythonlib/1.0@snap/stable")

class GTestConan(nlo.NloBasicCmakeConanFile):
    name = "gtest"
    version = "1.8.1"
    homepage = "https://github.com/google/googletest"
    exports_sources = ["CMakeLists.txt"]
    options = {"shared": [False, True]}
    default_options = {"shared": False}

    def source(self):
        tools.get(f"{self.homepage}/archive/release-{self.version}.tar.gz")
        os.rename("googletest-release-" + self.version, "libsrc")

        # patching gmain - main() must be always static. Google, hello, shared main()?!
        tools.replace_in_file("libsrc/googletest/src/gtest_main.cc",
            "GTEST_API_ int main(int argc, char **argv) {",
            "int main(int argc, char **argv) {")

    def build(self):
        self.simple_cmake_build()

Now, the buildscript, that calculates and exports target properties. And also builds them :-)

cmake_minimum_required(VERSION 3.12)
project(GTest LANGUAGES CXX)
include(${CMAKE_CURRENT_BINARY_DIR}/nlo_link.cmake) 

set(sources libsrc/googletest/src/gtest-all.cc)
if(NLO_OPTIONS_GTEST_SHARED)
    add_library(GTest SHARED ${sources})
    target_compile_definitions(GTest
        INTERFACE
            GTEST_LINKED_AS_SHARED_LIBRARY=1
        PRIVATE
            GTEST_CREATE_SHARED_LIBRARY=1)
else()
    add_library(GTest STATIC ${sources})
endif()

target_include_directories(GTest
    PUBLIC 
        libsrc/googletest/include
    PRIVATE 
        libsrc/googletest)

# default main for testrunners
add_library(Main STATIC libsrc/googletest/src/gtest_main.cc)
target_link_libraries(Main PUBLIC GTest)

nlo_export(GTest Main) # exporting both GTest::GTest and GTest::Main

That's it. Now, lets build this package with different options, and check resulting targets.yaml files:

• static version (shared=False)

GTest::GTest:
  include_dirs:
  - ~#package#~/include/GTest
  link_binary: ~#package#~/lib/libGTest.a
  type: static
GTest::Main:
  link_binary: ~#package#~/lib/libMain.a
  link_libraries:
  - GTest::GTest
  type: static

• dynamic version (shared=True)

GTest::GTest:
  compile_defs:
  - GTEST_LINKED_AS_SHARED_LIBRARY=1
  include_dirs:
  - ~#package#~/include/GTest
  link_binary: ~#package#~/lib/libGTest.dylib
  type: shared
GTest::Main:
  link_binary: ~#package#~/lib/libMain.a
  link_libraries:
  - GTest::GTest
  type: static

As one can see, targets.yaml totally defined by settings and options of the package - because it is result of calculation of build system, which is reliable defined by build setup. We extracted that information from the build system automatically, in nlo_export function, and we are ready to give this meta-information back to the build system (any build system, not just original CMake). In this example one can clearly see how GTEST_LINKED_AS_SHARED_LIBRARY=1 define is being automatically recognized as transitive and exported as compile definition for appropriate configuration.

[niosHD]

Very interesting discussion, thank you for your sharing you concept! I try to summarize my high level understanding of NLO's features and the current technical challenges here. Please correct me if I missed anything essential or interpreted key aspects incorrectly.

Feature Perspective

In my understanding, NLO gives you the following properties:

• DRY: NLO removes duplicated information from the build and packaging system.
  • Package dependencies are only defined in conan but they get forwarded to CMake via a custom generator.
  • Build properties are only described in the build system description (e.g., compiler flags) and get recorded in the package targets.yaml file.
• NLO reduces the amount of code in the conanfile and build system description by convention.
  • E.g. CMake code is simplified via helper functions (e.g., nlo_export) that supersedes CMake's standard install commands and omits find_package calls.
  • Conan recipes are simplified by providing a base class (e.g., nlo.NloBasicCmakeConanFile) with custom build helpers.
• NLO bypasses conan's current cpp_info model completely and provides a more fine grained target model via the custom targets.yaml files.

Did I miss anything else? I think these features are very valuable, we definitely should strive to continue to support them with conan as good as possible.

Technical Perspective

Looking at NLO from the technical side, I currently see the following three concepts:

1. Package Description: A custom targets.yaml file format is used to describe the targets in a package along with the necessary flags. This seems to be a custom alternative to CMake's config files, pkg-config files, and vector-of-bool's libman (see also cpp_info should contain relative path to individual libraries #3931).
2. Communication from conan to the build system: A CMake based file format (i.e. nlo_link.cmake), generated via a custom NLO generator, is used.
3. Communication from the build system with conan: This step is basically the generation of the targets.yaml via the build system. However, conan is currently oblivious to the file as well as its content and just packages it.

Please correct me if I am wrong but I think that the custom package description format as well as the generator part are working fine in NLO. In my understanding, the main problem is that conan is oblivious to the package description format. Subsequently, using the information in conan, e.g., in editable packages or for workspaces, is currently difficult.

As a side note, the CMake only approach that I currently use, suffers from exactly the same problem. In particular, to DRY the conanfiles, I heavily rely on CMake's install machinery, i.e., CMake config files are used for package description and the cmake_paths generator for communication with the build system.

My Conclusion/Opinion

I currently see two challenges that have to be considered when when employing an approach such as NLO, which employs external package description files.

1. Conan needs to be aware of the packaging information in order to exploit it internally. This does not mean that the information has to be repeated in the conanfile as we definitely should strive for DRY. However, it definitely means that we need infrastructure to map arbitrary package description formats more precisely.
2. The time at which information from external package description files is available to conan is crucial and dictates what conan features are affected.

Regarding the first challenge, I think one of the blocking issues is that conan has a cpp_info model that lacks support for fine grained targets with individual flags at the moment. This is a known issue that gets already discussed in #2387. Please join the discussion there and make sure that the new model can capture all the information you have in your package description. If we get the new model sufficiently generic, I would hope that it gets possible to populate the cpp_info, e.g., by parsing available package description file formats or by querying some API (see #2387 (comment)), in order to DRY the conanfile.

Regarding the second challenge, I am still not sure how specific conan features like editable packages should/can interact with external package descriptions that are generated by the build system. The problem I see is that it would require that an editable package is at least configured before conan has access to the required information. I assume that extending this into the workspace setting makes matters even worse.

Still, if we can solve these technical challenges, I would love to have first class support for approaches such as NLO in conan.

Regards,
Mario

[mbodmer]

This is an extremely interesting aproach. Can you share your nlo-cmake-pythonlib?

There are at least two new specification formats for this kind of package information, that I am aware of. One is CPS and the other libman. Both have a ISO C++ proposal:

• CPS spec
• CPS proposal
• libman spec
• libman proposal

Regards, Marc

[memsharded]

Thanks for all the valuable feedback and discussion here.

Conan has changed a lot since then, let me summarize:

• There are new modern CMake generators, including generation of toolchains, CMakePresets, etc, centered in creating a fully transparent integration via standard find_package() and targets, no longer management of individual variables
• For the avoidance of duplication in the definition of packages in package_info() we are collaborating in the C++ standard towards a CommonPackageSpecification (CPS) that would solve this.
• Conan 2.0 has doubled-down on flexibility and extensions, via things like custom user commands, plugins, etc. Python-requires as a mechanism to reuse code among recipes has been stabilized and is widely used.

So I think the approaches described here are very interesting, but at the moment is not something that we can pursue, that would mean entering the realm and responsibilities of a meta-meta-build-system mostly, and this is something not possible for us, but we are trying to collaborating instead with other players and build-systems to achieve the same final result all together. I am closing this as not planned, thanks very much again!