Godot integration with existing open-source or closed-source software (conan support - make all happy with scons, including CMake users)

[blockspacer]

Describe the project you are working on:
Proprietary information

Describe the problem or limitation you are having in your project:
Godot integration with existing open-source or closed-source software

Describe the feature / enhancement and how it helps to overcome the problem or limitation:

conan is awesome C++ Open Source Package Manager https://conan.io/

it can be used with scons, makefiles, pre-built binaries, CMake like so https://jfreeman.dev/blog/2019/05/22/trying-conan-with-modern-cmake:-dependencies/

If Godot used conan + scons, then it would be easier to integrate Godot modules with apps (especially important if you want to create server with custom netcode and main loop, but integrate some Godot modules, like physics).

Now if you want to integrate Godot with existing software - you must use scons or manually duplicate all defines, link flags, e.t.c. used during Godot build. conan can solve that problem (just make sure that Godot module used conan+scons and can be imported into cmake+conan based project)

NOTE: Godot can keep existing SCons build system and add conan integration with minimal changes

Related to
godotengine/godot#9694
godotengine/godot#81
godotengine/godot#8983
godotengine/godot#37182

copy of godotengine/godot#37180

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
conan can use python to create recipe file, it described at https://docs.conan.io/en/1.6/creating_packages/getting_started.html#getting-started

conan integrates with Scons like so:

class PkgConan(ConanFile):
    settings = 'os', 'compiler', 'build_type', 'arch'
    requires = 'Hello/1.0@user/stable'
    generators = "scons"
    ...

    def build(self):
        debug_opt = '--debug-build' if self.settings.build_type == 'Debug' else ''
        os.makedirs("build")
        # FIXME: Compiler, version, arch are hardcoded, not parametrized
        with tools.chdir("build"):
            self.run('scons -C {}/src {}'.format(self.source_folder, debug_opt))

For more details see https://docs.conan.io/en/latest/integrations/build_system/scons.html

If this enhancement will not be used often, can it be worked around with a few lines of script?:
No

Is there a reason why this should be core and not an add-on in the asset library?:
Cause conan recipes need to be up-to-date with scons recipes

[Xrayez]

If Godot used conan + scons, then it would be easier to integrate Godot modules with apps (especially important if you want to create server with custom netcode and main loop, but integrate some Godot modules, like physics).

Not sure if your proposal is related to XY problem to some extent, but you may be interested in:

• Custom modules build option: Add custom_modules build option to compile external, user-defined C++ modules godot#36922.
• C++ Game module C++ Game module #565.

In fact you can move around most of the built-in modules outside of Godot as described in: godotengine/godot#36922 (comment).

Regarding separation of modules godotengine/godot#37182 see #569.

It's not possible to compile them individually as they are dependent on core stuff within the engine None of the modules are meant to be used outside of Godot, except for the thirdparty code used by some of them perhaps.

I'm using the same SCons build system currently to build both a Godot project and the engine (with custom modules) itself, see my comments in:

• Building Godot as part of a larger project: Proposal: Building Godot as part of a larger project, changes to Build System godot#27842 (comment).

Not without issues of course, because the build scripts rely on custom Python-specific path resolution, hence the need to create a wrapper similarly as in your snippet.

If your proposal can alleviate some of the hurdles related to the recursive build, I'm not against the proposal at all, but likely the existing Godot build system needs to be fixed instead (talking mostly about my own use cases). Especially if it takes minimal changes and contributes to the community needs (note: we're all part of the community).

[blockspacer]

@Xrayez thanks for links to related issues

yes, conan can solve recursive build problems

Note that proposal will allow not only easier integration of other open-source projects into Godot, but also integration of Godot libs (modules like physics) into existing games, editors, tools or game servers (important for game servers cause them usually can use separate modules and not whole engine)

[fire]

I would disapprove a Conan package manager for Godot Engine and have spent collectively a lot of time debugging git submodules and other dependency tracking systems. They have caused me grief because debugging and managing versions is difficult. Godot Engine is a single repository.

Secondly, Conan uses cmake which requires us to change parts of Godot Engine from SCons to cmake. This has been discussed and the status quo is to not move to cmake. (I've written a few cmake build system for Godot Engine..)

For game servers we can already separate modules via flags for the reduction of footprint.

GDNative modules have the same sort of complicated behavior Conan has with the modules required to be compiled for as many platforms as release platforms. However, the GDNative system already exists. GDNative already allows you to use any build system including cmake and conan.

I don't see a place where Conan modules fit.

[edited]
Swap order: from scons to cmake.

[blockspacer]

@fire
Thanks for response.
Please note that

1. Conan can support scons and pre-build libraries (conan can wrap almost anything with conanfile).
2. There is no way to import separate Godot libraries (.lib, .dll, .so, e.t.c. files) into custom application. Godot does not provide any information about used compile flags. Conan just allows to collect that information while executing same scons build as usual, so any existing open-source software will be able to use separate Godot libraries. This is the main problem stated in that issue.
3. GDNative mainly used to import some code into Godot. If GDNative can be used to import Godot libraries into existing software (that can use any build system, not scons only), than issue may be closed.
4. Of course Conan can be used to manage Godot deps, but we can wrap existing build architecture (scons files, single repository) into Conan with minimal changes

[Shatur]

This has been discussed and the status quo is to not move to cmake.

Why are developers against CMake? Just wondering. CMake have out-of-the-box integration with almost all IDE's and have a lot of built-in tools.

[blockspacer]

@Shatur95
Issue is not about CMake, but about Godot usage (or some parts of Godot) in other projects (that can use CMake or any other build system)
Example: some project may want to create QT based IDE that works with Godot i18n API (So IDE will be able to use C++ library from Godot that works with translation files).

Why are developers against CMake?
godotengine/godot#81 (comment)
I think that because reduz prefers python to CMake. Also he thinks that Godot build process is way too complex for CMake. (very wrong, CMake used by a lot of projects bigger than Godot)

[Shatur]

Issue is not about CMake, but about Godot usage

Yes, I understand, just asked :)

I think that because reduz prefers python to CMake. Also he thinks that Godot build process is way too complex for CMake. (very wrong, CMake used by a lot of projects bigger than Godot)

Got it, thanks. I agree with you.

[Calinou]

Closing due to lack of support. Note that Godot is not meant to be used as a library. Also, a custom_modules SCons option was recently added so that you can compile modules located outside the Godot source repository.