Make modules... modular at compiling project (not the engine)

[Bethleem]

Describe the project you are working on:
An interactive fiction engine only-text, but the proposal will benefit all godot projects.

Describe the problem or limitation you are having in your project:
The release android apk is too big for a only-text project.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
So, the idea is to to be able to discard modules at compiling time. In my case of use will be everything but 2d basic rendering and will decrease remarkably the output size while 84% of it is the engine.
I know you can get a custom godot engine and templates if you recompile the whole enchillada engine source code. But, IMHO, this is a task only for advance users. Very advance. Your average indie developer might hit a wall when trying to get ride of some modules.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
Well, I´m not your very advance user and don´t know even how to properly compile the engine (try and failed).
But the approach should be like the one used in Defold. You configure a simple manifest file and the engine include only the modules you want to in your project release.
https://britzl.github.io/manifestation/
It will be a huge improvement in otrimization.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Few lines, no. Recompile full engine and templates.

Is there a reason why this should be core and not an add-on in the asset library?:
Strip parts of the engine could only be done... by the engine itself? Don´t know for sure, as I said I´m not your very advance user, just an average indie developer.

[groud]

Did you have a look to this ? https://docs.godotengine.org/en/stable/development/compiling/optimizing_for_size.html#disabling-unwanted-modules

Godot already allows you to disable parts of the engine at compile time.

[Xrayez]

Godot already allows you to disable parts of the engine at compile time.

I think that's the exact problem this proposal is trying to convey, but yeah it would still be very difficult to achieve seamless compilation for an average user given Godot's architecture.

@Bethleem the manifest generator you linked (https://britzl.github.io/manifestation/) reminds me of the feature that Godot kind of already has, not necessarily the generator part, but the ability to configure a build via configuration files (which are simple Python scripts).

See also Overriding the build options page for more info (just recently documented).

So, disabling all unneeded modules is a matter of listing those in a file, and compiling Godot with:

scons profile=manifest.py

Yet it's still up to you to generate the "manifest" file. There's also a marginally related proposal about this: #288.

But the approach should be like the one used in Defold.

I think that we should draw a clear distinction between C++ modules development and the Android/Java modules which have their own workflow to eliminate confusion, I'm not aware of the exact workflow used there (does a user still has to compile the engine?)

In any case, hopefully we can achieve a good integration of GDNative plugins which would replace the need for the "hardcoded" C++ modules, but personally I don't feel like GDNative can achieve seamless integration in comparison just yet, see #565, #119.

Perhaps non-essential modules may be safe to disable which are not used by the engine directly by default, like bmp (meaning that the official builds won't contain them by default, those who needs them would have to recompile the engine and enable them explicitly, like module_bmp_enabled=yes).

[Bethleem]

I think that we should draw a clear distinction between C++ modules development and the Android/Java modules which have their own workflow to eliminate confusion, I'm not aware of the exact workflow used there (does a user still has to compile the engine?)

Nope. You code your program in the editor and then add the manifest to the project. Then build (the project, not the engine) and defold read the manifest and strips off the modules you don´t want for the release.

As I said, I don´t know if that´s even possible in godot, but it will rock about optimizing releases.

[Zylann]

Disabling modules without recompiling Godot: I don't know how that would be possible without modules being dynamic C++ libraries. But the interesting part about that, is that "bloat" becomes a lower issue, since extra functionality can be stripped that easily.

Otherwise, it has to be at compile time of the engine anyways. Maybe the title of this proposal is misleading because that already exists. If the point is to make it more accessible, you'd need scons and a compiler supporting your target platform, if you want the export process to do it.

[Bethleem]

I was thinking... It could be done by a set of templates? I mean, for mobile target, will it be possible to have a set of two or three templates, one with full engine, one with extreme size optimization and so? Just distributed with the binaries/templates, ready to use, no scon and recompile need.

[Zylann]

If you start to go that route you'll end up with hundreds of versions of the engine. When we factor in CPU architecture, OS and presence of Mono, there are already 14 full-editor binaries to consider (20 when considering web, ios and android export templates).

[Bethleem]

Well, I was aiming to a single binarie engine per platform and several mobile templates for that binarie because in desktop size doesn´t matter so much. My idea was something like three template flavors: full , full minus 3D, extreme size optimization. Something like that. As I said, I not even know if it is possible, just throwing ideas out there hoping to improve godot.
And yes, in some of the templates part of the API won´t work due to the modules stripped, but users can choose which one of the three fits their project better.

[Calinou]

For those wanting to compile Godot with specific modules disabled, you can use the Godot build options generator to generate a custom.py SCons configuration file.

[Xrayez]

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Few lines, no. Recompile full engine and templates.

#1001 (comment):

If the point is to make it more accessible, you'd need scons and a compiler supporting your target platform, if you want the export process to do it.

This in theory could be solved by:

• https://github.com/godotengine/build-containers
• https://github.com/godotengine/godot-build-scripts

So you can compile the editor and export templates in a similar way as official builds (hopefully). 🤞🤞

Those toolchains are still work-in-progress, unfortunately.

Of course, it has to be fairly documented to be accessible to an average developer that is not deeply involved with docker and related build system tech stack (I'm learning myself currently)...

Either this or GDNative approach, which I believe is a nightmare to setup (and especially maintain) compared to current built-in and custom C++ modules mechanism.

[Bethleem]

For those wanting to compile Godot with specific modules disabled, you can use the Godot build options generator to generate a custom.py SCons configuration file.

Thanks, for the custom generator. :) Now I see clearly that my proposal is not viable, too many possible variants.
I´ll try to recompile the engine in order to reduce export size. One question about that: if I am working, just say, with standard godot 3.2.3 and I recompile only the templates with several exclusions, do I need to recompile the editor too or that custom export templates will work if I point to them in the custom templates options of the standar godot binaries?

Thx!

[Xrayez]

One question about that: if I am working, just say, with standard godot 3.2.3 and I recompile only the templates with several exclusions, do I need to recompile the editor too or that custom export templates will work if I point to them in the custom templates options of the standar godot binaries?

If you, say, disable regex module in export templates, you might get away with using official Godot editor, but to be safe, I'd compile both editor and export templates. It may be possible that you may use some feature available in the official editor builds without even knowing it, and then when you export the project, the game may crash etc.

If you do compile editor, make sure to compile it with target=release_debug, and not debug (which is default).