Create a built-in editor tool for organizing and mass/selectively compiling GDNative projects and addons

[willnationsdev]

Describe the project you are working on:
I have attempted in the past to build addons written in NativeScript C++, but the process is cumbersome and concerns me at scale.

Describe the problem or limitation you are having in your project:

Godot's NativeScript is very much a second-class citizen compared to GDScript, C#, and VisualScript. It requires a lot of technical know-how to set it up. Work that is done automatically for other languages is difficult or impossible to do for NativeScript.

1. It does not auto-compile before scene launch.
2. Unlike Mono, it has no GUI-based means of...
   1. setting up a project space.
   2. manually triggering a build.
   3. reviewing the build output/errors or project information.
3. The "Create a script" process is often a multi-step process. For example, with C++:
   1. Create .h file.
   2. Create .cpp file.
   3. Update one's init.cpp file so that godot_nativescript_init(...) properly registers the new class name.
      • Must also specify whether it should be a tool script or not.
   4. Create a new .gdns resource that points to the correct class name.
      • Must also specify the script class name and/or icon path, if any.

PluginScript is a concept that was designed to accommodate some of these concerns. However, there are many NativeScript languages already created. Improving the toolset to create, manage, build, and use NativeScript code would provide more value to Godot's addon ecosystem overall.

In addition, many NativeScript languages must deal with the dilemma of where to put their source code.

A user who installs NativeScript code via the Godot Asset Library should have access to the source code while still keeping the source code out of their project's final binary. This means:

1. The source code must be kept in the git repository tied to the asset, so that it can be easily downloaded.
2. The user should be able to access their project's code and the addon's code through a similar, intuitive interface that still distinguishes between the two.
3. The user should be able to build and use Godot resources (GDNativeLibrary/NativeScript) in the project without the source code files becoming part of the final binary, i.e. when downloaded, the source code files should be moved to a third party user:// location so that they can be referenced by the editor on a per-project basis, but not bundled into the final binary.

The editor should be able to scan directories (maybe /src and /addons/**/src as conventions) for necessary files (SCSub, config.py) which NativeScript bindings version they should use, etc. Then the editor can work out the minimal number of dynamic libraries necessary, generating all appropriate GDNativeLibrary resources, and likewise generating a NativeScript resource file for each registered class, along with optionally (in the python configuration perhaps, or ProjectSettings for project-wide configuration) assigning script class names to them all. The editor would ideally provide an intuitive GUI for configuring and automating all of this.

Describe how this feature / enhancement will help you overcome this problem or limitation:

Create a base GDNativeBuild resource that allows users to configure the construction of different types of GDNative projects. The engine could then provide a derived GDNativeBuildCpp resource tailor-made for building and managing NativeScript source code. Those managing repositories could then create their own derived versions to have a user-accessible GUI built into the editor with minimal effort.

For C++, we'd need to create an "official", highly generic, parameterized SConstruct script that Godot can generate on command (if we want the language to have better first-party support). It could simply download the bindings to the user settings under a version number, thereby allowing multiple "builds" to reference the same bindings version number.

We can then use an EditorInspectorPlugin to create buttons and other behavioral elements to the GDNativeBuild to provide a unified interface for doing all operations necessary for all NativeScript projects:

1. Regenerating the Godot API JSON (for master builds).
2. Downloading a particular version of the bindings (1.0, 1.1, master, etc.).
3. Generating the boilerplate for a project associated with the Build's target language.
4. Managing the project files created in step 3.
5. Building the static library for a particular bindings version.
6. For any one project...
   1. Building the source code with the static library into the dynamic library.
   2. Generating scripts for the language represented by the Build resource via templates.
   3. Generating the GDNativeLibrary and automatically hooking it up for the user.
   4. Generating the NativeScript files and automatically hooking them up for the user.
   5. Allowing the user to manage, through an interface in the Build's Inspector, which classes exist, which have .gdns files, which are tool scripts, etc. (and toggle on/off the presence of the latter two).

This would all shift work that would otherwise be difficult to learn, manually executed, and an absolute maze of chaos to manage into something simple, intuitive, built-in, automated, organized, and visual.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

Some generic interface for executing necessary NativeScript processes. There could be a lot more than just this, as mentioned in other sections, but this is just a general idea. The buttons at the top are inserted via an EditorInspectorPlugin.

"Sub builds" would be an array of resources that allow one to bunch multiple other builds under a single build recursively, i.e. requesting to build the top level one would also build the lower level ones, or the lower ones could be expanded to open their own toolbar for building independently, etc.

I would imagine you could have additional GUI elements for detailing where to place the source code, which classes are present / what their status is (registered? tool? NativeScript?) and GUI elements for generating files. Everything outlined above, though I'm not the best UI designer so maybe others have better ideas.

Describe implementation detail for your proposal (in code), if possible:

1. We would need to create a /gdnative/build submodule. It would need to have...
   1. GDNativeBuild
   2. GDNativeBuildCpp
   3. GDNativeBuildPlugin
   4. GDNativeBuildInspectorPlugi
   • Users could then add GDNativeBuild<OtherLanguage> via the Asset Library.
2. We would need to create the SConstruct file necessary to compile the scripts and make it a permanent addition to the godot-cpp repository. Default users who are just getting started could use it while power users that prefer to write their own could replace it with their own or ignore it.
3. The Cpp version would need to also support generating Visual Studio projects since that is a commonly used editor for Windows users (myself included).
   • Careful thought would have to be put in here to make sure we don't conflict with users who have both a C# and a C++ project in their Godot project.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

It would be used very often. I have already started working with it and then frequently stopped because of how much of a hassle it is to setup, manage, and use the C++ scripts compared to GDScript and C#. If it were easier, I would be more inclined to use them. And if we are going to improve the usability of one, we may as well ensure that the NativeScript ecosystem itself can benefit from the changes rather than just C++ alone (as a PluginScript would do).

Once we start having more first-party addons that port functionality out of the core and once the addon ecosystem is improved further (teaching more users how to setup Travis CI builds for their own multi-platform GDNative projects - godotengine/godot-docs#2815), I believe we will see more and more people wanting to use it for complex projects and performance-intensive code.

Any user that wants to include third-party addon in their project shouldn't have to worry about what kind of source code was used to write that addon. Including it in their project should mean that using it works out-of-the-box with minimal effort.

You can do most of this exclusively through script code, but the code itself is highly complex and C++ is supposed to an official Godot language (as the godotengine team members directly maintain the godot-cpp repository). We should also put forward the minimal effort of improving the C++ workflow for users through the editor, even if it has to separately download most of the required files.

Is there a reason why this should be core and not an add-on in the asset library?:

If the goal is for the manner of addon implementation to be irrelevant to the end-user, then the end-user should not need a separate tool to setup and use the code in an organized and efficient way. I should be able to use Godot as a lightweight IDE, just as it works for C#.

Furthermore, content creators should not have to go through an obscure and difficult process just to organize their addons adequately. It should all be built-in.

[Xrayez]

Given how this proposal ended up quite a read, I should note that instead of trying to compensate for GDNative lack of usability/flexibility, there could be done necessary steps to improve the core first, despite the naysayers. 😛

frequently stopped because of how much of a hassle it is to setup, manage, and use the C++ scripts compared to GDScript and C#

For gameplay programming, I really hope GDScript could be optimized further. As I recall, GDNative was not advertised as a go-to tool for gameplay programming initially, but rather as a quick way to bind third-party libraries without having to recompile the entire engine. So, GDNative's design decisions simply did not cover those gameplay use cases to full extent to improve usability in this regard yet. Godot would become a nice alternative for those with previous experience developing with UE4, for instance.

Having said that, Godot C++ modules are still more usable for me so far, it doesn't really take too much time to compile the entire source (5-15 mins on my machine, depending on what modules are enabled), and Scons only compiles the source that was changed (mostly), improving build time significantly. But if you plan to reuse the modules in other projects, or when you want to cater to user needs more/benefit from open-source community and free CI tools, this is where the pain of dealing with GDNative pays off I guess.

[willnationsdev]

Godot would become a nice alternative for those with previous experience developing with UE4, for instance.

Godot's GDNative in 3.0 was one of the major reasons I ditched Unreal for Godot initially. So yeah, this is definitely a big reason for people who want to just write C++ scripts and use them in a lightweight FOSS engine.

Godot C++ modules are still more usable for me so far

I agree that they are more user-friendly to setup and use at the moment.

it doesn't really take too much time to compile the entire source (5-15 mins on my machine, depending on what modules are enabled), and Scons only compiles the source that was changed (mostly), improving build time significantly.

This is just as true for GDNative except that it always builds far faster since there are fewer files involved. The initial build is faster, and fewer libraries/dependencies are involved usually. If I make a change to the header files or registration code, then the whole modules static library and the final binary all need to be rebuilt.

But if you plan to reuse the modules in other projects, or when you want to cater to user needs more/benefit from open-source community and free CI tools, this is where the pain of dealing with GDNative pays off I guess.

Exactly.

[DriNeo]

I thought that my difficulties while trying GDNative were solely due to my lack of cpp skills.
I was frustated that Godot editor doesn't "talk" with the build script provided by the binding author. In some bindings the build script is just an example who can't work by himself.

Many info required for native scripts compilation aren't related to a particular language. As instance, executable and sources location, required platforms... and could be provided by the Godot editor in some resource. Then the binding author is free to plug his build script to retrieve these infos, whatever if he uses Scons , Cmake or Meson...

[Wavesonics]

Having just spent the past two days trying to get native script c++ working in my project, I can't thumb this up enough.

So much of my experience with Godot so far has been incredibly intuitive and "batteries included". I was flying down the highway that is Godot, and then took a hard turn onto the unpaved road that is native script and my wheels came off!

[mnn]

I have attempted in the past to build addons written in NativeScript C++, but the process is cumbersome and concerns me at scale.

That's an understatement. I remember spending over half a day just to make C++ example running (outdated wiki, missing documentation).

I implemented literally same functionality as in C# before (for purpose of benchmarking) and I spent so much time with the C++ variant, definitely more than 3 times more of what it took for C# (to previously mentioned problems I also had to modify build script [uses SCons which isn't supported by CLion], hit bugs in bindings, it took significant time to figure out how mostly undocumented non-standard memory management works [random crashes in seemingly unrelated calls were really treacherous], undocumented [different than in GDS] behaviour of properties in init/ready and I probably forget a few). It was a giant pain which I don't intend to repeat any time soon. It seems to me Unity + C# + Burst is better choice (nice language, first-class, performance approaching C++) than Godot + C++ (tried only GDNative, not modules, since I wanted a C++ scripts, not modifying innards of the engine). Godot has a lot of work to do in this area.

[ShalokShalom]

I understand the will to comply with standards, I really do.

Consider, for a moment, that on the other hand, if this is all this project would have ever done, it would not even be started. Reading this gives me a headache and I ask myself if it is worth the effort.

I consider the work that has to go into this C++ integration as diametral to its use and can imagine going the route instead, that Godot got popular in the first place: Innovation and progress.

I understand that some people - especially those ones with the money - want to simply use C++, no matter what, so it will happen. I give to think, that you can implement a far superior solution, in the form of F#. F# is an amazing scripting language, clean, looks like Python, brings an ML type system and tons of goodies, plus its already halfway implemented.

I know it will not happen - since again, people are stubborn and stubbornness almost always wins over scientific logic - while I am optimistic that some of you can see its benefits and the benefits for this project.

In the end, even John Carmack switched from C++ to Haskell after couple of decades, so why going back instead of leading the future?

Thanks a lot

[Xrayez]

@ShalokShalom I think there must to be something really innovative in order to overcome the stubbornness, to the point where everybody starts to drop their existing tools within a short period of time, making it evolutionary, I think that's the only way. Are you one of those influencers? If so, start creating something which would be picked up by millions of developers. John Carmack cannot lead the future all by himself. 🙂

I'm not sure if this has something specific to do with C++ alone, you could likely conveniently compile any other GDNative-ready language, it's just that it would make the process more transparent to users. Godot is being written in C++ so that's the most obvious example which was used here in my opinion.

[willnationsdev]

@ShalokShalom I think you are slightly misunderstanding the purpose of this proposal, perhaps?

This would be something that would allow the editor itself to have a built-in solution for compiling GDNative projects. Those who create GDNative bindings would then also be able to create GDNativeBuild resources which would teach the Godot Editor how to properly build and use those bindings into libraries / include those bindings in a built dynamic library / GDNativeLibrary resource.

So, for example, Haskell has a godot-haskell set of bindings for GDNative. If someone created a GDNativeBuild resource and included it with the godot-haskell git repository, then anyone who downloads the addon for Haskell would be able to use the Godot Editor to build the bindings, build the binding library, build a Haskell project that uses the library, and even generate the GDNativeLibrary to wrap it or generate NativeScript files that reference Haskell classes. Ideally would also be able to generate boilerplate Haskell scripts.

The above proposal is more about creating the platform within Godot that would enable users to instruct the Editor on how to build/generate the script code. It effectively raises all languages to the level of C#, to a degree.

As for F# (which I agree is a brilliant language which I've already created editor tools for generating), I already created a separate proposal just for that purpose (#191).

If you bundle this proposal, #191, and #22 together, and if you then also added support for inheritance in VisualScript (don't think that's been implemented yet...), it effectively brings all possible scripting languages to an equivalent level of support within the engine.

The only reason C++ is covered so much in this proposal is because C++ is the only "officially maintained" set of GDNative bindings, and I would expect the Godot Editor to come with a built-in GDNativeBuild resource for that language for the same reason. Which makes sense since Godot itself is written in C++.

[ShalokShalom]

Oh thanks a ton. I got here from another issue and did not understand the proposal here. This seems like a sensible solution.

[realkotob]

Many thumbs up for this.

I've been trying to port over some performance-sensitive code (procedural mesh generation, custom Some IF Statements simulations, etc) from GDScript to C++ and it's been a constant hell.

I think that improving C++ work with Godot would also improve the amount and quality of contributions, since it would make it easier for users to start using C++ and understand more of the engine core.

I can speak for myself, that using GDNative Cpp code is what is making me more comfortable with going into the C++ code of the engine itself.

[kakoeimon]

This is not a solution by far but people coming here may find those tricks usefull.
Put the C++ bindings on a dir that will be on the same level with your projects dir.
For example projects\godot-cpp
A new project projects\cpp_example
This way you will have to generate and compile the bidings only once for debug and release.
Create a dir inside your project that will have all your c++ code.
For example projects\cpp_example\src
Put inside this code the SConstruct and modify it so it will point to the godot-cpp and in the root dir (the src dir that contains it) and to place the dll (or .so or whatever) in a dir bin in your project.
Create a batch file or sh (or whatever) so that you will run and to compile and run your godot project.

Last.. use Visual Studio Code!
Here it is my src dir of the GDExample you can use it as a base.

src.zip

[willnationsdev]

@kakoeimon One of the things I'd like this Proposal to do is enable people to have separately versioned builds of different languages' bindings in user folders so that they can be reused between projects, much like you are demonstrating above. :-)

[kakoeimon]

@willnationsdev I made a repo based on my previous post with some python scripts to make development with cpp a litle bit easier.
I am not posting this as a replacement of your proposal but as an alternative until we have your porposal.
https://github.com/kakoeimon/godot-cpp-tools

[Xrayez]

@kakoeimon the part which generates files reminds me of gdgen package I've written for C++ modules generation specifically. There's also some overlap between writing a transpiler and generating code.

[kakoeimon]

@Xrayez gdgen looks realy cool, thanks.
gdgen looks more mature.

[nonunknown]

This plugin, in my mind is the first step into approaching this: https://gitlab.com/turtlewit/godot_cpp_helper_plugin

[Calinou]

@nathanfranke Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.

[nonunknown]

I'm currently working on a implementation of this issue, First I'm trying with c++ then I'll move to rust

The plugin is installed locally, as any other one:

But, the source files (CPP Header Godot) is installed globally, this way N Godot Projects Can Interact with N Native projects:

This way the user Generate bindings only one time, and any script one time, when compiled the GDLIB and NativeScript is generated inside the current Godot Project!

[Calinou]

@nonunknown You can use OS.get_processor_count() to detect the number of CPU threads on the system. This way, you don't have to ask the user about the number of processor cores they want to use for building.

[nonunknown]

Already using :D But I let user choose in case they want to do other stuff while compiling.

[arunvickram]

@nonunknown This looks great, is there a possibility you could maybe take a look at the Nim language next after Rust?

[nonunknown]

@nonunknown This looks great, is there a possibility you could maybe take a look at the Nim language next after Rust?

Yeah, I'll try to support as many languages as possible, plz give me your discord so you can help me when implementing that support for Nim language

Also for those interested on the implementation of this proposal here's the link to it, its released:

https://t.co/LKt610O1iz?amp=1

[SIsilicon]

@nonunknown Dude.
I'm literally working on the same thing you're doing! I never even had this proposal in mind!
https://github.com/SIsilicon/GDNative-Helper

But now that I see this proposal, I can use this to determine how things should be done in my project. I never thought of the source files being outside of the godot project.

[SIsilicon]

@nonunknown Dude.
I'm literally working on the same thing you're doing! I never even had this proposal in mind!

But now that I see this proposal, I can use this to determine how things should be done in my project. I never thought of the source files being outside of the godot project.

Further more, I looked in your source code and found ProjectSettings.globalize_path("user://"). For some reason, it didn't work for me sometimes. Does it work when the editor is in self contained mode?

[nonunknown]

Wow very interesting, I think we should work on this toguether I'm working on 2.0 which adds support for android building and rust language, plans to add more (nim, and others if requested).

Also globalize_path works only in-editor , which should work fine for anyone!

[SIsilicon]

Could you explain what you mean by "work together"? We are both working on similar projects, so are you suggesting that I help with your project, you help with mine, or merge the two together?

[nonunknown]

merge the projects, get the best side of each one and make it one huge project!

[SIsilicon]

That sounds like a great idea!
How should the merging process go? Who should own the repository?

[nonunknown]

The owner should be GEL (Godot Extended Libraries) just talk with @willnationsdev at so he can add you to GEL organization! Also we should rethink the interface, I personally loved the way you did, but a mean between mine and yours should be good!

The repo:
https://github.com/godot-extended-libraries/native-integration

Talk with willnations here:
https://discord.gg/X3Q6nHUeMS

[SIsilicon]

Alright, gotcha!