An Organisational Framework for Automated Testing

[jejay]

The lack of any policy or organisational framework for automated testing at the Godot project should be discussed.

I don't remember where, but one of the first things I read somewhere about Godot, is that its pull-request-accepting-policy is move fast and break things. Just today I have seen a tutorial where someone found a bug/regression "live". The point isn't even whether it actually was a bug, but even the lack of confidence in Godot's stability that an experienced tutorial creator immediately called it a bug says something. A friend of mine works at Volkswagen research on product demo/visualisation, they use a lot of Unity. Just from following the community with taking the goggles of a commercial executive, I don't feel comfortable to recommend Godot to them, based on "perceived" stability.

And don't get me wrong, I don't want to complain about the code itself. It looks awesome from my perspective, and that also it what I hear from others playing with the engine code. Its just that some instability is to be expected without any automated testing.

And even when it comes to accepting pull requests. There are almost 400 open PRs, many of them labelled "needs testing". I guess then that means "playing around with it and checking whether it works". How is that not a nightmare. Every green arrow speeds up this process a little bit and also gives the contributor confidence. I have already read it somewhere else and also thought it myself: "Okay I can try to contribute, but how am I gonna be sure that I don't break something?".

There are already some issues about the technicality of testing (e.g. #2641 [2015!] #17408) but I don't want to make this issue too much about it. I know that it is not trivial to test non-deterministic things, but it also is not impossible. Some suggestions are already in these issues and when I started a discussion on discord, vivid ideas were spawning quickly. And I also think there are enough deterministic things to start with. (One note to add; in the discord chat many mentioned that they would like to see performance benchmarks/profiling in an CI-like manner).

There is already the godot-tests repository. As far as I understand, it was meant for testing the engine in GDScript. It is almost empty and I think no-one can expect that such a repository takes off on its own. Writing tests is annoying and if your are the only one doing it, it is even pointless. It needs to be enforced and embraced by the very top of the maintainers. And here I come to the very point of this issue. The further this awesome project grows the stronger I suggest you should think about adopting serious policies for testing, that are enforced and embraced by the project lead devs themselves.

Adding features is more fun obviously, and I too want to see Vulkan, occlusion culling, etc. It would be a necessary investment of time and could even slow progress down. But it could be reimbursed quickly by additional corporate money contributions of companies that otherwise won't be confident enough to use Godot. You should, at the very least, have that discussion.

[BastiaanOlij]

Anything regarding testing that can be automated, especially as part of the CI model, would off course be welcomed.

The problem ultimately is that there is only so much you can automatically test in a highly visual oriented environment as Godot. A lot of the types of enhancements that end up being added can only be tested by people actually using them, a lot of problems only present themselves on specific hardware or in certain drivers or certain OSs. When these are large new projects like the new GLES2 renderer or the up and coming Vulkan renderer, you won't know what sort of test scripts you need to build until most of the work is finished and merged, do this stuff to early and you just increase the workload exponentially.

This is why Godot uses the branching system with an alpha->beta->release work flow. Master is considered unstable and not suitable for production, this branch constantly will have new features added that require testing, they are merged so they can be tested, and yes that means merging broken PRs from time to time and introducing regressions into the system.

As we're close to a point that a new major release is to be made the project goes into alpha, that means no more PRs are merged with large feature changes, only small ones.
Some weeks later we go into beta which means no more feature PRs are merged at all, only bug fixes, and once stable release time comes along a new branch is made for that stable release.
These stable releases now purely receive bug fixes. It is these stable branches that would get the most bang for buck if we're able to introduce automated testing here imho.

This ensures that the stable releases contain a minimum of bugs and will improve with every patch with little risk of regression.

But the master branch will break, it is expected to break and no matter of automated testing is ever going to change that due to the nature of the changes that get merged in. Automated testing will likely just end up giving a lot of false positives as functionality is replaced or changed and you need new testing scripts to test the new truths.

[BastiaanOlij]

So yeah, bottom line, don't judge the stability of Godot on the master branch. Judge the stability of Godot on the releases that are deemed stable and meant for a production environment.
Youtube authors (including myself) often work off of master because we want to showcase new and upcoming functionality.

[mindinsomnia]

Personally I agree, automated testing is a great idea and should be considered. Long term it could speed up development.

[jejay]

Yeah actually, the tutorial I spoke of was based on a pre-stable version of godot. But if you just look for "is: issue is:open stable" there are almost 1000 issues and almost everyone has the red bug label. I also don't believe that the people I have heard complaining about finding bugs daily on their day-to-day project work, use a version of the master branch. That would be then a ridiculous complaint to make. The concept of master vs stable should be quite clear.

Not to forget also the long run benefits in the development process wrt to pull requests.

When it comes to the "automatibillity", I strongly disagree that it is not possible. I agree that it is not straight forward trivial. Blender uses a mixture of unit tests and visual output regression tests. The examples you mentioned are especially a good point for automated testing: you usually develop on one platform, but it should be tested on a variety of platforms. Unity has older articles about their core-testing framework (https://blogs.unity3d.com/2012/05/08/testing-unity/ and https://blogs.unity3d.com/2018/01/04/compiler-test-framework/). One quote from the older article: "Unity is one of the most testable products I have ever worked on." There is also an interesting presentation "The tools we've built to test a game engine" (https://www.infoq.com/presentations/tools-practices-game-engine/) that also covers stuff like profiling and graphics testing.

There is no reason why even a test coverage of close to 100% on all platforms shouldn't be possible. I agree though that it would be a little bit overshot. But the default state for godot seems to be not doing it at all and to me that seems to be just as extreme as the other one.

The biggest problem I see is that it requires a little more thought and creativeness than e.g. testing a command line tool would need. And that this needs to come either directly from core devs or at least should be tightly supervised by them, as this can't be just a random community contribution -- because of its politics involved.

Another issue of course can also be, that basically everything will be teared down and revamped for 4.0. Then of course, wait for 4.0 to start automated testing. But if then the argument is, everything will be teared down for 5.0, then hmmm...

[groud]

I will close this as a duplicate of #2641.
Please continue the discussion there :)

[Calinou]

For those stumbling upon this issue from a search engine, note that unit testing using doctest has been integrated into the Godot development process.