GLES2 Batching + CPUParticles2D texture + Label = Rendering issue on iOS

[nebs]

Godot version:
3.2.2.beta1.official

OS/device including version:
iPhone 6 running iOS 12.4.5

Issue description:
The CPUParticles2D texture particles are rendered as a flat color instead of showing the actual texture image.

This happens in a very specific scenario:

1. It only happens when running on an iPhone. The issue doesn't appear when running in Godot.
2. It only happens when there's a Label object present in the scene.
3. It only happens when the new "Use Batching" is ON (in the GLES2 section of the project settings).

Below are some screenshots taken from my iPhone with various settings toggled:

If GLES2 Use Batching = OFF and Label is visible: [CORRECT]

If GLES2 Use Batching = ON and Label is hidden: [CORRECT]

If GLES2 Use Batching = ON and Label is visible: [WRONG]

As you can see the first two cases it works fine. But it breaks if Use Batching is ON and the Label is visible.

NOTE: I tested this exact same project in the official 3.2.1.official release and the bug is not there. So the bug is only introduced in 3.2.2.beta1 with the Use Batching=ON.

NOTE: Again, this bug is not visible when running inside Godot. You must export an iOS project and build with XCode on a device. I didn't try an iOS simulator.

Steps to reproduce:

1. Download the sample project included below.
2. Run the project in the 3.2.2.beta1 beta build.
3. Export the project for iOS. I've included some dummy app icons so you can use those to bypass the warnings. You'll have to enter your own Team ID and bundle identifier. I removed the ones I use.
4. Once exported, open the generated XCode project and build on an iPhone device.
5. You should be able to see the bug right away. Notice that the particles appear as blue squares. The godot robot face is not shown.

To get the particle texture to render properly you can either: 1) Hide the label (just click the little eye icon) or 2) Open the project settings and under GLES2 turn OFF "Use Batching". Then export the project again and you should see the proper particle textures working again.

Minimal reproduction project:
ParticleBugTest.zip

[clayjohn]

@lawnjelly

[nebs]

[Additional context/notes that are semi-related to the issue, but might not be directly related]

I also just wanted to add some more info in case it helps. I've noticed some more issues with regards to using CPUParticles2D in 3.2.2.beta1 with the new "Use Batching" ON. If I use textures as my particles they seem to render very strangely. Sometimes they are rotated 90 degrees, sometimes they flicker on screen, sometimes they glitch out. Basically the rendering of texture particles seems very unpredictable. It works fine in 3.2.1.official and it works fine with "Use Batching" turned OFF.

Sorry I can't provide any more specifics because I'm not 100% sure what all the causes are. My main project is much too complex to break down into tiny chunks. So the project I included in my main post above is only one sample project where I was able to reproduce some issues with the label.

I guess I'm mainly adding this additional anecdotal context because hopefully it helps someone narrow it down. There definitely seems to be something strange happening with GLES2 + iOS + CPUParticles2D (with textures) and the new GLES2 batching feature in 3.2.2.beta1. So it might be worth digging into this specific intersection of components to see if anything stands out. Maybe someone who knows more about the new batching feature might have some insight on this.

I'm happy to test other configurations and settings in my main project if it's helpful. Let me know if you need further information. Thanks!

[volzhs]

I also have an issue with CPUParticle2D on iOS before batching. #32837
does CPUParticle2D have a bug on iOS originally?

[lawnjelly]

I also have an issue with CPUParticle2D on iOS before batching. #32837
does CPUParticle2D have a bug on iOS originally?

Yes there do seem to be some pre-existing (non batching) bugs on iOS with CPUParticle2D, see #38211 also. I'll be investigating this soon (I have some fixes to push first). It may be tricky because I don't have iOS hardware etc, I'll have to use detective work, so may need some help at some point.

@volzhs issue does look related, looks like it may be a state change / shader conditional that is getting out of sync, however the fact it is iOS only is interesting, that narrows it down a bit.

Edit: Looking at the other issue #34637 I think it is unlikely to be a coincidence. I think the batch draw method, the ninepatch and other drawing methods are causing some interference. Possibly the texpixel_size uniform, I have to look at the shader for the particles.

[lawnjelly]

I had foolishly assumed particles were being drawn as rects, but they are multimeshes. Here is a frame diagnosis log for the project, the first item is a single multimesh, the second, a batched set of 5 rects (the text):

canvas_begin FRAME 1140
items
	joined_item 1 refs
			batch D 0-1 MM 
	joined_item 1 refs
			batch D 0-0 
			batch R 0-5 [0] {255 255 255 255 } MULTI
canvas_end

The interesting thing is, that the particles are being drawn BEFORE the text, and yet it is the particles that are drawing incorrectly when the text is present. This is leading me to believe that this (and similar bugs pre-existing) are due to iOS perhaps having a different set of OpenGL state defaults at the start of a frame, or not resetting the GL state where other devices are.

If this is the problem it should be fairly easy to fix by ensuring some consistent GL state at the beginning of a frame.

Well I say easy, but not entirely easy because I don't have one of the devices! 😁 Anyway I'll see whether we can systematically set everything in OpenGL to some default state at the beginning of a frame, that may be the answer.

[volzhs]

@lawnjelly when PR comes for this, I will test it on my iPad. 😄

[lawnjelly]

Incidentally we can get a good idea whether it is the same issue as #32837 simply by replacing the text with a nine patch rect in this minimum reproduction project.

And try with and without batching. If the theory above is correct it should show it showing the same behaviour with a nine patch rect, with or without batching. Not sure yet whether the order matters, particles or other primitives first.

[lawnjelly]

I also just wanted to add some more info in case it helps. I've noticed some more issues with regards to using CPUParticles2D in 3.2.2.beta1 with the new "Use Batching" ON. If I use textures as my particles they seem to render very strangely. Sometimes they are rotated 90 degrees, sometimes they flicker on screen, sometimes they glitch out. Basically the rendering of texture particles seems very unpredictable. It works fine in 3.2.1.official and it works fine with "Use Batching" turned OFF.

Just to confirm, are these other issues also occurring on iOS only, or on the desktop as well? Any issues that occur in the editor are far easier to debug (and indeed may have been fixed already). If they are iOS only as well, they could all be related as part of the iOS 'mega issue'. 😬

[lawnjelly]

Me and @clayjohn have been discussing it, and we are suspecting it could be a UV issue in the particles, I gather you can make a custom shader?

If so if anyone could make a shader that just interpolates red across the U value, and green across the V, or something like that, that will let us know whether the UVs are varying correctly over the particles? 👍

[clayjohn]

Here you go, Please attach this to your CPUParticles2D node and let us know what happens.

shader_type canvas_item;

void fragment() {
	COLOR.xy = UV;
}

If our hunch is right, each particle will be a solid blue square.

[nebs]

Just to confirm, are these other issues also occurring on iOS only, or on the desktop as well? Any issues that occur in the editor are far easier to debug (and indeed may have been fixed already). If they are iOS only as well, they could all be related as part of the iOS 'mega issue'. 😬

Yes all these issues I've only seen on iOS. Sorry I should have been more clear.

I just tested it in an iPhone simulator (running from XCode) and the bug is reproducible there too. So that should be an easier way for you to test, you don't need to build on a hardware iPhone.

None of the bugs I saw were present when running directly in Godot. I've only seen the issues on a hardware iPhone, hardware iPad, and iPhone simulator.

I also tested a desktop MacOS build and the bug was not there either.

I haven't tested Android at all (maybe someone else with an Android can test?). But so far it seems specific to iOS.

[nebs]

Here you go, Please attach this to your CPUParticles2D node and let us know what happens.

shader_type canvas_item;

void fragment() {
	COLOR.xy = UV;
}

If our hunch is right, each particle will be a solid blue square.

I added this shader to CPUParticles2D. Here's a screenshot of it running on my iPhone 6:

The squares appear as purple, almost like it's 50% blue 50% red.

UPDATE

I had a color gradient on the CPUParticles2D which was fading out the particles. I removed that gradient to make it more clear.

So here's the result with the shader and a visible label:

Here's the result with the shader and a hidden label:

[lawnjelly]

Great work @nebs, really appreciate it. We are getting closer! So we now know it is the UVs, and also isn't purely a hardware issue.

I have tested on Android and couldn't get it to occur there.

Given that it appears in emulator that is pointing the finger towards a different state reset policy on new frames in iOS.

[lawnjelly]

Just a wild hunch, try changing the last letter of the text and see if the color changes. One idea @clayjohn suggested is that it is a 'leftover' UV from the last thing drawn (we are trying to work out why the UV would be that particular value).

[nebs]

Ok so here's another strange update. So I noticed that in my game I'm seeing weird CPUParticles2D issues even when I hide the label. I also noticed in my game I use tilemaps. So I tried adding tilemaps to this test project and I was able to reproduce some more weird rendering.

This time the particle textures are rendering, but it appears the UV is offset a bit which causes it to look off.

Here's the updated sample project with a new tilemap that you can toggle:
ParticleBugTest_tilemap.zip

Here are screenshots from my tests. In these tests the Label is hidden. Only the tilemap and the CPUParticles2D are shown. The tilemap is behind the particles.

If UV Shader applied and TileMap is hidden. [Looks Ok]

If UV Shader applied and TileMap is visible. [Notice the colors look more desaturated, like the UV coordinates are offset]

If shader is removed and TileMap is visible [Notice how the little godot guy is cutoff, suggesting again the UV is offset]

I'm not sure if this is the same issue or something else. I'm happy to open a separate issue to keep things organized. But I figured this might be helpful here.

So it looks like there's two ways to generate two slightly-related bugs:

1. Put a label behind or in front of a CPUParticles2D
   or
2. Put a TileMap behind a CPUParticles2D

[lawnjelly]

I think they may be both manifestations of the same bug. @clayjohn noticed there was actually a slight gradient in your earlier example, and we were wondering whether there was a relationship with the UVs used to render the font for the text.

[nebs]

Just a wild hunch, try changing the last letter of the text and see if the color changes. One idea @clayjohn suggested is that it is a 'leftover' UV from the last thing drawn (we are trying to work out why the UV would be that particular value).

I changed "HELLO" to "HELLA" and the color looks the same. But maybe there's a very subtle shift that I don't notice. Is that what you meant by changing the test? Or something else?

[nebs]

Actually you're right. If I change the label to something much more different the color changes more drastically.

[lawnjelly]

Yup that's it. I checked in gimp and they may be subtlety different, it is harder to tell because of the rotation.

Ah fantastic with the second screen shot, that really shows it! 😁

I'm off to bed now but I really think we are pinning this down, @clayjohn is looking at the glVertexAttrib calls to see if we are forgetting to reset. Many thanks and will catch again in the morning!

[clayjohn]

@lawnjelly Here is one candidate:

godot/drivers/gles2/rasterizer_canvas_base_gles2.cpp

Lines 644 to 647 in c8ea779

	if (p_uvs) {
	glVertexAttribPointer(VS::ARRAY_TEX_UV, 2, GL_FLOAT, GL_FALSE, stride * sizeof(float), CAST_INT_TO_UCHAR_PTR(uv_offset * sizeof(float)));
	glEnableVertexAttribArray(VS::ARRAY_TEX_UV);
	}

This is in _draw_gui_primitive which is used in many of the drawing commands (including rect drawing). It doesn't have an option to disable the uv array when uvs are not used like other functions (draw_polygon for instance):

godot/drivers/gles2/rasterizer_canvas_base_gles2.cpp

Lines 440 to 447 in c8ea779

	if (p_uvs) {
	glBufferSubData(GL_ARRAY_BUFFER, buffer_ofs, sizeof(Vector2) * p_vertex_count, p_uvs);
	glEnableVertexAttribArray(VS::ARRAY_TEX_UV);
	glVertexAttribPointer(VS::ARRAY_TEX_UV, 2, GL_FLOAT, GL_FALSE, sizeof(Vector2), CAST_INT_TO_UCHAR_PTR(buffer_ofs));
	buffer_ofs += sizeof(Vector2) * p_vertex_count;
	} else {
	glDisableVertexAttribArray(VS::ARRAY_TEX_UV);
	}

[lawnjelly]

Thanks to #38346 for pointing me along an avenue to check, I've been looking at the CPUParticles2D::_update_particle_data_buffer() and it turns out that ptr[8] at least is containing nan values.

I'm not sure whether this goes to the shader yet, but it is possible that the particle sending duff data could trigger strange errors on some hardware.

Yes, I think it is getting sent as part of the vertex format. I'm guessing the nan is because it is uninitialized. This may very well be causing a problem...

Edit : Ah, may be false alarm, as it is compressing the color into a float, hence showing up in the debugger as nan.

CORRECTION
Now I'm debugging and stepping through on the correct particle project (!) 😁 I'm getting a whole host of nans being stored in the particles on some frames. I changed the set_amount function to memset the Particle with 255 to test this out. It does seem that it has uninitialized values in it on some frames.

[nebs]

One other thing I noticed with CPUParticles2D is that textures appear rotated 90 degrees when running on iOS and GLES2. This was happening even in older versions of Godot before the batch rendering stuff. I'm not sure if it's related to this bug or not but I thought I should mention it here in case it's useful information.

If this is completely unrelated I can file a separate issue later.

[lawnjelly]

I'm not sure if it's related to this bug or not but I thought I should mention it here in case it's useful information.

It could well be related, I think it is highly likely this whole class of bug is the same, and exhibits in different ways. I'm currently of the opinion that although it appears when you switch batching on, the bug may not be IN the batching... i.e. it is the same bug as the previous ninepatchrect bug (which has exact same visual results), and it was pre-existing and just happened to be hidden in some situations previously (i.e. it is likely to be a bug in the multi-mesh / particle code).

[nebs]

Thanks for the quick response and work on this!

I tried out #38378 in my local fork and built my test project under 3.2.2.beta.custom_build.8426ed265. Unfortunately it seems like the original bug is still there (the particles appear as a flat color when a label is present) when testing on an iPhone.

I'm not sure if that PR was meant to solve that particular bug so if not then disregard this message, I just wanted to report my findings.

I should note that I used the export template from 3.2.2.beta1 because for the custom build there's no download for it, so I'm not sure if that has any impact.

Let me know if there's anything else I can do to test it out. Thanks again for all your work on this!

[clayjohn]

@nebs You need the updated export templates to test the changes. If you use the 3.2.2.beta1 templates, then the code that runs on your device does not include anything after 3.2.2.beta1.

That being said, #38378 is unlikely to have caused your particular issue, but we have a few ideas about what it might be. Stay tuned. :)

[nebs]

@nebs You need the updated export templates to test the changes. If you use the 3.2.2.beta1 templates, then the code that runs on your device does not include anything after 3.2.2.beta1.

Oh I see my bad, I wasn't quite sure what export templates were for. Thanks for clarifying.

[clayjohn]

@nebs No problem, its good for you to learn now because we are going to need you to do a lot of testing. :P Neither of us have any IOS devices to test on.

[nebs]

@nebs No problem, its good for you to learn now because we are going to need you to do a lot of testing. :P Neither of us have any IOS devices to test on.

Haha. I'm definitely happy to help test this out on my iDevices. But in case you guys want to test it quicker, I was able to repro the bug in an iPhone simulator too so you should be able to test it even without an iOS device.

[lawnjelly]

@nebs referring to the images you posted in:
#38318 (comment)

I'm wondering if the offset UVs are the same or related to the UVs used to render the rect in the tilemap (the thing that looks like a bit of hay). This is what appeared to be happening with the text and the the blue color.

It could be somehow reading the UVs from the previous VB (if the UV attribute call failed), or there could be an offset applied in the shader. It could be that when particles do work in iOS, it is only by accident (i.e. the UVs or offset coincidentally contained range 0 - 1).

Stream of gibberish debugging thought, probably only of interest to clayjohn...

Actually now I look at it, there are no UVs in the particle vertex format. The gravy thickens! 😁 Where do the UVs come from if TEXTURE_RECT is disabled?

It isn't clear as there are no comments in the source, but INSTANCE_ATTRIB_BASE + 4 location is where the UV is specified in the shader, and the glVertexAttrib calls for that location look suspicious.

I haven't worked out where the UVs are being set, as ATTRIB_BASE + 4 is being used for custom data in multimesh. Could also be an attribute numbering problem, assuming a certain location instead of querying?

glBindAttribLocation is deprecated on iOS it seems - that's a possible:
https://developer.apple.com/documentation/opengles/1617262-glbindattriblocation

Although I'm amazed it's working at all if that was it...
...
Ok, turns out the UV coords are originally coming from CPUParticles2D::_update_mesh_texture(). It gets converts from the particle array into some verts more friendly for rendering. The basic mesh is uploaded once, and only the attributes that are needed at runtime are overrided in a loop over the particles when rendering.

There is an if statement there for an atlas_texture, so it would be nice to confirm the original values going in are correct, but the issue has existed before the PR which added that (february 2020?).

_update_mesh_texture() could be wrong values (unlikely), then it calls
mesh_add_surface_from_arrays(). This function has a lot of things that could potentially go wrong on different platform, regarding compression, sizeofs etc.

The other possibility is a problem reading the values. Reading the values wrong or a wrong state still seems most likely to me, because if there was a problem uploading the mesh_texture thing, you would have thought it wouldn't work at all. And likewise for precision etc issues in the shader.

• I possibly may have it. Looking at the glVertexAttribPointer call for the UVs (index 4), I noticed the type being passed was 0x140B, which wasn't on the standard list in the glad.h file. It turns out that it is defined in rasterizer_storage_gles2.cpp, and the value varies according to whether running on desktop or not. I suspect that what might be happening is on iOS, it is not defined GLES_OVER_GL, and it is using the wrong constant, or vice versa (it is defined when it shouldn't be).

#ifdef GLES_OVER_GL
#define _GL_HALF_FLOAT_OES 0x140B
#else
#define _GL_HALF_FLOAT_OES 0x8D61
#endif

If this is indeed the case then the call might very well fail, and it may end up using whatever UVs are currently in the pipeline (the ones from the previous drawing!). This sounds very promising, fingers crossed! 😁

Is also possible some iPhone flat out don't support half float in GLES2 in vertex format:
https://developer.apple.com/library/archive/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/TechniquesforWorkingwithVertexData/TechniquesforWorkingwithVertexData.html

OpenGL ES 3.0 contexts support a wider range of small data types, such as GL_HALF_FLOAT and GL_INT_2_10_10_10_REV. These often provide sufficient precision for attributes such as normals, with a smaller footprint than GL_FLOAT.

If this is the case, we might consider changing the default for particles to use GL_FLOAT, even if slightly larger, it may be widely supported and I'm reading some reports that half float is very slow on some hardware.

[lawnjelly]

Ok I have a potential fix:

https://github.com/lawnjelly/godot/tree/ios_particle
lawnjelly@67d6680

I'm not 100% sure it will fix it, but it is looking likely, so I would really appreciate if someone with the problem could add this snippet and recompile to try. Note that you would need to compile the export template for iOS for it to actually do anything (otherwise it will only be compiling the editor).

If this does fix it, it will likely fix all the associated iOS particle problems (ninepatchrect etc).

However I'll need to discuss with @clayjohn how to do a more sensible fix for a PR - it's a bit of a chicken and egg problem. The decision to compression is made in the visual_server, before it gets to the renderer, but only the renderer knows whether the feature might be supported (aside from large selections, like all IPHONE_ENABLED devices). Half float may be available in GLES3 but not GLES2 for example.

[nebs]

Thanks for continuing to work on this! It sounds like we're getting close.

Note that you would need to compile the export template for iOS for it to actually do anything (otherwise it will only be compiling the editor).

Is there a simple guide online for how to create the .tpz export template file for iOS? I tried following this guide but it just results in a bunch of .fat.a files.

How do I generate the .tpz file that Godot is asking for? I tried using the misc/dist/ios_xcode folder and replacing files, zipping it, renaming it to iphone.zip and copy pasting an old template folder but I got a generic error that the template was wrong. I'm pretty sure I messed something up because the copy pasting of old templates and replacing files seems like a hacky approach. I'm sure there's some simpler method that I'm missing. I couldn't find any official Godot docs on this but maybe I missed a section. I'm fairly new to building the engine so sorry if this is obvious.

I'd love to help out to test this but I'm having trouble finding a simple workflow for building + exporting for iOS from source. Any help is welcome! (Sorry to pollute this issue with this, but I think it would be helpful).

[clayjohn]

@nebs that guide is the most up to date resource on building for iOS. See also the linked guide for exporting your game https://docs.godotengine.org/en/3.2/getting_started/workflow/export/exporting_for_ios.html#doc-exporting-for-ios the instructions shouldn't differ from what it says. You won't be creating your own tpz export file as you are doing your own custom export. You'll have to follow the instructions for linking your custom executable to xcode instead of the executable contained in the tpz export file

[akien-mga]

The docs are indeed missing the step to pack the compiled libraries into the iphone.zip archive for the templates.

Here's how it's done for official templates: https://github.com/godotengine/godot-build-scripts/blob/master/build-release.sh#L175-L187

The simplest way would be to unzip the iphone.zip from your 3.2.1 templates, replace the .a files with the ones you compiled (make sure to use the same name as in the iphone.zip, the compiled ones may differ slightly) and re-zip it. You can then put it in your 3.2.2.beta templates folder for use with a custom 3.2.2.beta macOS build from the same commit.

[nebs]

lawnjelly@67d6680

Good news, it appears this fixes the bug! I applied this change to my fork and the particles appear to render correctly on my iPhone even with the label and tile map present:

But it would be great if someone else could verify on their end too. The only reason is because to get the export templates to work I had to do some Frankenstein combination of following the docs and applying some of what akien-mga mentioned and also having to rename some files to fit. I'm pretty sure I did it correctly because it exported without errors, but I'm just worried about an accidental false positive.

But assuming I built things correctly it appears lawnjelly's commit is the correct fix for this bug. Good work guys!

[lawnjelly]

That's fantastic!! 😁
Actually my unofficial reaction was 'thank **** for that' 😆

[Ranoller]

Well, i was reporting same bug but.... thats fast!!!. I have other problem with Gles2 batch and render of the rectangle of the camera in the editor, and the Z index inheritance.... will open other issue for that when i can archieve a minimal reproduction project.

Interesting thing that the bug that was fixed here is that it was gone if you changed the Z index of the particles node. I suppose that it was because it doesn´t batch if Z index is different...

[lawnjelly]

Just a note that this bug has an experimental fix available in the latest 3.2.2 beta. Would appreciate if any of you guys with the issue can test, full instructions are in #38441 (it's slightly more involved because of my incompetance! 😀 )

[nebs]

Just a note that this bug has an experimental fix available in the latest 3.2.2 beta. Would appreciate if any of you guys with the issue can test, full instructions are in #38441 (it's slightly more involved because of my incompetance! 😀 )

Hey, I just tested 3.2.2.beta2. I can confirm that adding gles2/batching/disable_half_float=true to my project fixes this bug on iOS. I tested with both the tilemap and the label and in both cases the particles render correctly. Good job!

[HEAVYPOLY]

I think I might have an issue related to this, not sure if it should be separate or piggyback.
On iOS, UVs seem to be squished and offset. Shader here uses UV.x as color.
Here is a comparison between iPad(squished and offset) and in editor(correct):

I've tried "use half float" option on and off, no change. Also tried with batching on and off, no change.

Using MeshInstance2D node and shader for black and white color

[clayjohn]

@HEAVYPOLY what version of the engine are you using?

You may have to add gles2/batching/disable_half_float=true to your project file manually like in the above comment.

[HEAVYPOLY]

@clayjohn 3.2.2.rc1. I added the gles2/batching/disable_half_float=true to my project manually in the Godot.project file, no change. I just tried with GLES3 and it's working correctly

[lawnjelly]

With the settings rename (#39068) the setting in RC1 is working and it is now:
rendering/gles2/compatibility/disable_half_float

If the disable half float setting doesn't cure it (and the correct export template is being used) then this needs a new issue (preferably with the hardware version, the shader, and a minimum reproduction project).

[clayjohn]

This issue can be closed in light of #38318 (comment)

Further half_float bugs are being tracked at #38441