Optimization: draw polygons front to back in the GPU when possible

[simias]

Since the PSX doesn't have a z-buffer all the primitives are drawn from farthest to closest. This means massive amounts of overdraw as many polygons are drawn only to be hidden by a closer one later on.

This is basically the worst case as far as the OpenGL renderer is concerned, generally we'd prefer to render from front to back and ignore hidden pixels instead invoking our shader millions of time for fragments that will never make it to the screen.

Implementation

This shouldn't be too difficult to implement: when we receive primitives in the GPU we could store them in the vertex buffers in reverse order. We'd also need a way for the GPU to test which parts of the screen have already been drawn, I can think of two simple ways to do that:

• The Z-buffer: we can assign arbitrarily increasing values to the z-coordinate of the primitive (since we don't have the real Z-coordinate available at this point) and then enable the depth test in OpenGL. Not very elegant but it should work. One advantage of this method is that if we decide to implement a real Z-buffer later on (by recovering depth information from the GTE) we might be able to reuse this code directly by plugging the actual depth value instead of a dummy value. It's a bit early to say for sure though.
• The stencil: probably cleaner, at least from a conceptual standpoint: we enable the stencil test and we set the stencil flag after each draw. The stencil test will reject subsequent draws to the same pixel in the framebuffer. One potential difficulty is that we'll also probably need the stencil test to emulate the mask bit so we'll have to make sure we can make the two modes cohabit. Maybe we can use two stencil buffers? Or a single stencil with two bits per pixels? I'm not sure...

Caveats

There are a few limitations to this optimization however: semi-transparent polys will still have to be drawn back to front after all the opaque primitives for tranparency to work as expected. We'll probably need two passes, one for the opaque polys front-to-back and then one for the semi-transparent polys back-to-front. Maybe we could store the opaque vertex starting from the end of the vertex buffer and then decrementing while transparent primitives would be stored at the beginning and incrementing as it's currently done. Or we could use two buffers, not sure which is best.

Also we can't use this optimization when the "test mask bit" and "set mask bit" options are enabled since the order of the draws become significant for the end result regardless of the "depth" of the polygon. If it's the case we need to render everything in "native" order.