Apply correct fog+blending in shaders

[rreusser]

This PR applies fog to most of the shaders (exhaustive list to follow before merging).

To do:

• Decide: gamma correct or don't gamma correct
• Possibly decide now which fog function to use. Applying a power after the exponential removes the discontinuity at the fog start and seems defensible since real fog doesn't start at an arbitrary position and contain this discontinuity. By discontinuity, I mean the C1 discontinuity at x=5000 where the function goes from clipped to zero to increasing: https://www.desmos.com/calculator/x5gopnb91a
• Apply the functions to remaining layers (circles? background-pattern?)
• See if we can simplify the #if defined( FOG ) && !defined( RENDER_TO_TEXTURE ) conditional

Notes/explanations:

• Un-premultiplying the alpha, then blending fog, then re-premultiplying seems necessary to achieve correct blending. Not sure if that’s the cleanest to undo it and redo in the shader, but it does yield the correct results. The following shows a line-pattern with a transparent texture, correctly blending only as a result of the fog_apply_premultiplied function.

• The conditional #if defined( FOG ) && !defined( RENDER_TO_TEXTURE ) got the terrain/no-terrain behavior correct (while checking for TERRAIN did not, for reasons I didn't fully figure out), but it’d be nice if this condition weren’t so verbose since it’s repeated a lot

• I think the fog depends on the aspect ratio… need to revisit but perhaps outside of this PR

• most of the layers (including: heatmap, excluding: symbols and a couple exceptions like circles which I just haven’t addressed yet) work… correctly, I think?? :party-face: Here’s quite a few of them at least…

• heatmap works differently, so I just applied fog opacity while accumulating the kernel density estimation 🤷 Basically works like expected, I think?

[rreusser]

Here's a brief rundown of the effect of gamma. I think I was fairly wrong about it. It wasn't the source of the antialiasing/opacity problems (it was premultiplied alpha). As for the difference in color, it's quite noticeable but somewhat arbitrary, depending on the fog settings and color. The overall lightness is different though.

I tried fog opacity normal/squared as well as gamma correct/incorrect. The set of four isn't particularly interesting. Instead, two roughly equivalent things (since gamma correction is very roughly a square root) are:

• Gamma-incorrect color mixing + regular fog opacity
• Gamma-correct color mixing + square of the fog opacity

Here's a gif which flips back and forth between these two. The lighter one is with incorrect gamma, and the darker one is with gamma-corrected application of fog. It's subtle, but I think this is the kind of thing where correct gamma prevents annoying side effects like unexpected/unnatural changes in brightness when blending.

[karimnaaji]

Looks good! Testing the gamma in more details now.

[karimnaaji]

Are you positive we need / p.w? I thought u_cam_matrix would ensure w == 1.

[rreusser]

Good point. I guess as long as it's strictly affine, that is the case. I try to at least add a comment noting it's a stated assumption.

[karimnaaji]

I'm in favor of leaving it gamma correct. The best test case is black fog on a white style, making these transitions a lot darker than they should be:

without GAMMA_CORRECT

with GAMMA_CORRECT

Which acts exactly as the issue we could see with a simple gradient:

Concerning cost, the simple pow function are good enough here, I don't think we should use the more accurate versions (It's hard to notice a difference using them), and may only be a critical choice for complex lighting.

One nit: we don't have a linter in shader code, but it seems like snake_case is prevalent for function names/variables.

[arindam1993]

+1 for adding gamma correction on the fog, the greater spread definitely adds to the effect for me.

[rreusser]

@karimnaaji in the black-to-white gradients above, is the top one gamma-incorrect and the bottom gamma-correct? The bottom one looks unreasonably light, on average.

[karimnaaji]

The medium part of second gradient is at (0.73,0.73,0.73) srgb, which is exactly (0.5,0.5,0.5) linear.

I still had the shader around to generate these:

out vec4 out_color;
uniform vec2 resolution;

@color3
uniform vec3 color_1;
@color3
uniform vec3 color_2;

vec3 linear2srgb(vec3 color) {
    return pow(color, vec3(1.0 / 2.2));
}
vec3 srgb2linear(vec3 color) {
    return pow(color, vec3(2.2));
}
void main() {
    vec2 uv = gl_FragCoord.xy / resolution;

    if (uv.y >= 0.5) {
        out_color = vec4(1.0);
        vec3 color = mix(color_1, color_2, uv.x);
        out_color = vec4(color, 1.0);
    } else {
        vec3 color = mix(srgb2linear(color_1), srgb2linear(color_2), uv.x);
        out_color = vec4(linear2srgb(color), 1.0);
    }
}

The bottom one was was generated with gamma correct mixing between black and white, which is exactly what we do with fog now.

While looking at that I remember comparing the heatmap as well, which is another place we'd benefit from the same process (top is gamma corrected):

[rreusser]

The bottom one sort of looks like it has a dark ring where the opacity fades. This might be too much of a breaking change, but I do think it is objectively better by a small margin.

[arindam1993]

Nice work @rreusser, I like the gamma correction

[arindam1993]

+1 for adding gamma correction on the fog, the greater spread definitely adds to the effect for me.

[arindam1993]

inlining named chunks via a pragma can be applied here too

[rreusser]

@karimnaaji @arindam1993 I've worked through the feedback and made the requested changes, apart from things maybe handled in subsequent work. I checked and ensured that the color picker color (and not some gamma-dependent variant) is what makes it though the mixing function and onto the screen.

I haven't added render tests, which I think are best handled later, once we commit to things. I don't currently have other additions.

EDIT: Hold on, circles. Sorry, dog is barking, gonna need to handle that in just a few minutes.