[PBR extension] Add rendering mode for transparency

[bghgary]

@mlimper

Also known as alpha mode or blend mode.

@erich666 says this about transparency from #697 (comment)

Transparency is another area that is not addressed in the specification. Again, it might be fine to ignore this whole area for now, at best providing a traditional "blend" transparency factor and stress that the model presented is for basic opaque materials and transparency is just basic alpha blending (no Fresnel involved, for example). Similarly, clear coat materials (car paints), anisotropic materials (brushed metal, carbon fiber, etc.) are not included, which I think is fine for now. A basic, solid PBR material will be reasonable for 95% of materials for 95% of users, and can be expanded in future extension proposals.

Here is @cedricpinson's implementation:
https://github.com/sketchfab/Unity-glTF-Exporter#transparency

In order to differenciate between transparency types in Unity, an extra metadata is added to the material.
It allows to know which blendMode is used and the cutoff value.
For now, blendMode valid values are alphaMask and alphaBlend.

"extras": {
    "blendMode" : "alphaMask",
    "cutoff" : 0.5
},

I believe it is important to specify how to interpret the alpha channel of the PBR diffuse / baseColor texture. At the minimum, we need to document the expected implementation behavior in the spec.

This proposal is now out-of-date, please see #822 (comment) for the new proposal.

I propose we add the following two material parameters to the spec. Note that the rendering mode represents the usage of transparency instead of how the blending works technically, similar to what Unity3D is doing with their standard shaders.

alphaRenderingMode

Value	Description
Opaque	This mode ignores the alpha channel of the texture and renders as if alpha is 1. Use this mode if the texture does not have an alpha channel or if the alpha channel is for something other than transparency.
Cutout	Also known as Mask, this mode allows for hard edges between the opaque and transparent areas. The rendered result is either fully opaque or fully transparent depending on the alpha channel of the texture and the specified cutoff value. The implementation determines anti-aliasing support if any. Use this mode for materials that simulate geometry such as leaves or wire fences.
Transparent	Also known as Blend, this mode is for transparent or partially transparent realistic materials. Implementations should combine the source and destination colors using standard alpha blending equations and account for opacity Fresnel to increase the opacity at silhouettes. Use this mode for materials such as clear plastic or glass.

alphaCutoff
This value specifies the cutoff threshold when in Cutout mode. If the alpha channel of the texture is greater than this value then it is rendered as fully opaque, otherwise, it is rendered as fully transparent. This value is ignored for other modes.

[mlimper]

Thanks for the summary, and for the proposal.

Transparent

Maybe this could be called "Blend"

account for opacity Fresnel to increase the opacity at silhouettes

Here, we'll probably have to specify more precisely how this should be implemented

[erich666]

Having a cutoff (alpha test) value is good, it's common practice for foliage, grass, and other cutouts so that we can avoid needing to sort. If we're going to really support coverage alpha well, this opens up a new area to discuss, unfortunately. But, as Cedric notes, it's common practice to include an alpha test, so here we go.

With alphaRenderingMode, the main reason to have this is so that we can specify that an object is a cutout, that alpha itself should not be returned (it will be ignored if it is) and that the blending mode is off. Opaque vs. Cutout makes sense for shader control: check whether alpha testing is used. There's a fourth mode that should probably be added: CutoutBlend, or whatever you want to call it. Texels with alphas at or very close to 0.0 are discarded (and so save ROP costs), otherwise they're treated as semitransparent textures that use alpha blending. This is the fourth combination of alpha-test on/off, alpha-blend on/off, i.e., it's both options on. Useful? Well, it is for decals, but that's a separate topic. I can't swear it will be, but it seems pointless to leave out this fourth mode. In the shader the two uses of alpha are separate bits of code. Alpha blend entirely controls the ROP. I would put this fourth mode in for completeness, it costs nothing.

Another question to answer is whether an alpha-blended surface writes to the z-depth. If it does, then this fourth mode is actually very important, as you usually don't want to write to the z-depth if you're also alpha blending, as you get some really bad popping if the sort order changes. I'd recommend no z-depth writing if alpha blending is on, z-depth writing on if blending is off (e.g., for cutouts you definitely want it on).

As far as "or if the alpha channel is for something other than transparency" goes, be careful. If we want to ignore such data, great, we're done, and they're doing something else with their data that has nothing to do with PBR. People can stuff whatever they want in any channel they like. However, if they put roughness in the alpha channel, we can't access it currently. If we want to support this, OK, but then you'll need to add something that says "roughness is in this texture, in this channel", which I don't believe the specification has (maybe it does? I've lost track). It's assumed right now, AFAIK, that each element that could be textured uses its own separate texture. It's fine to go to a more flexible approach, mapping channels as you wish, but I don't think this is currently something a user can set.

account for opacity Fresnel to increase the opacity at silhouettes

Here, we'll probably have to specify more precisely how this should be implemented

I was pointing this out as one of the (many) transparency effects we are not addressing in this spec, and that's fine to not solve right now. I don't think we should start worrying about Fresnel and how it might boost the opacity of the fragment, since that would be a hack anyway - blending is about coverage, and we mostly like to imagine it is like a filter. We don't have programmable blending, so in the ROP we can't filter by the color of the glass and add in any specular reflection. There are various tricks that can be done, but we then leave the path of traditional forward rendering. See this post for a way to do it with weighted, blended OIT, for example, which could be applied to sorted transparency, at the cost of an extra buffer holding the opaque scene and getting filtered in the shader.

Me, I'd leave it at:

• alpha value for blending (really, coverage, like a gauzy fabric) - blending implies sorting and implies (to my mind) that z-write is off
• alpha test, for cutouts (blending off)
• alpha mode for choosing whether to blend and whether to test, independently, four states.

All this said, I'm not an expert in this area. For example, I certainly haven't though through WebGL's alpha to coverage support, commonly used for cutout antialiasing.

I should also mention we should be careful with a reference implementation, making sure it uses the proper WebGL texture access mode to avoid black fringing on cutouts (see the Epilogue of this article). PNGs are not premultiplied, but you need to premultiply before filtering the samples (e.g. bilinearly interpolating, or making mipmaps for that matter).

[bghgary]

Thanks @mlimper and @erich666 for the comments.

I have been discussing with my colleagues and prototyping in the past two weeks and we have come to the same conclusion that alpha should just represent coverage (or presence as some of us are calling it) for now. This means that PBR materials will only be able to represent “opaque” materials initially. An extension can be added to support opacity Fresnel and other "physically-based" transparency concepts.

There's a fourth mode that should probably be added: CutoutBlend, or whatever you want to call it. Texels with alphas at or very close to 0.0 are discarded (and so save ROP costs), otherwise they're treated as semitransparent textures that use alpha blending. This is the fourth combination of alpha-test on/off, alpha-blend on/off, i.e., it's both options on. Useful? Well, it is for decals, but that's a separate topic. I can't swear it will be, but it seems pointless to leave out this fourth mode. In the shader the two uses of alpha are separate bits of code. Alpha blend entirely controls the ROP. I would put this fourth mode in for completeness, it costs nothing.

Can you clarify this fourth mode or point to some information on it? It sounds very much like an implementation optimization. Can this be applied to the standard coverage blend mode all the time and thus removing the need for an additional mode?

As far as "or if the alpha channel is for something other than transparency" goes, be careful. If we want to ignore such data, great, we're done, and they're doing something else with their data that has nothing to do with PBR. People can stuff whatever they want in any channel they like. However, if they put roughness in the alpha channel, we can't access it currently. If we want to support this, OK, but then you'll need to add something that says "roughness is in this texture, in this channel", which I don't believe the specification has (maybe it does? I've lost track). It's assumed right now, AFAIK, that each element that could be textured uses its own separate texture. It's fine to go to a more flexible approach, mapping channels as you wish, but I don't think this is currently something a user can set.

The PBR materials extension specifically calls out what each channel does and where it goes. The diffuse/baseColor texture is a color texture with an alpha channel for opacity if present. Metallic and roughness texture is a separate texture on the red and green channel respectively. All I'm saying with this is that if there is an alpha channel and the alphaRenderingMode is Opaque, then the alpha channel is ignored and can be used by an extension for something else.

• alpha value for blending (really, coverage, like a gauzy fabric) - blending implies sorting and implies (to my mind) that z-write is off
• alpha test, for cutouts (blending off)
• alpha mode for choosing whether to blend and whether to test, independently, four states.

I'm having a hard time understanding what you mean. How do the first two values correlate with the alpha channel of the texture?

[erich666]

Can you clarify this fourth mode or point to some information on it?

My question here is, "what does the transparency setting actually do?" Does it also perform cutouts, or not? According to the proposal up top, it does not. Does it write the result to the z-buffer? It's not specified. And, "Use this mode for materials such as clear plastic or glass" needs to be removed if we treat alpha purely as coverage. Basically, you've said this proposal has changed, that you're now convinced alpha as coverage is the way to go. Maybe the best way forward is for you to make a new proposal. First, the new proposal should then not have the word "transparency" in it at all, only coverage. What follows is food for thought, and hopefully points out my concerns.

Right now the original proposal at the top of this thread has three modes: opaque, cutout, transparent. Cutout has to do with alpha test, transparent has to do with returning an alpha value and doing an over raster operation. They're separate shader/merge operations. I'm saying that having three modes like this and not including the fourth is making the proposal incomplete. You can't specify anything that has both a cutout and an alpha value for the remaining surface. Really, even a fourth mode is incomplete, in that whether z-buffer writing is done for partially-covered pixels is not specified. Blending (via over) is done; is the z-buffer value updated or not? It should be stated, as otherwise two renderers implementing the spec may get two different renderings.

Say you do use alpha as coverage in your new proposal. You have cutout grass, but really do want to have nice antialiased edges, so you then want both: alpha test for whether the fragment should be discarded (and so not affect the z-buffer - otherwise such invisible fragments will affect the z-buffer, which can lead to rendering errors) and whether the fragments partially covering a pixel should be blended in (these could give out of order rendering errors, but you may know that for your content it'll look acceptable, or at least better than aliasing).

Basically, I'm not sure of any advantage of leaving transparency + cutout out of the scheme proposed. Perhaps you meant for transparency to include cutout? But the proposal says, "alphaCutoff - This value specifies the cutoff threshold when in Cutout mode." Cutout affects only whether the fragment is killed or not. Transparency affects whether the alpha is output and whether blend mode is on. Whether something gets drawn to the z-buffer is unclear (and unspecified) currently, which I think is another potential problem. Having transparency always write to the z-buffer (which I guess you have to do, if it's truly just coverage), is one solution. This will make alpha much less usable as representing transparency. Which shouldn't matter to us (it's just coverage, right?), but...

The reality is that, despite whatever we specify, alpha is going to get treated as transparency by some no-doubt evil people. An example is a dragonfly's wing made by a craftsperson, e.g. https://s-media-cache-ak0.pinimg.com/originals/ce/f3/2d/cef32d7cf91b093fdc602e02953c2966.jpg . It is cutout, and the remaining surface is semi-transparent. It'll be a small break in PBR for glTf, "PBR is consistent, except for alpha, where people sometimes abuse it and treat it as transparency and not coverage, leading to inconsistent rendering results for some models."

Personally, I'd probably say "no alpha output from PBR, period" if I really truly wanted to enforce alpha not being treated as cheap transparency for glass, etc. You can compute an alpha from a texture and use it for cutout alpha testing, and that's it. Beyond that, use MSAA for coverage, tough luck. But that seems draconian. Still, it might be the way to go if you want to keep people from treating alpha as transparency.

Summary: please write a new proposal, if you wish. I have a number of problems with the original one, as you can tell. I can write more about the dangers of sorted transparency, but you sound like you're already convinced it should be interpreted only as coverage.

I'm having a hard time understanding what you mean. How do the first two values correlate with the alpha channel of the texture?

I honestly don't fully understand your question. If the alpha value is meant as coverage, then the first two examples show how alpha represents coverage. See McGuire's talk for an intro to coverage vs. transmission, if that's the confusion. Slide 6 on is good, showing how a gauzy object (truly coverage alpha, not transmissive) can have interior pixels have an alpha value that's not 1.0 or 0.0.

BTW, I'll probably use this for order-independent transparency for CAD work, if I ever feel the need. It's "plausible", not physically correct, but is good when there are a lot of almost-transparent surfaces visible (it falls apart when any of the surfaces is nearly opaque).

[bghgary]

@erich666 Thanks for your comments! Yes, I am going to work on a new proposal that just deals with coverage. At this point, I'm just making sure I understand your concerns and the overall picture before doing it.

[bghgary]

Here is the updated proposal (in a standalone comment for easy reference):

	Type	Description	Required
alphaMode	string	The material's alpha rendering mode.	No
alphaCutoff	number	The material's alpha cutoff value.	No, default:0.5

alphaMode

The material's alpha rendering mode enumeration specifying the interpretation of the alpha value of the main factor and texture. When alpha mode is not specified, the alpha value is ignored and the rendered output is fully opaque.

"MASK" - The rendered output is either fully opaque or fully transparent depending on the alpha value and the specified alpha cutoff value. Use this mode for materials that simulate geometry such as tree leaves or wire fences.

"BLEND" - The alpha value is used to composite the source and destination areas. The rendered output is combined with the background using the normal painting operation (i.e. the Porter and Duff over operator). Use this mode for materials that simulate geometry such as gauze cloth or animal fur.

• Type: string
• Required: No
• Allowed values: "MASK", "BLEND"

alphaCutoff

This value specifies the cutoff threshold when in "MASK" mode. If the alpha value is greater than or equal to this value then it is rendered as fully opaque, otherwise, it is rendered as fully transparent. This value is ignored for other modes.

• Type: number
• Required: No
• Minimum: >= 0

Implementation Notes for Real-Time Rasterizers

Real-time rasterizers typically use depth buffers and mesh sorting to support alpha modes. The following describe the expected behavior for these types of renderers.

When alpha mode is not specified, a depth value is written for every pixel and mesh sorting is not required for correct output.

"MASK" - A depth value is not written for a pixel that is discarded after the alpha test. A depth value is written for all other pixels. Mesh sorting is not required for correct output.

"BLEND" - Support for this mode varies. There is no perfect and fast solution that works for all cases. Implementations should try to achieve the correct blending output for as many situations as possible. Whether depth value is written or whether to sort is up to the implementation. For example, implementations can discard pixels which have zero or close to zero alpha value to avoid sorting issues.

Edit: Update wording based on feedback.

[bghgary]

@erich666

Blending (via over) is done; is the z-buffer value updated or not? It should be stated, as otherwise two renderers implementing the spec may get two different renderings.

This new proposal intentionally does not specify the whether depth is written for blending. In discussions with my colleagues, properly specifying the expected behavior such that all renderers do the same thing does not seem very practical since there are so many different ways to implement blending with different performance characteristics. Depending on the hardware or platform restrictions, different implementations must make different trade-offs.

I also don't think this spec should target a specific kind of renderer, which is why I put the depth/sorting information in the implementation notes for real-time rasterizers specifically. Ray-tracers or rasterizers that use A-buffers will work properly without additional specification.

Thoughts?

[lexaknyazev]

properly specifying the expected behavior such that all renderers do the same thing does not seem very practical

Can we say that alpha blending should be supported but it isn't included in glTF 2.0 conformance, since we can't demand any specific rendering result?

---

Mask - In this mode, the alpha channel represents binary coverage

Does it imply that for anti-aliased edges on cutouts, one need to use Blend mode? Why not implement this approach from the comment above:

You have cutout grass, but really do want to have nice antialiased edges, so you then want both: alpha test for whether the fragment should be discarded (and so not affect the z-buffer - otherwise such invisible fragments will affect the z-buffer, which can lead to rendering errors) and whether the fragments partially covering a pixel should be blended in (these could give out of order rendering errors, but you may know that for your content it'll look acceptable, or at least better than aliasing).

Is there any particular reason to define such cutoff value per material? Could we just say something like

Use alpha value 0.0 for fully transparent texels and 1.0 for fully opaque. Intermediate values could be used for anti-aliasing in Mask mode and for blending in Blend mode.

---

In other parts of glTF, only GL-based enums use ints (e.g., accessor.componentType: 5126 //GL_FLOAT), while others use strings (e.g., accessor.type: "VEC3", interpolation: "LINEAR"). We should be consistent and either use strings for alphaMode or switch all other core spec strings to ints.

[bghgary]

Can we say that alpha blending should be supported but it isn't included in glTF 2.0 conformance, since we can't demand any specific rendering result?

I would be okay with that.

Is there any particular reason to define such cutoff value per material? Could we just say something like

Use alpha value 0.0 for fully transparent texels and 1.0 for fully opaque. Intermediate values could be used for anti-aliasing in Mask mode and for blending in Blend mode.

The mask values are not binary (0 or 1) in the texture. It is a grayscale. This is so that subpixel interpolation (i.e. texture filtering/sampling) will work properly. You can think of these values as defining a surface curve for the alpha values.

As for specifying the cutoff value, it's not strictly required. We could say the cutoff value is always 0.5, but it's less flexible. The reasoning is along the lines of why we have scale for the normal map and strength for ambient occlusion map. Asset authors can use a different cutoff value for different mesh sizes (LOD) to improve the quality or for revealing more or less of the texture for actual variations of different meshes.

We should be consistent and either use strings for alphaMode or switch all other core spec strings to ints.

I didn't realize this was the convention. What do you recommend? I would be okay changing to strings so that we don't have to fix everything else.

[lexaknyazev]

This is so that subpixel interpolation (i.e. texture filtering/sampling) will work properly.

Is premultiplication implied here? If so, this must be stated in spec.

With current definition of Mask mode, we'll still get aliased edges, right? Could we allow engines to blend semi-opaque texels without sorting to reduce aliasing?

[bghgary]

Is premultiplication implied here? If so, this must be stated in spec.

My understanding is yes as it should be done for proper math, but it is an implementation detail to me. How about we add an implementation note?

With current definition of Mask mode, we'll still get aliased edges, right? Could we allow engines to blend semi-opaque texels without sorting to reduce aliasing?

Whether the edges are aliased or not is an implementation detail. For example, implementations can use alpha to coverage. I don't think the spec as it is currently stated precludes implementations from applying anti-aliasing.

[lexaknyazev]

Is premultiplication implied here? If so, this must be stated in spec.

My understanding is yes as it should be done for proper math, but it is an implementation detail to me.

It's data format issue. Either texture contains already premultiplied values, or they must be premultiplied on-load (e.g. via UNPACK_PREMULTIPLY_ALPHA_WEBGL param). We need to define exact image reqs or provide a configuration flag in image object.

Whether the edges are aliased or not is an implementation detail. For example, implementations can use alpha to coverage. I don't think the spec as it is currently stated precludes implementations from applying anti-aliasing.

Sorry, I'm not following. Fragment shader should write fragments alpha values to output for Alpha-to-Coverage to work. So, "the alpha channel represents binary coverage" sounds a bit misleading.

[bghgary]

It's data format issue.

Are you asking if we need to specify whether the image is premultiplied or not? As the spec currently stands, only BMP/PNG supports an alpha channel. PNGs are not premultiplied per http://www.w3.org/TR/PNG (Section 6.2). I believe BMP do not specify whether it is premultiplied or not, so I supposed we should specify this (i.e. BMPs are non-premultiplied)? Or remove BMP from glTF 2.0 since it has limited use? If we had additional texture formats #835, then we should specify if the image container/specification does not already specify.

"the alpha channel represents binary coverage" sounds a bit misleading

That's fair. I took out the first sentence. Sorry for the confusion. Is this better?

Mask - In this mode, the rendered output is either fully opaque or fully transparent depending on the alpha channel of the texture and the specified cutoff value. Use this mode for materials that simulate geometry such as tree leaves or wire fences.

[lexaknyazev]

remove BMP from glTF 2.0 since it has limited use

Yes, BMP and GIF are going to be removed from glTF 2.0.

PNGs are not premultiplied

One could premultiply pixels at asset's export time and get incorrect results. Let's state in baseColorTexture description that stored texels mustn't be premultiplied.

I took out the first sentence. Sorry for the confusion. Is this better?

Yes. We could add some implementation notes about alpha-to-coverage usage later depending on community feedback.

---

Will this issue be applied to diffuse component of Specular-Glossiness model as well?

[bghgary]

Will this issue be applied to diffuse component of Specular-Glossiness model as well?

Yes. It will be added to the root of the material which can be picked up by any material extensions. Will also need to add a note in spec-gloss extension.

[bghgary]

Here is an implementation of this:
https://sbtron.github.io/BabylonJS-glTFLoader/?model=FarmLandDiorama (Mask)

https://sbtron.github.io/BabylonJS-glTFLoader/?model=Hourglass (Blend)

@erich666
The reality is that, despite whatever we specify, alpha is going to get treated as transparency by some no-doubt evil people.

The hourglass model is doing the "evil people" thing that Eric warned about (i.e. the glass of the model is not physically accurate: no refraction, no opacity Fresnel, etc.), but there are lots of models out there that do this. If we expect these models to convert to glTF, we need to do something or they will all look opaque. This seems like a good compromise?

[lexaknyazev]

Should tree models in FarmLandDiorama and AppleTree enable double-sided rendering?
Currently, some branches are visible only from one side. Maybe additional boolean flag doubleSided is needed in the core material object.

If we expect these models to convert to glTF, we need to do something or they will all look opaque.

We should clearly state that alpha channel in MR model is intended only for coverage-based "transparency". Materials like glass or water will need new material model.

[bghgary]

enable double-sided rendering?

Yes, good eye :) We will add it to the PR.

We should clearly state that alpha channel in MR model is intended only for coverage-based "transparency".

Okay, I will add something to that effect. Though this will not stop these "approximations of transmission" models from being created or converted from other formats, even if the intended usage is for coverage.

[emackey]

I think this issue can be closed.