MToonMaterial does not seem to support tiling (offsets and scale)

[alankent]

Tiling of textures is a useful way to swap expressions on the face of a character, or swap the texture of eyes (e.g. replace the iris with glowing red eyes, or heart emoji). This works in Unity (with UniVRM etc). But as far as I can tell, the Three.js version of the MToonShader does not honor the repeat and offset properties of the texture. (Flipping to MeshStandardMaterial shows offset etc seem to be working, hence my believe it may be a missing feature of MToonMaterial.) This is a request to add support (or a plea to provide an example to help a poor ignorant soul!).

I started with animating VRMExpressionTextureTransformBind weight properties, but the face never changed. So I tried hard coding values as soon as a model is loaded. E.g.

material.map.repeat.set(2, 2);
material.map.offset.set(0.1, 0.1);
material.map.wrapS = RepeatWrapping;
material.map.wrapT = RepeatWrapping;
material.map.needsUpdate = true;

But it seems to have no effect. I then replace the MToonMaterial with a crude MeshStandardMaterial and things started happening.

Note: this is independent to texture uv animation. I am trying to use expressions to change the blend shapes and texture of the face together. Normally I do this with tiling of the textures. Example blogs and videos for Unity (I wrote these back in 2021):

• https://extra-ordinary.tv/2021/01/19/tiling-for-expressions-with-vroid-characters/
• https://youtu.be/i3kvjpRr3lc?si=sAlfBIVFnTwUTM4P&t=544

I am trying to bring this across into a JavaScript implementation, but I am getting stuck on the texture tiling. I am trying to build a website for VRM character animation. I have got character movements and blend shapes all working, tiling was next on my list for blushes etc in the cheeks, and I was trying to stick to standard VRM if possible.

[alankent]

Okay, I made a bit of progress. My understanding now is textures have "repeat and offset", but MToonMaterial ignores that. Instead it has its own material.uniforms.mapUvTransform.value which is a matrix3. So you can do things like setUvTransform(tx, ty, sx, sy, rot, rx, ry). That does work if set directly. So I can setUvTransform(0,0, 0.5,0.5, 0,0,0) for a 2x2 tilted texture. Then update the tx,ty values to pick one of the 4 tiles (0,0 / 0,0.5 / 0.5,0 / 0.5,0.5). (There are also normals and emissions matrices too, with independent normalUvTransform, emissionUvTransform etc.)

But my reading of VRMExpressionTextureTransformBind it seems to update texture.repeat/offset and not the mapUvTransform matrix (or normalUvTransform etc matricies). E.g. https://github.com/pixiv/three-vrm/blob/dev/packages/three-vrm-core/src/expressions/VRMExpressionTextureTransformBind.ts#L131-L132 . I have not seen any code to copy the map texture offset/repeat values over to mapUvTransform etc.

So perhaps my request is that for VRMExpressionTextureTransformBind with MToonMaterial materials, replace updating the texture offset/repeat with updating the mapUvTransform/emissionUvTransform/normalUvTransform/etc. matrix3 to capture the offset and scale using something like:

material.uniforms.mapUvTransform.value.setUvTransform(offset.x, offset.y, 1/scale.x, 1/scale.y, 0, 0, 0)

The workaround I am going to try next is every frame to copy the texture offset/scale for map/normal/etc over to the mapUvTransform matrix. That way I can use VRMExpressionTextureTransformBind as is.

Update: It seems to be working. Not pretty, but working. Only trick was 'map' is material.map (a property), whereas 'mapUvTransform' is material.uniforms.mapUvTransform. It would still be nice if VRMExpressionTextureTransformBind worked with MToonMaterials without this hackery. My test code in case of interest:

  updateTiling() {
    // I save away all the MToonMaterials in an array for faster access here.
    for (const material of this.mtoonMaterials) {
      const updateTilingFromTexture = (uvTransform, texture) => {
        if (texture && uvTransform) {
          uvTransform.value.setUvTransform(
            texture.offset?.x ?? 0, texture.offset?.y ?? 0,
            1.0 / (texture.repeat?.x ?? 1), 1.0 / (texture.repeat?.y ?? 1),
            0, 0, 0
          );
        }
      };
      updateTilingFromTexture(material.uniforms.mapUvTransform, material.map);
      updateTilingFromTexture(material.uniforms.normalMapUvTransform, material.normalMap);
      updateTilingFromTexture(material.uniforms.emissiveMapUvTransform, material.emissiveMap);
    }
  }

(Side comment: Yikes! I came across some threads (including from 2015!) talking about tiling data for repeat and offset such as mrdoob/three.js#5876. Lots of discussion about whether it should be in Material instead of Texture. It appears more recently the "node" system is the planned replacement allowing repeat/offset in the material instead of the texture. So no idea how to make VRMExpressionTextureTransformBind "generic" across different materials. In my case, I mainly care about MToonMaterial since it's the default for VRM models when loaded.)

[0b5vr]

mapUvTransform should be updated via MToonMaterial.update -> MToonMaterial._uploadUniformsWorkaround.

three-vrm/packages/three-vrm-materials-mtoon/src/MToonMaterial.ts

Line 558 in b35bc21

this._updateTextureMatrix(this.uniforms.map, this.uniforms.mapUvTransform);

Do you call vrm.update() before each render call?

three-vrm/packages/three-vrm/examples/dnd.html

Line 139 in 0970bda

currentVrm.update( clock.getDelta() );

[alankent]

Thank you for the quick response.

Ah, I had forgotten about that. I was having problems with vrm.update() as it was updating the raw bones from the normalized bones (overwriting my animation clips being played back on the raw bones). So I was experimenting (from a year ago?) calling selected methods from vrm.update() directly (e.g. springBoneManager). So I need to go back to vrm.update() and work out how to animate bones correctly from animation clips. I flipped back to vrm.update() the textures do now animate correctly (but bone animations are lost). I retarget the clips from mixamo.com, but I assume I have to do it on "normalized bones" instead of the raw bones - hopefully I just need to get the names right in the Animation track instances.

[0b5vr]

If you have a question about bone animations, please feel free to use discussions.