[Feature request] May need new Blendshape mode for AAO Merge Skinned Mesh in some usage

[ColorlessColor]

Since the avatar I am using https://avatarchan.booth.pm/items/5250102 does not support MMD blendshapes, I use this https://wontonton.booth.pm/items/5665288 , which will Face , Eyes, and FaceMMD meshes are merged into the Body object, which meets the assumptions of the MMD world. There are blendshapes with the same name on these three meshes, and they must be animated simultaneously (with the same values).

The main point is that not all animations target Face, Eyes, and FaceMMD, and many products that rely on MMD blendshape target Body; another part of my animation targets Face, Eyes, and FaceMMD. In order to easy to record animations, I also used MA Blendshape Sync to synchronize all blendshapes to Face.

Therefore, my avatar relies on the assumption that the blendshapes of Face, Eyes, and FaceMMD will be merged into the Body, and all animations involving these three meshes will eventually be redirected to the Body, the animation pointing to the Body (actually there was no Body mesh before) will be retained.

---

As of AAO 1.7, the AAO Merge Skinned Renderer silently merges blendshapes to the same name, and everything works fine. But there will has a problem with AAO Trace and Optimize: not all animations are pointed to both Face, Eyes, and FaceMMD so some blendshapes will be frozen.

Especially those Face Tracking animations that rely on OSC, they all point to the Body, which makes it more difficult for me to debug, because these animations may be deleted after being deemed invalid because the parameters can only be underdriven by OSC, or they may be deleted because the target(Body) does not exist.

For the above reasons, I ended up having to use MA Blendshape Sync to synchronize the Eyes and FaceMMD blendshapes to the Face.

But there is still a issue with this solution. Blendshapes pointing to the Body will be considered invalid by AAO and deleted, although they will eventually appear on the Body after AAO Merge Skinned Rendere. So my final solution was to add Face, Eyes, Body and FaceMMD to the Exclusions in the Debug menu of AAO Trace and Optimize.

---

So after explaining my avatar situation, there are two issues with AAO 1.8.0 that cause problems in my project:

• AAO Merge Skinned Renderer now either renames the blendshapes (which I don't want to do) or throws an exception warning me that Differencetly animated Blendshape is detected (I know, and I want it to stay as is). 1.7.X compatibility mode works, but it can no longer be selected whenever another mode is selected.
• Using AAO Trace and Optimize will cause the Blendshape mode of AAO Merge Skinned Renderer to be invalid. No matter what mode I choose, it will rename blendshapes which have same name, unless I add Face, Eyes, Body and FaceMMD to the Exclusions list.
• If I do not add Face, Eyes, Body and FaceMMD to the Exclusions list, all my animations pointing to the Body will be cleared and the related blend shapes will be frozen.

My current temporary solution is:

• AAO Merge Skinned Renderer remains in v1.7.x or older compability mode
• Add Face, Eyes, Body and FaceMMD to the AAO Trace and Optimize Exclusions list.

Looks like it should work, but v1.7.x or older compability mode is not always available, and I don't really want to keep using Exclusions in Debug menu.

---

So I would like to have an option to force merge the meshes I need, just like v1.7.x or older compability mode which ignore Differencetly animated Blendshape is detected. And AAO Trace and Optimize should be run after AAO Merge Skinned Mesh to ensure that the animation of the blendshape pointing to the new mesh is now valid. This way I no longer have to use the debug menu to ignore all objects or doing some other hacking.

[anatawa12]

Did you tried Merge Same Name mode of Merge Skinned Mesh blendshape mode?

[ColorlessColor]

Did you tried Merge Same Name mode of Merge Skinned Mesh blendshape mode?

I got a error window said Differencetly animated Blendshape is detected; And using AAO Trace and Optimize will ignore Blendshape mode of AAO Merge Skinned Renderer, it always rename the blendshape.

Another issue is that all the animation curves I target for the Body will be deleted by AAO Trace and Optimize, because AAO seems to think that there have no blendshapes exist on the Body, although the blendshapes does appear on the Body after AAO Merge Skinned Renderer, because all meshes (Face, Eyes, FaceMMD) are Merged into Body

[anatawa12]

I got a error window said Differencetly animated Blendshape is detected;

It's not a error, just a warning. you can ignore if you wish.
I think it's better to add way to suppress warning but it's not implemented yet.

using AAO Trace and Optimize will ignore Blendshape mode of AAO Merge Skinned Renderer and always rename the blendshape.

Is the name of the Skinned Mesh Renderer Body will be changed?
(I mean is the Body mesh will be merged with other meshes automatically by Trace and Optimize?)

If MMD World Compatibility mode is enabled (enabled by default), BlendShapes for MMD World would be kept as is by Trace and Optimize, and (first) Body will not be merged to another renderer.

Another issue is that all the animation curves I target for the Body will be deleted by AAO Trace and Optimize, because AAO seems to think that np blendshapes exist on the Body, although the blendshapes does appear on the Body after AAO Merge Skinned Renderer, because all meshes (Face, Eyes, FaceMMD) are Merged into Body

I could not reproduce this issue.
I tried renaming existing Body mesh to Face and create MergeSkinnedMesh named Body with "Face" as a source.
This works well so animations targeting after MergeSkinnedMesh seems works well.

---

After writing the message above, I come up with one situation fits your report.
Do you have multiple "Body" mesh on hierarchy?
If so and the second "Body" is Merge Skinned Mesh, your animation targets "Body" will be assumed to be targeting first "Body" mesh, and will not redirected to second merge "Body" if animating blendShape not exists on the first body.

[ColorlessColor]

There is my setup (AAO 1.8.1):

Animation clip sample:

And play mode I got:

[ColorlessColor]

I'm trying to binary search the cause and currently I suspect that the animation generated by MA Blendshape Sync cannot be treated as the same animation curve by AAO. you are right this just a warning.

Update:

There also seems to be an issue, MMD blendshapes on FaceMMD don't seem to be considered part of MMD compatibility if I use Merge BlendShapes with same name... and I don't know how to tell AAO that my FT/v2/* parameters are read and written by VRCFT.

I've been completely confused.

[ColorlessColor]

Before NDMF built my avatar, Body is an empty object, and there is no Skinned Mesh Renderer yet. Some of my animations point to Face, Eyes, FaceMMD, and some animations point to Body( At this point it is still an empty object).

My expectation is that AAO Merge Skinned Renderer will merge Face, Eyes, FaceMMD into Body, and fix all animations' target to Body; and those animations pointing to Body will be valid now, because there really is a Body now. Finally, AAO Trace and Optimize should be run on this basis, because at this time the animation has been corrected and the MMD blendshapes are in place.

[anatawa12]

My expectation is that AAO Merge Skinned Renderer will merge Face, Eyes, FaceMMD into Body, and fix all animations' target to Body; and those animations pointing to Body will be valid now, because there really is a Body now.

It is the expected behavior of Trace and Optimize that Animations pointing to Body blendShapes that will be copied from Face / MMDFace will not be removed by Trace and Optimize.
So not working well with your avatar is a bug. (if your configuration is correct)

Umm... Could you show me the Trace and Optimize configuration? and Could you confirm you don't have whitespace in your Body mesh name?
(We cannot see whitespace on the Unity Editor so it's popular to mistakenly insert WhiteSpace at the end of object name).

[ColorlessColor]

Could you confirm you don't have whitespace in your Body mesh name?

Yes I can comfirm this:

Could you show me the Trace and Optimize configuration

I remember I uploaded it, but just realized I might have accidentally deleted it while editing, anyway this is my configuration:

[ColorlessColor]

It is the expected behavior of Trace and Optimize that Animations pointing to Body blendShapes that will be copied from Face / MMDFace will not be removed by Trace and Optimize.

Actually I have some animations (dozens to be exact) pointing to the Body.

I have explained the reason for use AAO Merge Skinned Mesh above. Many of my animations assume that there are MMD blendshapes on the Body and Face Tracking animation also assumes that the face tracking blendshapes is on the Body, but But 9b3d doesn't have these. So I use AAO Merge Skinned Mesh to generate a GameObject Body with needed blendshapes.

If AAO tries to track animation curve before merging mesh...

If this use case is not supported, I will continue to use Exclusions in the Debug settings to exclude Face, Eyes, Body, and FaceMMD.

But anyway, I do need an option to suppress the warning on a specific AAO Merge Skinned Mesh when using Merge BlendShapes with same name.

[ColorlessColor]

Actually I have some animations (dozens to be exact) pointing to the Body.

I tried to change all animation paths from Body to Face. Even if I selected Merge BlendShapes with same name, the blendshapes(not MMD blendshapes) were still renamed.

Update:

I guess this is expected behavior of AAO T&O?

Update2:

It seems that T&O thinks that the MMD blendshapes on my Face, Eyes, and FaceMMD are not part of MMD compatibility, so they are all treated as regular blendshapes for optimization and deleted. Because these blendshapes are not on the Body.

[anatawa12]

Sorry for late response.

It is the expected behavior of Trace and Optimize that Animations pointing to Body blendShapes that will be copied from Face / MMDFace will not be removed by Trace and Optimize.

Actually I have some animations (dozens to be exact) pointing to the Body.

I mean your usage is expected so it should work but I'm struggling reproducing your problem.

But anyway, I do need an option to suppress the warning on a specific AAO Merge Skinned Mesh when using Merge BlendShapes with same name.

Suppressing warning is needed for almost all warning in AAO so please wait for future improvement.

For MMD BlendShapes, I found that the BlendShape names are not familiar to me.
Does your MMD BLendShape name is listed on this list?

AvatarOptimizer/Editor/AnimatorParserV2/AnimatorParser.cs

Lines 581 to 676 in 0c42ece

	private static readonly string[] MmdBlendShapeNames = new[]
	{
	// New EN by Yi MMD World
	// https://docs.google.com/spreadsheets/d/1mfE8s48pUfjP_rBIPN90_nNkAIBUNcqwIxAdVzPBJ-Q/edit?usp=sharing
	// Old EN by Xoriu
	// https://booth.pm/ja/items/3341221
	// https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/i/0b7b5e4b-c62e-41f7-8ced-1f3e58c4f5bf/d5nbmvp-5779f5ac-d476-426c-8ee6-2111eff8e76c.png
	// Old EN, New EN, JA,
	// ===== Mouth =====
	"a", "Ah", "あ",
	"i", "Ch", "い",
	"u", "U", "う",
	"e", "E", "え",
	"o", "Oh", "お",
	"Niyari", "Grin", "にやり",
	"Mouse_2", "∧", "∧",
	"Wa", "Wa", "ワ",
	"Omega", "ω", "ω",
	"Mouse_1", "▲", "▲",
	"MouseUP", "Mouth Horn Raise", "口角上げ",
	"MouseDW", "Mouth Horn Lower", "口角下げ",
	"MouseWD", "Mouth Side Widen", "口横広げ",
	"n", null, "ん",
	"Niyari2", null, "にやり２",
	// by Xoriu only
	"a 2", null, "あ２",
	"□", null, "□",
	"ω□", null, "ω□",
	"Smile", null, "にっこり",
	"Pero", null, "ぺろっ",
	"Bero-tehe", null, "てへぺろ",
	"Bero-tehe2", null, "てへぺろ２",
	// ===== Eyes =====
	"Blink", "Blink", "まばたき",
	"Smile", "Blink Happy", "笑い",
	"> <", "Close><", "はぅ",
	"EyeSmall", "Pupil", "瞳小",
	"Wink-c", "Wink 2 Right", "ｳｨﾝｸ２右",
	"Wink-b", "Wink 2", "ウィンク２",
	"Wink", "Wink", "ウィンク",
	"Wink-a", "Wink Right", "ウィンク右",
	"Howawa", "Calm", "なごみ",
	"Jito-eye", "Stare", "じと目",
	"Ha!!!", "Surprised", "びっくり",
	"Kiri-eye", "Slant", "ｷﾘｯ",
	"EyeHeart", "Heart", "はぁと",
	"EyeStar", "Star Eye", "星目",
	"EyeFunky", null, "恐ろしい子！",
	// by Xoriu only
	"O O", null, "はちゅ目",
	"EyeSmall-v", null, "瞳縦潰れ",
	"EyeUnderli", null, "光下",
	"EyHi-Off", null, "ハイライト消",
	"EyeRef-off", null, "映り込み消",
	// ===== Eyebrow =====
	"Smily", "Cheerful", "にこり",
	"Up", "Upper", "上",
	"Down", "Lower", "下",
	"Serious", "Serious", "真面目",
	"Trouble", "Sadness", "困る",
	"Get angry", "Anger", "怒り",
	null, "Front", "前",
	// ===== Eyes + Eyebrow Feeling =====
	// by Xoriu only
	"Joy", null, "喜び",
	"Wao!?", null, "わぉ!?",
	"Howawa ω", null, "なごみω",
	"Wail", null, "悲しむ",
	"Hostility", null, "敵意",
	// ===== Other ======
	null, "Blush", "照れ",
	"ToothAnon", null, "歯無し下",
	"ToothBnon", null, "歯無し上",
	null, null, "涙",
	// others
	// https://gist.github.com/lilxyzw/80608d9b16bf3458c61dec6b090805c5
	"しいたけ",
	// https://site.nicovideo.jp/ch/userblomaga_thanks/archive/ar1471249
	"なぬ！",
	"はんっ！",
	"えー",
	"睨み",
	"睨む",
	"白目",
	"瞳大",
	"頬染め",
	"青ざめ",
	}.Where(x => x != null).Distinct().ToArray()!; // removed null with Where

If not, I'll add MMD BlendShapes. could you let me know the source (website / world or others) that shows your BlendShapes are used in MMD Worlds if you have.

[ColorlessColor]

If not, I'll add MMD BlendShapes. could you let me know the source (website / world or others) that shows your BlendShapes are used in MMD Worlds if you have.

The names of these BlendShapes partly come from 【オリジナル3Dモデル】サフィー-Sapphy- V2 :

I'm not sure if these BlendShapes are actually used in the MMD world...

I mean your usage is expected so it should work but I'm struggling reproducing your problem.

I can provide a private repository for testing. In fact, I am trying to make a minimal reproduction project these days.

[anatawa12]

Sorry I thought blendshapes in your screenshots in #1358 (comment) are for mmd blendshapes but you're using well known blendshapes so it should not be problem.

I can provide a private repository for testing. In fact, I am trying to make a minimal reproduction project these days.

Thank you.