[Intermittent] crash on [State -2] when mashing "recovery" command (PP) while in mid air, in chars that use unpatched CvS/POTS system

[AxeLanderMoreira]

Steps to reproduce:
While playing single arcade mode (character doesn't seem to matter), if I get hit and try mid-air recovery (that is, press M+H or L+M+H punch buttons simultaneously and repeatedly), the engine will sometimes crash before the recovery is event attempted. Crash invariably occurs due to "panic: runtime error: index out of range" at bytecode.go.

Expected behavior:
It should perform the air recovery move (or none, if not applicable to the selected character), instead of crash.

Platform:

• Ikemen_GO_Linux compiled from source
  I'm specifically using Linux port, but I'm not sure if the issue happens in other platforms. Ikemen-GO Linux compiled from source. I just made sure that the issue still occurs in current HEAD version:

commit 6d0b839 (HEAD -> master, origin/master, origin/HEAD)
Author: K4thos [email protected]
Date: Sun Aug 28 18:18:57 2022 +0200

Fix sound panning localcoord calc

But it's been happening for quite some time.

Additional context:
I will attach screenshots and stack trace logs.

[AxeLanderMoreira]

Hereis also a log of one occurrence:

2022/08/29 18:16:17 Using OpenGL 4.6 (Compatibility Profile) Mesa 21.2.6 (Mesa Intel(R) HD Graphics 620 (KBL GT2))
Loading module: external/mods/multipleface2.lua
Loading module: external/mods/stagetext.lua
panic: runtime error: index out of range [451] with length 152
goroutine 1 [running, locked to thread]:
github.com/yuin/gopher-lua.(*LState).PCall.func1()
github.com/yuin/[email protected]/state.go:1988 +0x21f
panic({0x8b8260, 0xc002e07ef0})
runtime/panic.go:884 +0x212
main.BytecodeExp.run_const({0xc0033146a0, 0x17, 0xc003a50a00?}, 0xc004f6b500?, 0xc002e6e2b8, 0xc003a50a00?)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:1698 +0x73b1
main.BytecodeExp.run({0xc0033146a0?, 0x17, 0x20}, 0xc003a50a00)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:1298 +0x70be
main.BytecodeExp.evalB({0xc0033146a0?, 0xffffffff?, 0x0?}, 0x9?)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:1999 +0x1e
main.StateBlock.Run({0x1, 0xffffffff, 0xffffffff, 0x0, {0xc0033146a0, 0x17, 0x20}, 0xc001e70960, {0x0, 0x0, ...}}, ...)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:2099 +0x16c
main.StateBlock.Run({0x1, 0xffffffff, 0xffffffff, 0x0, {0x0, 0x0, 0x0}, 0x0, {0xc0011e1c00, 0x34, ...}}, ...)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:2113 +0x317
main.(*StateBytecode).run(0xc004b35900, 0xc0036be120?)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:8018 +0x22a
main.(*Char).actionRunStates(0xc003a50a00)
github.com/ikemen-engine/Ikemen-GO/src/char.go:5391 +0x42a
main.(*CharList).action(0xbbfa38, 0x1?, 0xc002e782b4?, 0x0?, 0xc202ba93c202ba93?, 0x0?, 0x0?, 0x0?)
github.com/ikemen-engine/Ikemen-GO/src/char.go:6004 +0xd8
main.(*System).action(0xbbed40, 0xc002e782b8, 0xc002e782b4, 0x3f800000)
github.com/ikemen-engine/Ikemen-GO/src/system.go:1140 +0x34f
main.(*System).fight(0xbbed40)
github.com/ikemen-engine/Ikemen-GO/src/system.go:2076 +0x16c5
main.systemScriptInit.func57.2()
github.com/ikemen-engine/Ikemen-GO/src/script.go:934 +0x558
main.systemScriptInit.func57(0xc00012a370)
github.com/ikemen-engine/Ikemen-GO/src/script.go:977 +0x3bb
github.com/yuin/gopher-lua.callGFunction(0xc00012a370, 0x0)
github.com/yuin/[email protected]/vm.go:202 +0x37
github.com/yuin/gopher-lua.init.3.func26(0xc00012a370, 0xc0c0001?, 0xc00030a000?)
github.com/yuin/[email protected]/vm.go:821 +0x379
github.com/yuin/gopher-lua.mainLoop(0xc00012a370, 0xc0002be240?)
github.com/yuin/[email protected]/vm.go:31 +0xfa
github.com/yuin/gopher-lua.(*LState).callR(0xc00012a370, 0x0, 0xffffffffffffffff, 0xc000014018?)
github.com/yuin/[email protected]/state.go:1211 +0x1b4
github.com/yuin/gopher-lua.(*LState).Call(...)
github.com/yuin/[email protected]/state.go:1967
github.com/yuin/gopher-lua.(*LState).PCall(0xc00012a370, 0x0, 0xc0002be240?, 0x0)
github.com/yuin/[email protected]/state.go:2030 +0x145
github.com/yuin/gopher-lua.(*LState).DoFile(0xc00012a370, {0xc0000ac738?, 0x0?})
github.com/yuin/[email protected]/auxlib.go:396 +0x6b
main.main()
github.com/ikemen-engine/Ikemen-GO/src/main.go:92 +0x21a

stack traceback:
[G]: in function 'game'
./external/script/start.lua:1403: in function 'f_game'
./external/script/start.lua:1836: in function 'launchFight'
external/script/default.lua:20: in function <external/script/default.lua:0>
./external/script/start.lua:1457: in function 'f'
external/script/main.lua:3097: in function 'loop'
external/script/main.lua:3181: in function 'loop'
external/script/main.lua:4148: in main chunk
[G]: ?

goroutine 1 [running, locked to thread]:
main.main()
github.com/ikemen-engine/Ikemen-GO/src/main.go:102 +0x45e

[AxeLanderMoreira]

After looking at the stacktrace, repeating the bug a few times, and tweaking the codes of the chars involved, I have a few more details to add, that might be useful:

1. The bug definitely happens always while processing the char's states classified under Statedef -2;

This code chunk here in char.go (at func (c *Char) actionRun()) is common to all stacktraces I've got so far:

		c.minus = -2
		if c.player || c.keyctrl[1] {
			if sb, ok := c.gi().states[-2]; ok {
				sb.run(c) // <--- line 5391
			}
		}

2. It seems related to processing the command labeled "recovery", in triggers inside a [State -2] definition.

3. More specifically, it seems related to how MUGEN 1.x characters using the CvS/PoTS system originally handled the "Safe Fall" technique. Here is an example from a Balrog char:

[State -2, P2 Safe Fall]
type = targetstate
trigger1 = numtarget = 1
trigger1 = target, command = "recovery"
trigger1 = target, pos y >= -48 && target, vel y > 0
trigger1 = target, alive && target, hitfall && target, gethitvar(fall.recover)
trigger1 = target, stateno = 5030 || target, stateno = 5035 || target, stateno = 5050 || target, stateno = 5071
value = 5205
ignorehitpause = 1

4. Apparently, the author PoTS (original creator of the CvS/PoTS system) stumbled on this issue before, because in his latest updates, all of his characters contain a workaround for this issue, and mention Ikemen specifically. Here is an excerpt from his Ryu char:

[State -2, P2 Safe Fall]
redirectID = enemynear, ID
type = selfstate
trigger1 = numenemy
;trigger1 = enemynear, command = "recovery";currently broken in Ikemen
trigger1 = (enemynear, command = "x" + enemynear, command = "y" + enemynear, command = "z") >= 2 || (enemynear, command = "$D")
trigger1 = (enemynear, stateno = [5030, 5050]) || (enemynear, incustomstate)
trigger1 = enemynear, pos y >= enemynear, const(movement.air.gethit.groundrecover.ground.threshold)
trigger1 = enemynear, gethitvar(isbound) = 0
trigger1 = enemynear, vel y > 0
trigger1 = enemynear, hitfall
trigger1 = enemynear, movetype = H
trigger1 = enemynear, statetype = A
trigger1 = enemynear, gethitvar(fall.recover)
trigger1 = enemynear, dizzy = 0
trigger1 = enemynear, alive
value = 5200
ignorehitpause = 1

5. I cannot confirm that this is the sole root cause for the crash. If I remove the original 'Safe Fall' state, the stability seems to be improved, but crashes can still be reproduced ocasionally (and with the same stack trace). That being said, the characters originally created by PoTS seemingly work fine on Ikemen-GO (I can't get them to crash).

6. I don't have Windows or Mac systems available to use, but since this issue happens at the core state machine processing, and its occurrence is known to char creators (PoTS at least), I think it's fair to assume that it affects all platforms. Therefore, I'm removing the 'Linux' tag from this issue.

7. I can provide or attach examples (and counter-examples) of chars that reproduce the issue, if necessary (and allowed).

8. I'm also editing the issue title for more clarity.

If the issue that I describe here has been reported before, please let me know so I can link accordingly.

[AxeLanderMoreira]

P.S.: I'm not sure how to edit the labels on this issue (nor if I have the required permissions), but I suggest the 'Linux' label to be removed from it.

[potsmugen]

That comment in the code is related to an issue with the handling of the "recovery" command itself in Ikemen. I'll make a thread about it eventually unless it gets fixed in the meantime. For the purpose of this bug I think you can ignore it.

[Lazin3ss]

Well, it seems the crashes are related to character constants. When Ikemen attempts to get a character constant (probably via const() trigger), an out of range error happens when accessing a string pool. A test case would probably be needed since I wasn't able to recreate the bug myself.

[AxeLanderMoreira]

Some additional comments:

• In the version compiled from source code, I found a method to easily reproduce the issue without interaction. I go to Watch mode (CPU vs CPU) and let 2 tag teams of 4 random fighters battle each other. I know it's not deterministic and not a formal test case, but usually the crash happens in less than a minute.
• IMPORTANT: The issue definitely doesn't happen when using the prebuilt Linux executable and the 'data/*.zss' scripts from the latest official build 0.98.2. I can mash the buttons as much as I want, and matches in Watch mode complete successfully.
• Curiously enough, I am unable to build from the source, in the 0.98.2 tag. I get some errors related to the openal-dev package (apparently some mismatch between the openal-dev version installed in my Ubuntu machine, and the one expected by that version of Ikemen):
  ../../go/pkg/mod/github.com/ikemen-engine/[email protected]/openal/alcCore.go:88:100: cannot use _cgo0 (variable of type *_Ctype_struct_ALCdevice) as type *_Ctype_struct_ALCdevice_struct in argument to _Cfunc_alcGetError
• Since this does not happen in the official build, feel free to close this issue.
• Nevertheless, I will try to use 'git bisect' with intermediate builds between 0.98.2 and current HEAD, and try to locate which commit may have introduced this bug.

[Lazin3ss]

Hey, I found some crashes were happening because char execution expected already-executed chars to have their "minus" field to be 1. Try to build from this branch and let me know if this helps with your issue or if introduces other crashes.

https://github.com/Lazin3ss/Ikemen-GO/tree/experimental-fix

[AxeLanderMoreira]

Hi @Lazin3ss ,

I have successfully built from your branch, and I have still reproduced the crash under the conditions above (Watch mode between 2 tag teams of 4 players):

2022/09/26 18:49:26 Using OpenGL 4.6 (Compatibility Profile) Mesa 21.2.6 (Mesa Intel(R) HD Graphics 620 (KBL GT2))
2022/09/26 18:50:58 Failed to open bgm: open sound/DMsskirmish.ogg: no such file or directory
panic: runtime error: index out of range [4] with length 0
goroutine 1 [running, locked to thread]:
github.com/yuin/gopher-lua.(*LState).PCall.func1()
github.com/yuin/[email protected]/state.go:1988 +0x21f
panic({0x8b95c0, 0xc003597050})
runtime/panic.go:884 +0x212
main.varAssign.Run(...)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:2136
main.StateBlock.Run({0x1, 0xffffffff, 0xfffffffe, 0x0, {0xc0034366a8, 0x18, 0x18}, 0x0, {0xc001f0d940, 0x3, ...}}, ...)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:2113 +0x317
main.StateBlock.Run({0x1, 0xffffffff, 0xffffffff, 0x0, {0xc00334da60, 0x9, 0x10}, 0x0, {0xc000f86e00, 0x2, ...}}, ...)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:2113 +0x317
main.StateBlock.Run({0x1, 0xffffffff, 0xffffffff, 0x0, {0xc00334d1b0, 0x9, 0x10}, 0xc0013895e0, {0xc003912900, 0x5, ...}}, ...)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:2101 +0x24b
main.StateBlock.Run({0x1, 0xffffffff, 0xffffffff, 0x0, {0xc001329840, 0x17, 0x20}, 0xc002c7aa00, {0x0, 0x0, ...}}, ...)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:2101 +0x24b
main.StateBlock.Run({0x1, 0xffffffff, 0xffffffff, 0x0, {0x0, 0x0, 0x0}, 0x0, {0xc007e72400, 0x28, ...}}, ...)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:2113 +0x317
main.(*StateBytecode).run(0xc00514c8c0, 0xc0042d6180?)
github.com/ikemen-engine/Ikemen-GO/src/bytecode.go:8021 +0x22a
main.(*Char).actionRun(0xc004386000)
github.com/ikemen-engine/Ikemen-GO/src/char.go:5385 +0x2ea
main.(*CharList).action(0xbc1a38, 0x1?, 0xc003b99494?, 0xc2586cd3?, 0x0?, 0x0?, 0x0?, 0x0?)
github.com/ikemen-engine/Ikemen-GO/src/char.go:6002 +0xd8
main.(*System).action(0xbc0d40, 0xc003b9e308, 0xc003b9e304, 0xc003b9e310)
github.com/ikemen-engine/Ikemen-GO/src/system.go:1047 +0x315
main.(*System).fight(0xbc0d40)
github.com/ikemen-engine/Ikemen-GO/src/system.go:2009 +0x164d
main.systemScriptInit.func57.2()
github.com/ikemen-engine/Ikemen-GO/src/script.go:935 +0x558
main.systemScriptInit.func57(0xc0000b46e0)
github.com/ikemen-engine/Ikemen-GO/src/script.go:978 +0x3bb
github.com/yuin/gopher-lua.callGFunction(0xc0000b46e0, 0x0)
github.com/yuin/[email protected]/vm.go:202 +0x37
github.com/yuin/gopher-lua.init.3.func26(0xc0000b46e0, 0xc0c0001?, 0xc00013a000?)
github.com/yuin/[email protected]/vm.go:821 +0x379
github.com/yuin/gopher-lua.mainLoop(0xc0000b46e0, 0xc000543240?)
github.com/yuin/[email protected]/vm.go:31 +0xfa
github.com/yuin/gopher-lua.(*LState).callR(0xc0000b46e0, 0x0, 0xffffffffffffffff, 0xc000350000?)
github.com/yuin/[email protected]/state.go:1211 +0x1b4
github.com/yuin/gopher-lua.(*LState).Call(...)
github.com/yuin/[email protected]/state.go:1967
github.com/yuin/gopher-lua.(*LState).PCall(0xc0000b46e0, 0x0, 0xc000543240?, 0x0)
github.com/yuin/[email protected]/state.go:2030 +0x145
github.com/yuin/gopher-lua.(*LState).DoFile(0xc0000b46e0, {0xc000216738?, 0x0?})
github.com/yuin/[email protected]/auxlib.go:396 +0x6b
main.main()
github.com/ikemen-engine/Ikemen-GO/src/main.go:92 +0x21a

stack traceback:
[G]: in function 'game'
./external/script/start.lua:1401: in function 'f_game'
./external/script/start.lua:1834: in function 'launchFight'
external/script/default.lua:2: in function <external/script/default.lua:0>
./external/script/start.lua:1456: in function 'f'
external/script/main.lua:3037: in function 'loop'
external/script/main.lua:3121: in function 'loop'
external/script/main.lua:4093: in main chunk
[G]: ?

goroutine 1 [running, locked to thread]:
main.main()
github.com/ikemen-engine/Ikemen-GO/src/main.go:102 +0x45e

[AxeLanderMoreira]

Some status update: With the help of git bisect, apparently I've managed to trace the exact commit where the crash issue started to happen. Revisions including and after the commit below present the crash:

---

Author: unknown [email protected]
Date: Thu Aug 18 19:47:53 2022 -0300

Refine chars' action processing order

Character action code was divided in various passes, as to fix bugs like assert SCTRLs (AssertSpecial, Screenbound, AssertInput) not being able to be properly redirected. Now, action code is executed like this:
- actionPrepare(): Initial asserts are set/unset, characters perform default operations
- actionRunStates(): Chars' states are executed, with chars with MoveType = A having priority
- actionFinish(): Post-states execution code
- update(): Chars' internal values are updated according to action code

---

Revisions before this commit work fine.

If I manually revert the changes in this commit in src/char.go, on the current HEAD of master branch, it works fine as well. I'm attaching the patch here just for reference, but of course it's not the ideal solution:

ikemen_go_revert_refine_chars_action.txt

@Lazin3ss , since you are the author of the original commit, perhaps you may check what might be wrong.

[Lazin3ss]

Indeed, the branch I shared with you was an attempt to fix what was introduced in that commit. This "index out of range [4] with length 0" crash seems to be related to these lines in tag.zss:

Ikemen-GO/data/tag.zss

Lines 326 to 330 in 741670d

	if standby && redLife > 0 && (time % const(TagRedLifeRegenFrames)) = 0 { # every 30 frames (0.5s) by default
	let regenVal = min(redLife, ceil(lifeMax * const(TagRedLifeRegenPercent))); # 0.5% lifeMax by default
	lifeAdd{value: $regenVal; absolute: 1}
	redLifeAdd{value: -$regenVal; absolute: 1}
	}

If you comment them, the crash should no longer happen, in theory.

I've been trying very hard to crash the game with the code from the branch I shared, and I wasn't able. Do you have any particular character/commonStates combination that might make the error trigger? For now, I tried with kfm and ryu_pots. If you could upload your Ikemen GO folder, it would be great.

It is best not to revert that commit, but instead improve on it. Reverting it would mean losing assert redirection, among other things.

[AxeLanderMoreira]

This "index out of range [4] with length 0" crash seems to be related to these lines in tag.zss:

OK, I will try that later today, on a clean build from HEAD.

I've been trying very hard to crash the game with the code from the branch I shared, and I wasn't able. Do you have any particular character/commonStates combination that might make the error trigger? For now, I tried with kfm and ryu_pots. If you could upload your Ikemen GO folder, it would be great.

In my experience, kfm and the characters made originally by POTS and recently patched for Ikemen (including ryu_pots) apparently do not cause the crash. But other characters with CvS/POTS but not particularly adapted for Ikemen do. My library of chars is relatively large now (~70 chars) but I will try to narrow down to one single character. If I'm successful with that, I will upload it here.

It is best not to revert that commit, but instead improve on it.

I totally agree.

[AxeLanderMoreira]

@Lazin3ss , some status update: Regarding the removal of those lines in tag.zss, yes it seemingly fixes the crash in the Tag Team Watch Mode scenario (with the expected side effect of disabling red lifebar regeneration).

[AxeLanderMoreira]

I have also uploaded a minimal set of my Ikemen_GO folder.

It has two characters: Gambit and Kaede; and one stage only.

If I assemble a tag team of 4 Gambit's vs 4 Kaede's in Watch Mode, I can reproduce the crash 5 out of 5 times; all before the countdown reaches 80.

You may download from the following link:

https://drive.google.com/file/d/1McWYGzFQun-ydaKP7hs5Gz3gVhSddiTu/view?usp=sharing

[Lazin3ss]

We did various changes that are related to this issue. Hope you can tell us if the issue got fixed on your side or if it still persists.

[AxeLanderMoreira]

Hi @Lazin3ss . Sorry for the long delay on posting the feedback.

I've changed my PC since the original report, didn't reinstall Ikemen-GO, and completely forgot about this issue.

I'm on a different hardware now, and I don't have the previous build for comparison anymore, so it's not 100% accurate.

Anyway, I have cloned the HEAD of Ikemen-GO and built it on my new Linux PC, tested the above steps multiple times, and couldn't find one single crash.

So it definitely seems like the crash has been fixed, and I'm going to close the issue.

Side note: While trying the steps above (Tag Team 4v4 on Watch Mode), even though I didn't get the crash, rarely did I get the match to actually finish. Often what would happen is the exact same situation described in issue #1043.