PPSSPP: Preliminary report on games with Multithreaded enabled. (a.k.a the return of GE Timing issues with MT on)

[solarmystic]

I haven't seen a catch-all issue report opened yet to discuss the effects of enabling the brand new and experimental Multithreaded (MT) function implemented by @unknownbrackets, so I've decided to take the initiative and report my findings here.

This issue report can be used by others to share their own reports about their own games with MT enabled too.

All settings used are default, with the exception of enabling the Multithreading option, 3x Rendering Resolution and the Speed/VPS display (modified to display actual VPS numbers)

UPDATE:-

In case scrolling through raw data and screenshots are a hassle, here're the results tabulated without the fluff. (updated to reflect the latest [Unknown] fix #3127

Raw data used to make table:-

Monhun Freedom Unite:-

Internal FPS during FMVs drops ridiculously.(to as low as 9-10FPS)

Internal FPS during gameplay drops from 30 to 20

Monhun 3rd Portable HD:- just hangs after the CAPCOM title screen. Unplayable with MT on.

Tekken 5 DR:- Internal FPS drops from 60 to 20 FPS during gameplay.

Tekken 6 :- Internal FPS drops from 60 to 20 during FMVs and gameplay.

FF VII Crisis Core:-

Playable, and FMVs function at the right speed.. Seems to be one of the few without any issues with MT enabled. Turbo can be enabled without any issues.

Gundam v Gundam Next Plus:-

Internal FPS slowdown during FMVs and title screens from 60 to a ridiculous number (similar symptoms to MHFU)

Internal FPS drops from 60 to 20 FPS.

DO NOT turbo this game in MT mode, it will hang.

The 3rd Birthday:-

Playable as it was before. No regressions, FMVs and gameplay internal FPS are sustained at 30.

As an added bonus, the flickering elements of the HUD and dialog are now fixed(?) when MT is turned on, although there seems to be an added graphical filter applied to the image with MT on.

MT OFF:-

MT ON:-

Final Fantasy Type 0:- Playable just like before, but with a strange white filter at the extremities of the screen.

MT OFF:-

MT ON:-

(I seem to be noticing a trend here)

Danganronpa:-

Internal FPS at the main menu drops below 60 to a variable number (50-60)

Actual gameplay is unaffected, and using Read Framebuffers to Memory to enable object detection for Investigation mode works as expected. Internal FPS during gameplay remains steady at 30 FPS.

Gods Eater Burst:-

Internal FPS during title screens plummets to below 30. Intro FMV Playback is also slowed down to 20 FPS at times.

Internal FPS during gameplay drops from 30 to 20 FPS.

Legends of Heroes Trails in the Sky:-

Title Screen FPS drops from 30 FPS to 0 FPS (!)

Game effectively hangs after, since you can't select any of the options.

If you're fast enough, you can skip the title screens and load your savegame, but then realise that the ingame internal FPS has dropped from 30 FPS to 17-20 FPS.

Weirdest thing is that the music seems to be playing smoothly as usual with the lower internal FPS.

Ys Seven:- Title screen slowdown from 30 to 20 FPS. Internal FPS during Gameplay drops from 30 to 17-20 FPS.

Black Rock Shooter:- Title screen and gameplay FPS drops from 30 to 20 FPS.

Hatsune Miku Project Diva 2nd:-

Severe internal FPS slowdown during the main menu featuring your character from 30 FPS to 13 FPS. Eventually drops to 0 VPS if left without input (or turboing), so the game effectively hangs after a short while.

If you somehow manage to get through and play a song, notice that the internal FPS during gameplay has also dropped from 30 to sub 20 FPS, rendering this highly timing sensitive game unplayable.

Tactics Ogre:- Fluctuating internal FPS in the title screens from 60 FPS to the 50s. Internal FPS at the world map drops from 60 FPS to 30 FPS. Internal FPS during gameplay drops from 60 FPS to 30 FPS.

Soul Calibur:- Probably one of the few native 60 FPS game I tested which managed to retain almost 60 FPS in all aspects with MT ON; FMVs, title screens and gameplay. No regressions here.

Persona 3 Portable:- Internal FPS slowdown during gameplay from 30 FPS to 20 FPS outside and inside Tartarus.

Untold Legends Brotherhood of the Blade:- Internal FPS during gameplay reduced from 60 FPS to 30 FPS.

MT ON:-

MT OFF:-

Kingdom Hearts BBS:- Another rarity. The internal FPS holds steady at 30 FPS during gameplay, title screens and the FMV cutscenes. Using the cheatcode to increase the internal FPS to 60 does not work anymore during gameplay however, and the FPS just holds steady at 30 FPS.

FF 1:- The Opening FMV does not play back smoothly, and looks like its running in slow motion. Internal FPS has lowered from 60 to the 30s.

Internal FPS during gameplay has also dropped from 60 to the upper 40s/lower 50s. One thing to note is that BGM playback is unaffected by the internal FPS drops.

FF 2:- FMV playback is unaffected. Internal FPS during gameplay drops from 60 to 30 FPS however.

FF 3:- FMV playback is unaffected and holds steady at 30 internal FPS. Internal FPS during gameplay holds steady at 30 FPS, as it was before.

Conclusion:-

It seems to me that a vast majority of the 60 FPS games, and a big number the 30 FPS games I tested have GE Timing issues again once Mutlthreaded is enabled. A minority of them do not however, but those are the exceptions.

The only 60 FPS game that does not have GE Timing issues again from my list is Soul Calibur.

Feel free to add any other games which haven't been covered by this list.

[unknownbrackets]

The screenshots really make this issue hard to read and not so useful. If you want to test so many games a chart/table would be way more helpful.

-[Unknown]

[solarmystic]

With #3119, a few games are now improved when MT is enabled.

Danganronpa is now perfectly normal as it was before, in both the cutscenes and gameplay

Monster Hunter Portable 3rd HD no longer hangs at the title screens and you can now head ingame. But the ingame FPS remains abysmal at sub 20 FPS.

Ditto for Monster Hunter Freedom Unite which now has improved FMV playback that is running smoothly, but horrendous internal FPS.

Legend of Heroes:- Trails in The Sky has also been fixed

Ys Seven has also been fixed

Black Rock Shooter has also been fixed

Tekken 5 DR and Tekken 6 also runs at 60 FPS internally once again with MT on.

Gundam Vs Gundam Next Plus is also back to 60 FPS internally.

@unknownbrackets

Fine then, I'll rework this into a chart or table. Just thought screenshots would be necessary for proof, lest anybody asks for it.

[unknownbrackets]

I don't think you're lying though, so it just makes for a lot of scrolling. I appreciate that you're trying to provide a ton of information, and I know all to well that it takes a lot of time to test so many games.

-[Unknown]

[solarmystic]

@unknownbrackets

Here are the results, tabulated in an easier to understand table without the screenshots getting in the way, updated with new ones obtained via further testing after your fix #3119 had been merged to master in 0.8.1-1222

[unknownbrackets]

Well, of those that still have problems I only have God Eater Burst... but I can reproduce it.

I've also noticed a random hang in Tales of Phantasia X (it seems like a process queue operation is being dropped, but I can't figure out why... the list is pending and a thread is waiting on it, but there's no process event so it hangs forever.)

Both of these are "fixed" (this is not a fix) by changing Core/ThreadEventQueue.h as so:

    void ScheduleEvent(Event ev) {
        {
            lock_guard guard(eventsLock_);
            events_.push_back(ev);
            eventsWait_.notify_one();
        }

        if (!threadEnabled_) {
            RunEventsUntil(0);
        } else if (EventType(ev) != EVENT_SYNC) {
            SyncThread();
        }
    }

This basically makes it process display lists synchronously, as before (but still on a separate thread.) That at least tells me the mechanics/scheduling is right, and either there's likely some threading interaction wrong or bugs when displaylists are executed asynchronously.

So far I also know Virtua Tennis has a similar issue, but with a diferent symptom: it's totally black afaiu.

Does the above not solve any of the issues you've experienced? If it doesn't, it means a totally different class of issue.

-[Unknown]

[solarmystic]

Actually, from the table that I made, you can clearly see that there are only a few games left that are unfixed by #3119 :-

Monster Hunter Freedom Unite
Monster Hunter Portable 3rd HD
Gods Eater Burst
Project Diva 2nd

Anyway, I'll try out that alteration you suggested to Core/ThreadEventQueue.h and report back.

[unknownbrackets]

Right, I don't have any of the others on that list. There's definitely other games with issues still, though.

-[Unknown]

[unknownbrackets]

So, GEB is having horrible interrupt scheduling issues:

*** Trigger interrupt for 6786326356 ticks (at 6847936821 ticks, so in -61610465 ticks)
*** Advanced interrupt at 6848367604 (62040886 late)
*** Running interrupt type 0 at 6848367604

That's at least 10 frames (!) late, I'm not sure why, but I have an idea.

-[Unknown]

[solarmystic]

Heh, that seemed to have done the trick with the remaining games listed in my previous comment.

All of the games below are now rectified somewhat with that Core/ThreadEventQueue.h alteration you suggested:-

Monster Hunter Freedom Unite (back to 30 FPS everywhere)
Monster Hunter Portable 3rd HD (ditto)
Gods Eater Burst (ditto)
Project Diva 2nd (ditto)

As an added bonus, it's safer to turbo through a majority of the games again with MT on, previously there was a very high probability that the game would crash if you turboed with MT on.

There's still a minor visual sideeffect (?) when MT is on in a number of games, like a very occasional black screen that flickers in certain games like the Monhun games and Gran Turismo, for instance.

When and if you decide to formally open a pull request to add those changes in, every game on that table would be fixed, at least in terms of the internal FPS, with MT on.

[unknownbrackets]

Hmm, don't have those games with the flicker. Anyway, the above isn't a fix, just a diagnosis. If we use that code, which was just for diagnosis, the GPU does not run at the same time as the CPU ever. In other words, it's exactly the same (and as slow) as single threaded, just more complicated.

The reason I asked was just to make sure they didn't have a totally different class of issue (that more complicated bit being broken.)

This fixes God Eater Burst, and it makes sense:

unknownbrackets@639ab83

Some interrupts are still running late, but not by such amazing degrees anymore.

-[Unknown]

[solarmystic]

There could definitely be other games that could be problematic and untested when it comes to MT in various other ways.

I'd still like to leave this issue report open (even after #3120 gets merged) , at least until Orphis buildbot starts working again, and we can get an idea of the far reaching effects of enabling Multithreaded in a bigger variety of games from other users who don't self compile PPSSPP and rely on said buildbot for builds.

[unknownbrackets]

Yeah, it's a shame it's still broken.

I just reproduced the random hang in God Eater Burst again, so it's still there. Have not found it yet, but it must be due to some hole in the listLock locking, I guess. I wish I had a consistent way to reproduce.

Standing right in front of Hibari I get ~1880% (except when it hangs), and ~1100% without. That's a ~63% improvement. With your weaker CPU and ATI card, what are you getting? I was hoping this would benefit you (and others with similar hardware.)

-[Unknown]

[solarmystic]

Not as much as you obviously.

With my Core 2 Duo T9550 and ATI Mobility 4670 card I'm getting:-

575% (345 VPS) with it on, and... the same with it off?

Yikes, let me check that again.

EDIT:- Yeah, I seem to be getting roughly the same Speed with or without MT in that game. Odd.

EDIT2:- The funny thing is that when a scenario like that happens in a game, enabling MT still caps CPU usage at 50% which means that the extra thread(?) is not being spawned, and the other core is not being used. Which is strange.

EDIT3:- Rofl, I know what happened, I used the build which had the "not-fix" implemented that you suggested for testing. It seems to give no benefits to MT for me.

[unknownbrackets]

Right, that's what I said - that diagnosis kills all the benefits. How is it without that?

-[Unknown]

[solarmystic]

Let me get back to you on that Unknown, I can't really tell until I manually merge your proposed "real fix" in the pull request you opened, since the game has a propensity to behave erratically with MT on in every single build before that fix. (as you already know)

[dbz400]

Regarding the overbright issue in FF Type-0 , it happned before and it is related to this

#1740

[solarmystic]

@unknownbrackets

I merged your proposed proper fix #3120 manually and tested Gods Eater Burst again, with and without MT, and now I could see the difference.

575% (345VPS) without MT
792% (472VPS) with MT

Not bad at all.

On the other hand, turboing in the Monhun games is now hazardous to health. Both games tested will just hang after a bit if you hold down the Unthrottle button.

[daniel229]

God of War Ghost of Sparta hang after this screen with multithreaded when Force 60 FPS or less disable since this commit abc396c . merged this #3120 still not help.

[unknownbrackets]

@daniel229 I added a couple more commits to #3120. The demo seems okay there, so maybe you hit the lock up that I'm having trouble reproducing.

-[Unknown]

[unknownbrackets]

@raven02 I wonder if the white in Type-0 is not just the wrong color (like the shadow)?

Unfortunately, the same hack as used in that pull will not work here. It will (as @solarmystic showed) be the same as turning the feature off, and it's already optional.

-[Unknown]

[daniel229]

@unknownbrackets merged the updated commits in #3120 fixed that issue.

[solarmystic]

@unknownbrackets

Yep, those two extra commits you added in #3120 also solved the crashes in both the Monhun games as well when unthrottling the framerate, nice work. That allowed me to see the gains when swicthing on MT, which are definitely present.

Seems like both cores are not being fully used though. Only getting 70-71% utlization on both, and I'm not GPU limited either in those situations. (running with GPUz open in the background for monitoring GPU Usage)

[unknownbrackets]

I'm still not sure why that is, although it has improved vastly for me (usually I have a pegged thread now.) There are some realities that could be causing it (synchronization/locks are not free, a lock-free model would be better but there are complexities...), but 70% is awful low...

-[Unknown]

[solarmystic]

Another interesting sideeffect of MT on a puny (relatively speaking) Core 2 CPU is that the emulator becomes much more "sensitive" to external events occuring in background applications.

e.g. opening up an instance of GPUz while turboing causes the screen to flicker black and performance to dip for a bit, before settling down, and heading back to the usual numbers.

Note to self:- Encoding a Video while enabling MT and running Tekken 6 on PPSSPP at the same time is a recipe for disaster on a Core 2.

ST (Single Threaded) is much more "resilient" somehow to these background changes, at least on my Core 2.

[unknownbrackets]

That's not really unexpected. The 2 threads running on separate cores depend on each other heavily. If you open another app, and it slows down only one of the 2 threads, it'll take the other down with it.

Single threaded doesn't have this problem.

-[Unknown]

[daniel229]

nVidia threaded optimization seems much effective in cpu usage.

ST about 25%~27% usage of 4cores

ST+nVidia threaded optimization about 39%~44% usage of 4cores

MT about 26%~30% usage of 4cores

MT+nVidia threaded optimization about 45%~52% usage of 4cores

[solarmystic]

Updated table with the latest results after fixes from #3120 are merged manually(will be known as 1230 when Henrik merges into master and if [Unknown] doesn't add anymore commits):-

Only 1 game left with Internal FPS trouble when MT is on; Project Diva 2nd.

[DonelBueno]

@solarmystic, I would like to test with this build too (1229), can you provide it to me?

I don't know how to compile the builds on my own. I installed Visual C++ 2010 Express, downloaded the ppsspp repository as a .zip file (I don't even know if this is the right way of getting the code for manual compilation) and tried to compile using the visual studio solution file inside the Windows folder, but the compilation failed due to a lot of errors Can you make a noob guide for that? Do I need the GitHub app installed?

[hrydgard]

@i30817 , I know it's too late but the fix for the fragmentgenerator issue is to #include Core/System.h somewhere early. On that note I think we have some include dependencies to clean up...

[solarmystic]

I'd also like to share a table of some rather rough VPS measurements I've taken in a rather limited (5) number of games to highlight performance changes across a wide range of revisions.

The games are Danganronpa, Crisis Core, Monster Hunter Freedom Unite, Monster Hunter Portable 3rd and Gran Turismo.

Note in particular the revisions after 1207, when I include MT ON numbers.

Testing locations

Danganronpa:- Loaded up a savefile in front of a doorway.

Crisis Core:- Loaded up at a save point at the Ground Floor of the Shinra Building

MHFU:- Right at the entrance to the village

MH3rdPHD:- Same as MHFU

GT:- At the starting line of the first quick race you have access to.

The point I wanted to highlight was that it seems like with every GPU/CPU sync fix, the Performance of every game in the list when MT is ON is demonstrably dropping with each subsequent revision, coming closer to MT OFF numbers. It seems like in the process of attaining perfect sync between the two threads, performance has been sacrificed for MT.

[hrydgard]

More sync = more overhead. However I think there will be plenty of room for tuning.

At least MT seems to provide a speedup almost all of the time.

[i30817]

@unknownbrackets
seems to work to boot up LH:TitS even with the SeparateIOThread to 'True'

Jean of arc (eventually hangs) after this:
33:34:750 idle0 W[HLE]: ppsspp/Core/HLE/sceKernelThread.cpp:2615 sceKernelReleaseWaitThread(): Refusing to wake HLE-delayed thread, right thing to do?

But that probably has more to do with the numerous font errors in that game.

[unknownbrackets]

Yeah, @solarmystic basically that's why I'm not so worried about only one game at near-acceptable levels of FPS. I can probably fix that, but it will mean losing e.g. 5-10% performance across the board. There's already the option to turn it off...

Probably there's a more perfect way to solve it, though. Also, for me the performance drop of the recent change was actually pretty minor, it's unfortunate to see that it affects weaker CPUs more.

-[Unknown]

[solarmystic]

On a rather happier note, it seems like the buildbot is alive again.

I hope to see more reports coming in, especially with more users enabling the MT function on a lot more different games.

[VIRGINKLM]

I'm gonna do reports of all of my games, do you need Android reports too?
The tests are with both [SeparateCPUThread = True] and [SeparateIOThread = True] at the same time right?
64bit or 32bit?(I guess it could differ?)

[solarmystic]

@VIRGINKLM

I'd advise you to keep [SeperateIOThread = False] for now, since a number of users seem to have issues with it. But otherwise, you're good to go. Android test results would be much appreciated.

[VIRGINKLM]

OK I'll start immediately.

[VIRGINKLM]

Android : HTC EVO 3D MSM8260 1.5GHz Adreno 220 320MHz - ICS 4.0.3

PPSSPP v0.8.1 r1283
Tekken 6 - MT ON: 12% - 60FPS MT OFF: 13% - 60FPS
Tekken Dark Resurrection - MT ON: 28% - 60FPS MT OFF: 30% - 60FPS
Where Is My Heart? - MT ON: 57% - 53FPS MT OFF: 37% - 53FPS
Ultimate Ghosts n Goblins - MT ON: 102% - 52FPS MT OFF: 98% - 52FPS
Patapon - MT ON: 189% - 52FPS MT OFF: 225% - 52FPS

All experienced some screen flashing and all seemed to not refresh the screen while you are on Turbo Mode (the last frame would remain onscreen). Also Ultimate Ghosts n Goblins has some graphical garbage here and there with MT ON.

[vsub]

Single code P4 2.8GHz with Hyper Treading:

0.8.1-1372 x86
Gods Eater Burst - 30fps everywhere with and without MT.
Legends of Heroes Trails in the Sky - 30fps everywhere with and without MT and the game work fine
YS 7 - Same as above
Project Diva Extended - flickers on a lot more places with MT enabled in non buffered mode and the fps is between 20 - 25 and sometimes freeze...work fine with MT off
Persona 3 - Work fine...30fps everywhere

I think all games are getting +30% to the Speed: percentage counter.

[hrydgard]

On a hyperthreaded P4, a 30% speed boost is pretty damn good actually.

I'm seeing bigger speed boosts in some games but I have an Ivy Bridge so it's a little different :)

[solarmystic]

The distinction has to be made between % Speed boosts from the Speed % indicator, or the actual % increase in the VPS numbers.

I'll give an example.

For Danganronpa, in build 0.8.1-1395 x64

MT OFF :- 390 VPS (650% Speed)
MT ON :- 478 VPS (~797% Speed)

If you consider the VPS % increase from 390 to 478, that's only a 22.5% increase in actual terms.

However, if you consider the +% to the speed counter itself from 650% to 797% Speed, that's an increase of 147% Speed.

Thus, it is vital when reporting increase percentages to indicate whether you're considering actual VPS numbers or the Speed percentage increases.

[vsub]

What I mean was for example if I get 250% and I enable MT,I'll get 280%

...it's really better to have vps numbers than percentage,it's much more informative
Currently(0.8.1-1397)to set a correct speed limit(for example +50% the normal speed),you have to divide the game internal speed by 2 so if the internal is 20,you must set the limiter to 30 to make it run at 150%

[solarmystic]

@vsub

Ah okay then.

About the the speed limiter issue, an issue report has already been opened on the github by @ShadowLady #3145

[unknownbrackets]

Ah, so that's a 12% boost. I also thought 30% was too good to be true on a hyperthreaded P4.

-[Unknown]

[vsub]

Can we consider this a bug?
You can enable\disable MT while ingame from the screen menus which lead to weird FPS rates

Start a game with MT enabled and then disable it from the screen menus.
On Fate Unlimited Codes,when I disable it like that,the FPS goes below 20 and is constantly changing to different number

[hrydgard]

Yeah I need to fix that. It shouldn't be possible.

[thedax]

Kingdom Hearts BBS: Final Mix seems to hang during one of the beginning in-game sequences where you wake up after having that dream when MT is on. While it's off, it has no issues, in -1686-g9e53f2a. I'm unsure at this very moment if it has worked thus far with MT. I'll check shortly.

Edit: I went back to -1269-g7ba54b5 and it doesn't work there, either. It's probably safe to say it's never worked correctly with MT on.

[prateem]

Can confirm that BBS: Final Mix shows hangs with MT on, and works perfectly fine with MT off. Several cutscenes in the game actually trigger the MT failing (the cutscene of the Mark of Mastery trials, for example, and also when Terra opens the glowing purple door in Enchanted Dominion, as another one).

[unknownbrackets]

Hmm, this happens right after a pause signal. So pause signals probably don't work right, and that's probably why this happens.

It looks very likely to have something to do with that.

-[Unknown]

[unknownbrackets]

So, if I understand correctly, this issue remains only in Outrun 2006 and Project Diva, right? Any other games with issues?

If no, I think we should close this and open an issue for that more narrow thing, since this is long and covers many other problems. I think Outrun already has an issue but it seems to not be very specific either.

-[Unknown]

[unknownbrackets]

Anyway, #5987 makes one of the Diva demos run at ~27 fps on my computer. At the expense of performance, it could be improved to 28-29. Not sure what it actually runs at on a PSP, maybe not exactly 30?

Anyway, the problem is that he cpu is getting too far ahead of the gpu, because the gpu is doing slow things. The problem is, this is the entire goal of multithreaded - to allow it to go ahead.

These lines will mostly "fix" it (~29):

const int geIntervalUs = 1000000 / (60 * 10);
const int geBehindThresholdUs = 1000000 / (60 * 10);

However, they will also reduce e.g. Gods Eater Burst's speed by ~30% (in fact it will barely if at all be faster than multithreaded off.)

Personally I think 27 is good enough.

-[Unknown]

[hrydgard]

There can visually be a big difference between 30fps and 27 fps - the former will have uniform motion while the latter may stutter slightly 3 times per second. But maybe that's happening on the real PSP.

Either way there's indeed no reason to keep this mega-issue open any more, we should open new issues as necessary.

[dbz400]

@unknownbrackets , Yes , outrun 2006 still running abnormal FPS with multithread on .