Stuntman - FPU rounding errors for AI pathing

[Souzooka]

PCSX2 version:
PCSX2 1.5.0-dev3032

• Notes:
  In Stuntman, AI paths are slightly erroneous due to FPU rounding issues; the game is unable to be finished due rounding issues. This can be witnessed in any level with a car the player is made to follow. Specifically, in the fourth level of the game, the player follows a lead car using a police car. On the "Chop/Zero" FPU rounding mode, the lead car fails to make a corner in the level, softlocking the level. Changing the rounding mode to "Nearest" allows the car to make it past this corner, but it fails the very next turn.

How to reproduce the issue:
Start a new career in Stuntman, finish the first three levels, then attempt to finish the fourth level.

Last known version to work:
While the game is marked playable, there is no evidence that the game was ever checked that it was able to be completed. The game is pretty resource-intensive, and it is more likely that the first level was tested at most by most testers.

Attached is a picture of the lead car failing to make a turn on the "Chop/Zero" setting.

[Souzooka]

I attempted to try this PR and while the AI gets even further on interpreter, he still fails a turn later.

[ghost]

Try to set also VU clamping/rounding. This game use both FPU, and VU to calculate physics there.
Problematic calculations results are from in game functions dyForwardDynamics, and dyStepRunge1 to dyStepRunge5 (0x211E60 to 0x2127AC for US release).

[LoStraniero91]

This also happens in Driv3r (and probably Driver: Parallel Lines).
Could be the same issue since all of them were made by Reflections.

[refractionpcsx2]

I know it probably isn't fixed, but can this be tested on a recent master to see if it's any better?

[ghost]

Yes, issue is still present on latest master.

Edit:
I made few tests, and looks like game don't like hardcoded DIV/SQRT/RSQRT nearest rounding on FPU. Although removing it help only little bit, you still can't finish level that way. That's for recompiler, interpreter seems to be broken even more on that game.

Here are settings, and explanation where you can get with them.

FPU / VU (All test done with removed hardcoded FPU nearest hack)
Negative / Zero
Best result, car crash at the wall in the end.

Nearest / Zero
Result like on Souzooka picture.

Positive / Zero
Crash earlier, car goes too much on right side.

Negative / Positive or Nearest
Close to first picture, but crash to wall at left little bit earlier than on VU Zero. There is no way to test properly VU negative rounding because that mess VU1, and picture. But setting it for moment make car path worse when we set it back. Tried also negative div hack, seems to have no effect. So that suggest sqrt/rsqrt nearest rounding make it bad.

Edit2:
Looks like i missed that pcsx2 use different file for doubles, and didn't removed hack there previously.
So results stay as above, but now we can get to "best result" even on Zero/Zero rounding. Still no way to finish stage.

[arienap]

I did some more testing with the PAL version of this game and compared the behavior of the AI car to an actual PS2.
I used a later level (Woopin and a hollerin) and by setting the EE rounding to nearest I get the AI to behave closest to actual PS2 AI. EE rounding to zero, positive or negative make the AI drift further sooner from the game on PS2. For all other settings (VU rounding and EE/VU clipping) I could not see the effect on the AI car.
Tested with v1.7.0-dev-546-g64010cf79 (latest GIT build).
Game plays nicer on PCSX2 than on my PS2+TV, I think because of the better analog sticks of the Xbox360 pad combined with less input lag.

[ghost]

areinap, if you can send me a save of your game to the Woopin and a hollerin mission as well as a savestate before the car get stuck I would appreciate it, thanks.

[arienap]

Stuntman_debug.zip
I have uploaded a memory card with the 100% completed career from my PS2.
I have only completed the PAL version and the debugging was also done with the PAL version.
You can try the Woopin mission (nr.2 corkscrew) by loading the game "Arie" from the main menu and then go into the filmography in the career menu. For your convenience, I attached a save state of doing just exactly that. The save state is at the beginning of the mission, the cars path is affected right from the start. By changing the EE FPU settings you must restart to see the full effect (the path of the cars drifts earlier / later).

[ghost]

Thanks a lot for uploading booth the memorycard and save state, I will look into the game.

[ghost]

I can make the car almost right with 4 settings, EE Rounding to negative - EE Clamping to none - VU Rounding to nearest and VU clamping to none. It will however stops in the middle of a straight road.

[arienap]

No setting the EE rounding to negative makes the car go further in the level, but its behavior is much different compared to the path it takes on the actual PS2. In the level you are testing, the car on the PS2 goes between the van and the car after the right hand corner. With EE rounding to negative it makes the car go left of both.

I am pretty sure it is related to the FPU rounding of the recompiled EE code. The VU rounding might also contribute a bit, but from trying the settings I could not see the effect. Would it make sense to make some homebrew code to exercise the DIV/SQRT/RSQRT instructions and compare the outcome of the actual PS2 HW versus the recompiled code?

[refractionpcsx2]

@arienap Did you try my FPU rounding PR when I had it open? DIV/SQRT/RSQRT are forced to "nearest" rounding by default in PCSX2, that PR removed that so you could change the rounding for the EE and it would actually affect it.

[arienap]

You mean this PR: #3816 ?
I have not set up the toolchain to build PCSX2 myself, so I was using the available dev builds from master (PCSX2 v1.7.0-dev-546-g64010cf79 in this case).
I still see significantly different behavior when changing EE settings, so I guess there must be more to it than just the DIV/SQRT/RSQRT instructions. I am happy to do more testing, but I need some time, so I can figure out how to build PCSX2 myself.

[refractionpcsx2]

Each PR has its own builds at the bottom, however when it gets closed they get removed so of course they're gone now.

I guess, i can make you a build temporarily.

[refractionpcsx2]

pcsx2-noforceround.zip
Here, try this, you should be able to adjust the EE rounding modes with this and have more impact. Chop/Zero will actually be Chop/Zero for those commands now, so be sure to try that too.

Edit: Also be sure to try all rounding modes with Full clamping as well, as full clamping actually changes the FPU to use doubles which is completely separate code for handling FPU.

[arienap]

I tested the build, thanks for providing this.
I tested with multiple levels (stunt drivers, ice river drive in Switzerland).
The AI drivers are driven by the game in open loop, giving throttle and steering inputs.
The EE rounding mode messes with the physics which in turn mess up the AI drivers path.
It is a little bit misleading to judge the correctness of the emulation by the progress of the AI driver, as the AI driver has very little margin. In some cases the even more inaccurate emulation allows for the AI driver to progress further. Especially in the Switzerland level, the most accurate mode lets the AI get stuck the earliest.
I tried many different settings for the EE clamping / VU rounding / VU clamping modes, but I could only really detect changes with the EE rounding mode.

All four EE rounding modes behave differently, but by comparing them to the actual PS2 I would rate them the following (accurate to least accurate):

1. Nearest
2. Negative
3. Chop/Zero
4. Positive

The EE rounding positive gives more errors than just the AI driver, you feel more grip (I can achieve faster lap times in the speed tests with EE rounding to positive) and the sound gets messed up.

[refractionpcsx2]

Okay well the rounding that is forced is forced to nearest, so I guess we're already doing the best we can. Apparently the AI on the PS2 isn't spectacular anyway, but I assume it doesn't get stuck

[ghost]

Actually VU clamping to none improves the ia somewhat too, but it get stuck one turn on the another still.

[arienap]

OK I moved to the NTSC version for some more testing. The NTSC version seems affected more by the emulator compared to the PAL version. This may be due to the physics running at a higher frequency (60Hz vs 50Hz). The first chase in the "Live twice for tomorrow" seems perfect for testing as the AI does not slide around as much, so it can drive for much longer without crashing. I do see an improvement by using the EE clamping mode "Full" compared to normal, so this setting is affecting the physics code albeit much less compared to the round mode. I will do an extensive test with all combination of settings later and provide the data.

[ghost]

Hello, 3y old issue, but the wiki states that the pathing is "very close to a real PS2" which is wrong, as on the 2nd level of the 2nd movie, the ai breaks and stops you from completing the mission https://youtu.be/BGNTgZ9BrrI
Tested on PCSX2 1.7.2352

Round mode and clamping mode for the EE, FPU and VUs are all on default. settings

[SoapyMan]

Did tests on the 1.7.3069 AVX2 and when EE rounding mode set to Nearest on Whoopin and Hollerin chase - the getaway doesn't hit the pickup truck just like on hardware (reference: https://youtu.be/Goda4uRzrH8?t=94) but after that it hits the rear bumper at the right side when passing the van and hits the fence.

[ColeTrickIe]

Unfortuneately still there in 1.7.3327

[BenoitAdam94]

Guy here couldn't finish the game on PCSX2 : https://www.youtube.com/watch?v=5mbT7HRa7YQ

[Shoegzer]

Right, here is the spot where ColourShed encounters the issue with PCSX2. I haven't tested Driv3r lately but it's likely still the same given that it also abuses PS2 floating-point inaccuracy as reported above and here. The fact that it's also a Reflections game and probably using the same engine is a big hint too.

I'm guessing this is one of the hardest problems remaining to solve given that both titles (especially Driv3r) are arguably the highest profile games still not playable according to the compat list. It's not surprising given the undocumented complexities of the EE, though at least a happy accident that resolves this would be nice to see down the road.

[Shoegzer]

Can someone with the appropriate game saves please test Stuntman and Driv3r per #7707 (comment)? Would be so great if the new split-VU rounding/clamping feature allowed for a decent workaround to this issue.

[Sukotto-1999]

Regarding this issue, September 2016, a user named FlatOut on the PCSX2 fourms, Found a slightly-BIG issue that completely makes DRIV3R unplayable! THIS is the most affected game on PCSX2 developed by Ubisoft Reflections. The police car always crash into a different location. Issues may not be fixable at the moment.

[refractionpcsx2]

We're fully aware.

[Shoegzer]

Yeah, I guess I should have been more specific about what we'd be testing for here so it's good to have the clarification.

[iyudar]

For what it's worth, I remember using the director mode over anything else in DRIV3R over a decade ago, and I occasionally remember the recording of your pathing randomly messing up causing this identical bug that appears in Stuntman, on ORIGINAL PS2 hardware. I would do some cool cop chase, go to edit the scene and find out a lamp post i drove very close to in-game, ends up being in the way of my driving path in the recorded clip and the scripting illusion completely fails.

Whatever caused the error in the code on PCSX2 doesn't seem necessarily exclusive in an emulator, it's just more prominent or consistent on an emulator. It's still present on a regular PS2; what causes it there?

[sittingduck77]

I know nothing about programming. But could there be something to gain by looking at how Stuntman (and Driv3r) runs on the PS3?

[refractionpcsx2]

Probably not, the PS3 emulator has some measure of soft float emulation, which is the solution we've previously mentioned.

[mirh]

Both run like crap (even on ps4).
To be super fair, the other two built-in software emulators might have different results but I wouldn't really try to grasp at such straws (even because, even in the most rosy situation, you'd still eventually have to figure out what's the magic).

[sittingduck77]

My thinking was, that the PS3 was pretty much completely hacked with CFW and so that it would be easier to figure out the magic. But maybe it's not THAT open.

[stenzek]

Legally, we can't use anything from the PS3/PS4 emus. Had this discussion several times, but even looking at the emu itself is questionable, because it's presumably protected from reverse engineering by access control.

[JulianWgs]

I've stumbled across this article today (good and short read by the way). @F0bes already is a contributor to PCSX2, but I dont know whether they is aware of this issue. May be they can offer some advice on how to tackle this.

PS: Please don't pressure any PCSX2 contributor into doing something on this issue. They are doing this in their free time and the software is provided "as is".

[refractionpcsx2]

Fobes works very closely with us, helps out quite often, but I'm pretty sure this is a little out of his reach, as it is the rest of us.

we know what we need, we need a soft float implementation, fixing 1*N isn't going to make Stuntman magically work, unfortunately.

[terremoth]

This also happens in Driv3r (and probably Driver: Parallel Lines). Could be the same issue since all of them were made by Reflections.

Yes, it HAPPENS in Driv3r indeed!

[terremoth]

We would also have to know how the EE rounds for several instructions, even for softfloat, which is currently unknown. AFAIK add/sub is fairly accurate, but mul/div/etc are still unknown.

Is this information in the EE Official Manual? If not, is there a way to "discover" with a game trying to simulate some way to compare with the IEEE 754? If not, is there a way to create a math function O(1) to do exactly (or almost the same) rounded numbers accuracy?

[F0bes]

• The exact information is not present in the manuals.
• There are of course ways to determine the behaviour. Write some homebrew that crunches up the numbers and compare. It's a laborious task and no one has done it.
• I can't imagine a solution not being O(1) unless you have a ridiculously sized LUT. I can assure you that wouldn't get merged here ;)

[terremoth]

Take it easy, that sounded like a passive-aggressive, I am just trying to help to think in a solution

[F0bes]

My apologies, was just trying to be a little lighthearted on the last point.

[iyudar]

Is there any reason why the PAL version works a little better? Still not perfect or 100% playable. Is some internal framerate messing up the timing with game physics? Like the game compensates for some PS2 constant that the emulator is smoothing out or not emulating precisely. I assume PAL runs at a different framerate than PS2

[Souzooka]

This seems to also affect in-engine demonstration replays in Sky Odyssey. The plane tends to crash fairly early with default EE FPU rounding, though it performs better with alternate rounding and clamping EE settings.

[SecularSteve]

That replay thing reminds me of a very particular issue I ditto get in "Need for Speed: Hot Pursuit 2", which is currently mentioned in and as #7877

Likewise with "Tokyo Xtreme Racer Zero" as #5597

Probably isn't even completely related, but it seems useful to mention if it brings this long-lingering issue any closer to a solution.

[terremoth]

I think this issue, when solved, will fix many many ps2 games problems. Seems like there are more games having problems with this than we think.

Anyone let me know if I can help with anything. I really wanna help solving this.

[sittingduck77]

I would love for Stuntman to run on pcsx2, and having read this thread I reached out to Robin Wardle a while ago - he was the programming project manager on Stuntman. He doesn't have input that could help get the game to run, but below is what he remembers about Stuntman. It's a good read. Enjoy.

---

I can give you some background information that might be useful. I suspect those developing the emulation code know far more about IEEE numerical standards than I do.

I mainly developed the Stuntman physical terrain model, ground interaction and, with three very capable colleagues, the physics engine, before becoming a producer. I didn't do any programming at all on Driv3r and Parallel Lines, although with my background I was very involved in overall technical design.

One of the posters in the thread you linked identifies correctly that the chases in Stuntman and Driv3r aren't AI, they are input recordings. One of the design team drove the chase car and the pad inputs were recorded as a data stream. The pad inputs are replayed in the actual mission just as if it were a person controlling the car. This gave more realistic-looking chases; actual AI wasn't good enough at the time. It was also something of a design style preference left over from Driver and Driver 2 where memory was at a premium and pad recordings offered a very low storage option for chase vehicle control. The chases were recorded by members of the design team who were really good at the games, which is partly why they have a high difficulty level.

Unfortunately the pad recording approach means that the chases are quite fragile. Unless the recording simulation and the playback simulation are exactly the same in all respects, then at some point the chase will fail. Even very slight numerical differences will build up until it all goes wrong; the vehicle will continue receiving pad inputs even though its actual location and orientation differ from the location and orientation it was in when the pad input was recorded, until the errors become too great and the chase vehicle deviates completely from its path.

As you might imagine, this was a problem in development and especially testing, as the chases needed to be continually re-recorded as elements of the game changed. I'm not sure now but I think the chases might have had to have been recorded separately on Xbox and PS2 for Driv3r. By the time Parallel Lines was in development we were able to move onto an actual AI system, which had its own problems but which was more robust.

I can imagine it will be very difficult to get the games to work properly on an emulator, as even the slightest difference to the emulated processor intructions will result in differences between recording (hardware) and playback (emulator). The GitHub thread you linked identifies this issue with the various interpretations of IEEEE rounding. I'm afraid I don't have any insight into the PS2 architecture at this level.

A re-implementation of the games could solve this by figuring out the pad recording stream format and re-recording them in the emulator. However this isn't really what you want.

I seem to recall that other vehicles and objects in Stuntman were animated on splines, so these don't go wrong. The vehicles turn into physical objects if you crash into them. Vehicles like the helicopters are animated on paths. A small handful might have pad input sections but these are very short so if there are discrepancies they won't ever be noticed.

Yes, I also worked on Driver 76, Parallel Lines, and on the PC version of Parallel Lines, as well as on Stuntman, Driv3r, and Driver San Francisco. I was also the producer on Stuntman 2. We had had a pre-production phase on Stuntman 2 lasting about 18 months, and had a complete technical and mission design completed, a load of art assets, and a partial development team, but the demands of Driv3r became a priority; so Stuntman 2 development was moved (at our suggestion) to another Atari studio, Paradigm Entertainment in Texas. Paradigm had recently completed Terminator: Dawn of Fate which was quite a good game but mainly it had a similar technology base to Stuntman. Stuntman 2 came out as Stuntman: Ignition, which was pretty good and implemented the innovative stunt string system. The actual level concepts and general design having been developed principally by Mark Mainey at Reflections.

We also worked on a prototype of a game based on The Transporter that used the Stuntman Engine, with Martin Oliver leading the design. We hadn't got as far as actually looking at licensing or anything but it was looking promising. Sadly, it didn't fit into Ubisoft's plans for the studio when the sale by Atari happened, so it was cancelled.

I did work on Driver San Francisco too, taking the game from concept through to the establishment of the main plot and game mechanisms. The company under Ubisoft was at that stage quite different to what it had been as Reflections however - and my own life had moved on too - and I parted ways with Ubisoft (or they with me) in 2010.

Although I can't really be of much more help with the emulator problem I hope this mini-retrospective has been interesting to you. I'm only a little bit older than you and I do still play a lot of games, although not really games I worked on - I think I had my fill of hearing One Way or Another at the time. It's very nice to hear from someone who got enjoyment from the games we made. We were well aware that our games were often too hard and divided audiences. We changed things slightly in Driver: Parallel Lines and I think that game had about the best content and difficulty balance of all the games I worked on, and it's is probably the one I'm most proud of being involved with.

I had a thought that it's possible that you might have contacted other people involved in Reflections' games. The people most likely to have worked on the vehicle recordings would have been Russ Lazarri and Will Musson. Russ has essentially retired from the industry now and I doubt he's contactable. Will was one of the first Reflections employees and still works at Ubisoft, and might respond; I haven't seen him in years, but he might have some additional insights.

[BenoitAdam94]

One of the posters in the thread you linked identifies correctly that the chases in Stuntman and Driv3r aren't AI, they are input recordings. One of the design team drove the chase car and the pad inputs were recorded as a data stream.

Yup, REDRIVER 2 even have new inputed chase, the chase player is even displayed on bottom left on the screen.

https://github.com/OpenDriver2/REDRIVER2

@SoapyMan

[thisnameislame]

No idea how feasible this is or how to do it, but would it be possible to do something like setting up a modded PS2 to listen for and record the transform values for the chase cars and then try and reverse engineer a formula to compensate for the rounding errors?

[sittingduck77]

This is nothing, I'm sure, but if you look at this segment of a Driver 2 review played on the original PS1, it looks like the same kind of error: https://www.youtube.com/watch?v=GiWCkOScitc&t=6m40s

I don't know anything about PS1 and PS2 hardware, but it would be funny if looking at the PS1 could solve something on the PS2 :-)