[BUG] MPS tests fail when running the full suite

[spernsteiner]

I built the MPS locally using the same build process as in CI and ran the full test suite without QUICK=1. Two of the test cases fail partway through:

scenarios/normal_4 (../src/rts.no_self_test)
Opening scenarios/normal_4.cases...
<...>
Running case 272
SENDING:
CHECKING: #A 0 \[0 0\]
CHECKING: #A 0 \[0 0\] succeeded
CHECKING: #A 1 \[0 0\]
CHECKING: #A 1 \[0 0\] succeeded
SENDING:
SENDING: B 0 0 0
SENDING: B 1 0 0
SENDING: B 2 0 0
SENDING: B 3 0 2
SENDING: B 0 1 0
SENDING: B 1 1 0
SENDING: B 2 1 0
SENDING: B 3 1 2
SENDING: B 0 2 0
SENDING: B 1 2 0
SENDING: B 2 2 0
SENDING: B 3 2 0
SENDING:
CHECKING: #A 0 \[1 0\]
CHECKING: #A 0 \[1 0\] failed
Test 273 failed

scenarios/normal_5a (../src/rts.no_self_test)
Opening scenarios/normal_5a.cases...
<...>
Running case 272
SENDING: 
CHECKING: HW ACTUATORS OFF OFF
CHECKING: HW ACTUATORS OFF OFF succeeded
SENDING: 
SENDING: B 0 0 0
SENDING: B 1 0 0
SENDING: B 2 0 0
SENDING: B 3 0 2
SENDING: B 0 1 0
SENDING: B 1 1 0
SENDING: B 2 1 0
SENDING: B 3 1 2
SENDING: B 0 2 0
SENDING: B 1 2 0
SENDING: B 2 2 0
SENDING: B 3 2 0
SENDING: 
CHECKING: HW ACTUATORS ON *OFF *
CHECKING: HW ACTUATORS ON *OFF * failed
Test 273 failed

The current CI doesn't detect these errors because it runs the test suite with QUICK=1, which causes it to run only the first 3 cases in scenarios normal_4 and normal_5a.

Acceptance criteria

• MPS tests pass when running python3 run_all.py without QUICK=1

[podhrmic]

Good find. I can indeed replicate the issue. I will look at it.

[kiniry]

Note: GaloisInc/HARDENS#130

…

On Jun 2, 2024, at 10:09, Michal Podhradsky ***@***.***> wrote: Good find. I can indeed replicate the issue. I will look at it. — Reply to this email directly, view it on GitHub <#56 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADU5P3JTVHKZPTHC6XRJ2DZFNGTZAVCNFSM6AAAAABIR2CZKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTHE2DKOBVGU>. You are receiving this because you are subscribed to this thread.

[podhrmic]

The generated scenario has a bug (see the updated HARDENS ticket).

[podhrmic]

Given the issue seem to be with the test infrastructure (the generated test cases) rather than with the code itself, I suggest we either run the tests with QUICK=1 or delete the offending cases (maybe cap at 272 cases for each scenario).