Verilator incorrectly simulates some regression suites

[davidharrishmc]

running regression-wally with Verilator as the defaultsim fails on 24 test suites, mostly on floating-point suites with large signatures. These tests pass in Questa. The f tests mostly fail at result 1020 and d at 510, which suggests a memory size issue.

harris@chips:~/cvw/sim/verilator$ wsim -s verilator rv32gc arch32f
Config=rv32gc tests=arch32f sim=verilator gui=False args=''
Running Verilator on rv32gc arch32f
make: Entering directory '/home/harris/cvw/sim/verilator'
mkdir -p /home/harris/cvw/sim/verilator/logs
wkdir/rv32gc_arch32f/Vtestbench +TEST=arch32f
%Warning: System has stack size 100000 kb which may be too small; suggest 'ulimit -c 156273' or larger
  Error on test rv32i_m/F/src/fadd_b11-01.S result        1020: adr = 8007c100 sim (D$) deadbeef signature = 00000000
%Error: /home/harris/cvw/testbench/testbench.sv:879: Verilog $stop

The bad sw to 8007c100 occurs at time 163,010 in both Questa and Verilator. The value DEADBEEFDEADBEEFDEADBEEF00000000 should be written to the cache way 2 word 0 address 4. Both Questa and Verilator show this happening.

DCacheFlushStart is asserted at 3,006,250 in Questa but not in Verilator.

PCM is 80023EA4 at time 999999 in Verilator but at 999355 in Questa.
PCM is 80012074 at time 499999 in V but 505290 in Q
80003468 at time 99,999 in both
200k Questa and Verilator in sync
V at 80012068 at 499825, Q at 505260
300k sync
400k out of sync
349924 out of sync (maybe ok)
320k in sync
330k in sync
339,887 in sync
keep searching later

[davidharrishmc]

PCM is the same in Verilator and Questa up to 8000E1EC at time 390630.
At this time, Verilator advances to 8000E1F0, while Questa doesn't advance until 390800.
The discrepancy comes up during a flw instruction at e1ec. At 390440, LineDirty = 0 in Verilator but 1 in Questa, so Questa has to write back the line, causing it to take longer.

The read is from 8006f100 at 390430 (flw in Execute stage), so NextSet = 100.
At 390440, AnyMiss = 1 because the address is not in the cache. Verilator chooses VictimWay = 0x2, while Questa choose 0x4.

At 390440, CacheSetTag = 0x04 (based on NextSet).
CurLUR = 011 for Verilator, 110 for Questa.
In Questa, LRUMemory[0:8] each are 6, and [9:63] are all 0. It does not show up in Verilator.

LRUMemory[4] is the relevant one. It is 0 up to time 376320, then becomes 110 in both sims. In Questa it remains 110, while in Verilator it becomes 011 at 376870, which is during a fsw to 807d104 (PCM = 800d9f8). CacheSetTag is 4 here too just before the clock, and so is PAdr[5:0]. However, NextLRU is 011 in Verilator and 110 in Questa, explaining why LRUMemory[4] changes.

LRUUpdate = 101 in Verilator and 110 in Questa at 376860. WayExpanded is 100 and 000, respectively. This comes from Way = 10 (Verilator) / 00 (Questa), which come from the same in HitWayEncoded, which come from HitWay = 0100 in Verilator and 0001 in Questa.

The LRUMemory[0:9] remain the same up to time 376600, when LRUMemory[0] becomes 011 in Verilator but remains 110 in Questa.

At 376600, the FLW at 8000d9e0 enters writeback and updates LRU. CacheSetTag was 00, so LRUMemory[0] is affected. At 376590, HitWay was 0100 on Verilator and 0001 on Questa. The logic computing NextLRU is correct, so the root issue so far is the difference in HitWay.

The tags are different at time 376600. Tags and Valid bits for set 0 are at the end of the lines below (Way 0, 1, 2, 3). The tag 8006f is in way 2 on Verilator and Way 0 on Questa.

Verilator 376600: 011 000 000 000 110 000 000 000 000 000 08006d 1 08007c 1 08006f 1 08007d 1
Questa# 376600: 110 000 000 000 110 000 000 000 000 000 08006f 1 08007c 1 08006e 1 08007d 1

Set 0 tags are identical up to time 376540. At 376540, in set 0, 8006f is written to way 2 in Verilator and way 0 in Questa
because of a flw starting at 376360.

SetValid is pulsed at 376530 to write the fetched line. At that time, VictimWay = 0001 in both sims. Thus, SetValidEN is also pulsed at that time in Way0 for both sims.

[davidharrishmc]

In summary, Verilator writes the wrong cache way memory. At time 376540, both CacheWays[0] and [2] receive addr=00, ce = 1, and din = 080006F. CacheWays[0] receives we=1, while CacheWays[2] receives we=0. However, CacheWays[2] ram0 is written with the 80006F, while CacheWays[0] is not. See the waves below.

[davidharrishmc]

Submitted to Verilator:
verilator/verilator#5177

[davidharrishmc]

Changing DCACHE_NUMWAYS to 1 in rv32gc and rv64gc config.vh avoids the memory write to wrong module problem and temporarily allows some progress here.

The remaining failures are in buildroot and wally64priv

v64i_m/privilege/src/WALLY-cboz-01.S succeeded.  Brilliant!!!
  Error on test rv64i_m/privilege/src/WALLY-cbom-01.S result          30: adr = 0000000080003200 sim (D$) 0000002100000020 signature = deadbeefdeadbeef

buildroot has a problem passing INSTR_LIMIT (see issue #832)

harris@chips:~/cvw$ wsim --sim verilator  buildroot buildroot --args +INSTR_LIMIT=1400000
Config=buildroot tests=buildroot sim=verilator gui=False args='+INSTR_LIMIT=1400000'
%Error: Invalid option: +INSTR_LIMIT=1400000

[davidharrishmc]

Fixed with PR #835

[davidharrishmc]

Fixed with PR #847
Verilator is now fully working with testbench, but not testbench_fp.