DPU timeout on VCK190-ES

[ksstms]

Hi!
I'm trying to build the XVDPU-TRD (version 2021.1) for the VCK190-ES board.
The build succeeds, but when I run test_dpu_runner_mt I get the following error:

root@xilinx-vck190-2021_1:~# test_dpu_runner_mt resnet50-xvdpu-batch-1.xmodel x_0 2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1028 08:34:23.363327   969 test_dpu_runner_mt.cpp:399] create runner ... 0/2
I1028 08:34:27.011866   969 test_dpu_runner_mt.cpp:399] create runner ... 1/2
I1028 08:34:27.097680   969 performance_test.hpp:73] 0% ...
I1028 08:34:27.808851   978 test_dpu_runner_mt.cpp:339] thread 0 batch 0 error_counter 1 ok_counter 162
I1028 08:34:28.925685   979 test_dpu_runner_mt.cpp:339] thread 1 batch 0 error_counter 1 ok_counter 415
I1028 08:34:33.097848   969 performance_test.hpp:76] 10% ...
I1028 08:34:39.097992   969 performance_test.hpp:76] 20% ...
W1028 08:34:39.225816   979 xrt_cu.cpp:188] cu timeout! device_core_idx 0  handle=0xaaaaf80ce820 ENV_PARAM(XLNX_DPU_TIMEOUT) 10000 state 1 ERT_CMD_STATE_COMPLETED 4 ms 10009  bo=1 is_done 0 
I1028 08:34:39.225914   979 xrt_cu.cpp:99] Total: 10009632us	ToDriver: 18446744073705156us	ToCU: 2096us	Complete: 2193us	Done: 10009737us
F1028 08:34:39.226002   979 dpu_control_xrt_xv_dpu.cpp:193] dpu timeout! core_idx = 0
 AP 1  LSTART 363  LEND 363  CSTART 348  CEND 346  SSTART 0  SEND 0  MSTART 429  MEND 429  CYCLE_L 1501170466  CYCLE_H 0 
*** Check failure stack trace: ***
W1028 08:34:39.228508   978 xrt_cu.cpp:188] cu timeout! device_core_idx 0  handle=0xaaaaf80a2570 ENV_PARAM(XLNX_DPU_TIMEOUT) 10000 state 1 ERT_CMD_STATE_COMPLETED 4 ms 10010  bo=1 is_done 0 
I1028 08:34:39.243094   978 xrt_cu.cpp:99] Total: 10010122us	ToDriver: 18446744073705160us	ToCU: 2092us	Complete: 2195us	Done: 10010225us
F1028 08:34:39.243213   978 dpu_control_xrt_xv_dpu.cpp:193] dpu timeout! core_idx = 0
 AP 1  LSTART 363  LEND 363  CSTART 348  CEND 346  SSTART 0  SEND 0  MSTART 429  MEND 429  CYCLE_L 1503747873  CYCLE_H 0 
*** Check failure stack trace: ***
Aborted

And this is my kernel log: dmesg.txt

In order to build the project for the ES device I made 2 modifications:

• Vivado project tcl script:

  set proj_board [get_board_parts "*:vck190_es:*" -latest_file_version] 
  

• and the ISP IP's make call in the platform Makefile:

  $(ISP_IP): 
      make -C $(ISP_DIR) ip XPART=xcvc1902-vsva2197-1LP-e-S-es1 
  

I also set BATCH_N to 1 in the DPU configuration in order to save build time.
Should I change anything else to make this work on the ES board? Unfortunately I can't test the production version.

[xlinyg]

For 2021.1 version ES1 platform , before running apps of xvdpu, need firstly run workaround for ES1 silicon.
After VCK190-ES1 board is booting up , create a script with below content, and run it on the ES1 board.

'''
for i in {0..39}
do
for j in {1..8}
do
a=0x20000000000
b=0x31000
devmem $[a+b+(i<<23)+(j<<18)] 32 0
done
done
'''

[ksstms]

Thank you @xlinyg, that solves the problem.
It works when I boot from an SD card. However, we'd like to use TFTP boot and NFS rootfs for this project. I configured them in the PetaLinux project, and rebuilt it. Now the system boots up, but running test_dpu_runner_mt just freezes everything. I tried to run your workaround script, but it also freezes at the first devmem call.
Do you have any tips to solve that? Should I open a new issue for this?

[xlinyg]

SD card is the supported way. Not sure the issue for TFTP boot.
Please open a new issue.