Error when running on Vitis Getting Started tutorial: "Vitis Flow 101 – Part 4 : Build and Run the Example" on Alveo U250

[danna2019]

Following errors occured when running ./app.exe using same code of a vector add sample successfully ran on sw_emu/hw_emu with 'TEST PASSED' on my Alveo U250 host.

starra@alveo:~/Vitis-Tutorials/Getting_Started/Vitis/example/u250/hw$ ./app.exe 
INFO: Found Xilinx Platform
INFO: Loading 'vadd.xclbin'
XRT build version: 2.11.634
Build hash: 5ad5998d67080f00bca5bf15b3838cf35e0a7b26
Build date: 2021-06-08 22:08:45
Git branch: 2021.1
PID: 5819
UID: 1000
[Fri Sep 10 06:43:07 2021 GMT]
HOST: alveo016
EXE: /home/starra/Vitis-Tutorials/Getting_Started/Vitis/example/u250/hw/app.exe
[XRT] ERROR: See dmesg log for details. err=-2
[XRT] ERROR: failed to load xclbin: Invalid argument
[XRT] ERROR: Unexpected error memidx == -1
[XRT] ERROR: Unexpected error memidx == -1
[XRT] ERROR: Unexpected error memidx == -1
Segmentation fault (core dumped)

dmesg said:

...
[  257.381157] xocl 0000:d8:00.1:  ffff8b7eb6e860b0 xocl_read_axlf_helper: check interface uuid
[  257.381166] xocl 0000:d8:00.1:  ffff8b7eb6e860b0 xocl_fdt_check_uuids: Can not find uuid f2f6c5e1273e78948f2c4806221462f2
[  257.381203] xocl 0000:d8:00.1:  ffff8b7eb6e860b0 xocl_read_axlf_helper: interface uuids do not match
[  257.381224] xocl 0000:d8:00.1:  ffff8b7eb6e860b0 xocl_read_axlf_helper: Failed to download xclbin, err: -22
[  257.390111] show_signal_msg: 18 callbacks suppressed
[  257.390113] app.exe[2599]: segfault at 0 ip 0000561184c13e6d sp 00007ffd3656c810 error 6 in app.exe[561184c13000+c000]
[  257.390119] Code: ff ff 8b 95 bc fe ff ff 48 63 d2 48 8d 0c 95 00 00 00 00 48 8b 95 08 ff ff ff 48 01 d1 99 c1 ea 14 01 d0 25 ff 0f 00 00 29 d0 <89> 01 e8 cc f6 ff ff 8b 95 bc fe ff ff 48 63 d2 48 8d 0c 95 00 00
...

How can I fix it?

[randyh62]

It seems like there might be a version mismatch for XRT between the development environment and the deployment environment.

[danna2019]

xbmgmt said:

$ xbmgmt examine
System Configuration
  OS Name              : Linux
  Release              : 5.4.0-84-generic
  Version              : #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021  Machine              : x86_64
  CPU Cores            : 72
  Memory               : 386615 MB  Distribution         : Ubuntu 20.04.2 LTS
  GLIBC                : 2.31
  Model                : ProLiant DL380 Gen10 

XRT
  Version              : 2.11.634
  Branch               : 2021.1
  Hash                 : 5ad5998d67080f00bca5bf15b3838cf35e0a7b26
  Hash Date            : 2021-06-08 22:08:45
  XOCL                 : 2.11.634, 5ad5998d67080f00bca5bf15b3838cf35e0a7b26
  XCLMGMT              : 2.11.634, 5ad5998d67080f00bca5bf15b3838cf35e0a7b26

Devices present
  [0000:d8:00.0] : xilinx_u250_gen3x16_base_3 
  [0000:86:00.0] : xilinx_u250_gen3x16_base_3 
  [0000:37:00.0] : xilinx_u250_gen3x16_base_3

xrt version.json said:

$ cat /opt/xilinx/xrt/version.json 
{
  "BUILD_VERSION" : "2.11.634",
  "BUILD_VERSION_DATE" : "Tue, 08 Jun 2021 22:08:45 -0700",
  "BUILD_BRANCH" : "2021.1",
  "VERSION_HASH" : "5ad5998d67080f00bca5bf15b3838cf35e0a7b26",
  "VERSION_HASH_DATE" : "Tue, 8 Jun 2021 20:06:48 -0700"
}

xclbinutil said:

$xclbinutil  --info --input Vitis-Tutorials/Getting_Started/Vitis/example/u250/hw/vadd.xclbin
XRT Build Version: 2.11.634 (2021.1)
       Build Date: 2021-06-08 22:08:45
          Hash ID: 5ad5998d67080f00bca5bf15b3838cf35e0a7b26
------------------------------------------------------------------------------
Warning: The option '--output' has not been specified. All operations will    
         be done in memory with the exception of the '--dump-section' command.
------------------------------------------------------------------------------
Reading xclbin file into memory.  File: Vitis-Tutorials/Getting_Started/Vitis/example/u250/hw/vadd.xclbin

==============================================================================
XRT Build Version: 2.11.634 (2021.1)
       Build Date: 2021-06-08 22:08:45
          Hash ID: 5ad5998d67080f00bca5bf15b3838cf35e0a7b26
==============================================================================
xclbin Information
------------------
   Generated by:           v++ (2021.1) on 2021-06-09-14:19:56
   Version:                2.11.634
   Kernels:                vadd
   Signature:              
   Content:                Bitstream
   UUID (xclbin):          0ef1b67e-3fd9-a428-91af-087ba0aee0f5
   UUID (IINTF):           f2f6c5e1273e78948f2c4806221462f2
   Sections:               DEBUG_IP_LAYOUT, BITSTREAM, MEM_TOPOLOGY, IP_LAYOUT, 
                           CONNECTIVITY, CLOCK_FREQ_TOPOLOGY, BUILD_METADATA, 
                           EMBEDDED_METADATA, SYSTEM_METADATA, 
                           PARTITION_METADATA, GROUP_CONNECTIVITY, GROUP_TOPOLOGY

I seem they said XRT versions are same as '2.11.634'.

If I should take some information, please how I can take them.

[randyh62]

That looks pretty consistent. Are you able to run any other examples on your U250?

[danna2019]

No, it's first time run.
So, I think I have to started the "Getting Started" first example.
I believe the "Vector Add" example is a most simple example for using Vitis to Alveo U250.

If you have anthor simple examples for the alveo which are I should run before the Vector Add, please let me know.

[randyh62]

There are a number of different examples in the Vitis_Accel_Examples repository. You can build some of these pretty quickly, and run them on your hardware.
I would start with something like Hello World which also happens to use the vadd kernel. These examples have Makefiles to let you build them quickly, and can target a number of different platforms. The examples are also useful for exploring additional features of the system.

[danna2019]

Using Hello World example, 'make all' could be successfully done, but 'make run' caused error below:

./hello_world ./build_dir.hw.xilinx_u250_gen3x16_xdma_3_1_202020_1/vadd.xclbin
Found Platform
Platform Name: Xilinx
INFO: Reading ./build_dir.hw.xilinx_u250_gen3x16_xdma_3_1_202020_1/vadd.xclbin
Loading: './build_dir.hw.xilinx_u250_gen3x16_xdma_3_1_202020_1/vadd.xclbin'
Trying to program device[0]: xilinx_u250_gen3x16_base_3
XRT build version: 2.11.634
Build hash: 5ad5998d67080f00bca5bf15b3838cf35e0a7b26
Build date: 2021-06-08 22:08:45
Git branch: 2021.1
PID: 473665
UID: 1000
[Sat Sep 11 16:36:40 2021 GMT]
HOST: alveo016
EXE: /home/starra/Vitis_Accel_Examples/hello_world/hello_world
[XRT] ERROR: See dmesg log for details. err=-2
[XRT] ERROR: failed to load xclbin: Invalid argument
Failed to program device[0] with xclbin file!
Trying to program device[1]: xilinx_u250_gen3x16_base_3
[XRT] ERROR: See dmesg log for details. err=-2
[XRT] ERROR: failed to load xclbin: Invalid argument
Failed to program device[1] with xclbin file!
Trying to program device[2]: xilinx_u250_gen3x16_base_3
[XRT] ERROR: See dmesg log for details. err=-2
[XRT] ERROR: failed to load xclbin: Invalid argument
Failed to program device[2] with xclbin file!
Failed to program any device found, exit!
make: *** [Makefile:169: run] Error 1

dmesg said same as previous vector add ran:

[154128.592603] xocl 0000:37:00.1:  ffff8b4eb6ab10b0 xocl_read_axlf_helper: check
 interface uuid
[154128.592611] xocl 0000:37:00.1:  ffff8b4eb6ab10b0 xocl_fdt_check_uuids: Can not find uuid f2f6c5e1273e78948f2c4806221462f2
[154128.593307] xocl 0000:37:00.1:  ffff8b4eb6ab10b0 xocl_read_axlf_helper: interface uuids do not match
[154128.593965] xocl 0000:37:00.1:  ffff8b4eb6ab10b0 xocl_read_axlf_helper: Failed to download xclbin, err: -22
[154128.656988] xocl 0000:37:00.1:  ffff8b4eb6ab10b0 xocl_destroy_client: client exits pid(473665)

I seem xocl_fdt_check_uuid is also failed same as previous vector add ran and caused the host application error.
How can I fix it?

I think it's troublesome that my Alveo host had be installed 3 alveo U250 cards.
If 3 cards would need some special configurations or command options, please let me know.

[randyh62]

It seems like there is a configuration issue, though I am not sure where the problem lies. You can debug the Alveo installation using the following guide: https://xilinx.github.io/Alveo-Cards/master/debugging/docs/card-validation.html
Run some of the validation tests to see if your cards are working as expected.

[danna2019]

I ran some of the validation tests on the guide, but all validation tests are successfully done.
How can I handle it?

[randyh62]

If you have three cards, you can try to direct the xclbin to a different card in the system. This would be done in the host application. Currently it defaults to choosing the first card. You could change that to specify a card or hard code it to card 2 or 3?

    std::vector<cl::Device> devices = get_xilinx_devices();
    devices.resize(1);
    cl::Device device = devices[0];

[uday610]

Hello @danna2019 ,

I see the shell is not loaded for U250 platform, assuming you are using latest U250, which is a 2RP platform.

Please do execute this command first:

 sudo xbmgmt partition --scan 

This would show you the name of the shell partition installed in the system. Something like this:

Partitions installed in system:
    xilinx_u250_gen3x16_xdma_shell_3_1
        logic-uuid:
        bd5fb8abab266c3265918257b5048e88
        interface-uuid:
        f2f6c5e1273e78948f2c4806221462f2

For example, in my above sample output you see a Shell Partition (the name contain the word "shell" which is installed in system. Now copy paste that name and install the shell partition. For example

 sudo xbmgmt partition --program --name xilinx_u250_gen3x16_xdma_shell_3_1

After doing this do xbutil list again and you will see the shell partition is showing.

[uday610]

I forgot to mention in my previous reply regarding the --card option. As you have 3 cards you need to specify BDF to choose the card. So you should do this one after another for each card to load the shell.

 sudo xbmgmt partition --program --name xilinx_u250_gen3x16_xdma_shell_3_1 --card <bdf>

[danna2019]

Thank you @uday610.
After 'xbmgmt partition', I could successfully run 'Hellow World' example like below:

$ make run DEVICE=/opt/xilinx/platforms/xilinx_u250_gen3x16_xdma_3_1_202020_1/xilinx_u250_gen3x16_xdma_3_1_202020_1.xpfm 
./hello_world ./build_dir.hw.xilinx_u250_gen3x16_xdma_3_1_202020_1/vadd.xclbin
Found Platform
Platform Name: Xilinx
INFO: Reading ./build_dir.hw.xilinx_u250_gen3x16_xdma_3_1_202020_1/vadd.xclbin
Loading: './build_dir.hw.xilinx_u250_gen3x16_xdma_3_1_202020_1/vadd.xclbin'
Trying to program device[0]: xilinx_u250_gen3x16_xdma_shell_3_1
Device[0]: program successful!
TEST PASSED

Unfortunately, I got another error when running the Vector add example on 'Getting_Started/Vitis/example' (originally this issue concerning, but it's a different error) below:

$ ./app.exe
INFO: Found Xilinx Platform
INFO: Loading 'vadd.xclbin'
terminate called after throwing an instance of '__gnu_cxx::recursive_init_error'
  what():  std::exception
Aborted (core dumped)

How can I fix it?

(bit curios, I already ran 'xbmgmt partition' accoding to the Alveo getting started instructions before open this issue, why and when partitions have been cleared...)

[danna2019]

Oh, I got solve this error.
xbutil reset -d 0
ran before executes ./app.exe.

Thank you for your advices, @randyh62 and @uday610.