How to correctly setup vortex software after synthesis for Xilinx FPGA?

[spisladqo]

In https://github.com/vortexgpgpu/vortex/blob/master/docs/fpga_setup.md, there's no information on how to build xrt driver and runtime after synthesis on Xilinx FPGA. Without doing so, when trying to run the tests, the following error is shown:
libvortex.a: No such file or directory

It seems to work if I do $ make software from vortex/build, but I'm not sure if it is correct, because no xrt driver or runtime is built there.

Also, simple $ make build seems to lead to incorrect results, because, after following instruction on synthesis from document above (synthesized vortex with 2 cores), building with $ make build and running test using following command:
user4ferrum@ferrum:~/Vlasenco/1/vortex/build$ FPGA_BIN_DIR=/home/user4ferrum/Vlasenco/1/vortex/hw/syn/xilinx/xrt/05_02_2025_xilinx_u50_gen3x16_xdma_5_202210_1_hw/bin TARGET=hw PLATFORM=xilinx_u50_gen3x16_xdma_5_202210_1 ./ci/blackbox.sh --driver=xrt --app=kernel1 --args=-n4096

Where kernel1 is simple matrix multiplication, the following output is shown (suspicious part is in bold):

Running: make -C ./ci/../runtime/xrt > /dev/null
Running: OPTS="-n4096" make -C ./ci/../tests/opencl/kernel1 run-xrt
make: Entering directory '/home/user4ferrum/Vlasenco/1/vortex/build/tests/opencl/kernel1'
...
CONFIGS: num_threads=4, num_warps=4, num_cores=1, num_clusters=1, socket_size=1, local_mem_base=0x1ffff0000, num_barriers=2
...

It looks like default configuration when running under rtl/simx simulation, and the command above takes very long to execute. My hunch is that it somehow messes up simulation and execution on FPGA. Additionally, with $ make software this line in bold isn't shown and execution is much faster. Could you please tell me if $ make software is the correct way to setup execution on FPGA (though xrt driver and runtime are not built by it), and if not, which command should one use?

[Udit8348]

Hi, those docs may be slightly out of date --

You can take a look at my instructions for fpga syn on my fork

Note: There is a bug that we have not solved yet meaning that you need to run the synthesis makefile in the root of the project not the build folder. The link I shared above, mentions this as well as how to work around it. As long as you read all the steps carefully, the bitstream generation should work for you. If you have any questions, feel free to follow up here!

[spisladqo]

Hi, those docs may be slightly out of date --

You can take a look at my instructions for fpga syn on my fork

Note: There is a bug that we have not solved yet meaning that you need to run the synthesis makefile in the root of the project not the build folder. The link I shared above, mentions this as well as how to work around it. As long as you read all the steps carefully, the bitstream generation should work for you. If you have any questions, feel free to follow up here!

Thanks for your quick answer! Well, it seems I understood original instruction correctly and did almost everything as you advise. Though original instruction doesn't say to insert nohup into this command:
PREFIX=test1 PLATFORM=xilinx_u50_gen3x16_xdma_5_202210_1 TARGET=hw NUM_CORES=1 nohup make > build_u50_hw_1c.log 2>&1
When I tried to insert it as your instruction says, build failed with the output of the following form:

WARNING: [IP_Flow 19-3833] Unreferenced file from the top module is not packaged: '/home/user4ferrum/Vlasenco/vortex/hw/syn/xilinx/xrt/test1_04_feb_xilinx_u50_gen3x16_xdma_5_202210_1_hw/src/VX_sfu_perf_if.sv'.
WARNING: [IP_Flow 19-3833] Unreferenced file from the top module is not packaged: '/home/user4ferrum/Vlasenco/vortex/hw/syn/xilinx/xrt/test1_04_feb_xilinx_u50_gen3x16_xdma_5_202210_1_hw/src/VX_ibuffer.sv'.
...
INFO: [Common 17-14] Message 'IP_Flow 19-3833' appears 100 times and further instances of the messages will be disabled. Use the Tcl command set_msg_config to change the current settings.

    while executing
"ipx::package_project -root_dir $path_to_packaged -vendor xilinx.com -library RTLKernel -taxonomy /KernelIP -import_files -set_current false"
    (file "/home/user4ferrum/Vlasenco/vortex/hw/syn/xilinx/xrt/package_kernel.tcl" line 146)

    while executing
"source ${script_path}/package_kernel.tcl"
    (file "/home/user4ferrum/Vlasenco/vortex/hw/syn/xilinx/xrt/gen_xo.tcl" line 38)
INFO: [Common 17-206] Exiting Vivado at Tue Mar  4 13:35:58 2025...
make: *** [Makefile:168: test1_04_feb_xilinx_u50_gen3x16_xdma_5_202210_1_hw/bin/vortex_afu.xo] Error 1

Without nohup, bitstream generation works...

[Udit8348]

That is interesting, for our systems nohup seems to work. nohup runs the command in the background such that it does not block your current terminal's input command line. So, I would expect it to be functionally the same. However, it is good that you were able to get the bitstream generated without nohup.

Can I ask a few questions about your setup?

• how long did you successful bitstream generation take?
• what are your system's details (cpu, arch, os, etc)
• now that you have a successfully generated bitstream, what hardware are you planning on running it on?
• what hardware configuration did you synthesize for? (Cores/Warps/Threads)
• can you share the log files for the successful and unsuccessful run?

[spisladqo]

1. Around 3-4 hours.
2. CPU AMD Ryzen 9 7900X (24) @ 4.700G, x86_64; OS Ubuntu 22.04.01; kernel version 5.15.
3. I'm using Xilinx Alveo U50.
4. Are threads and warps configurable for FPGA? How can I configure them? I didn't find any information about this in README's or Makefiles, so I didn't configure them. And for the cores, I chose 2 just to test.
5. It seems like nohup was not the reason of unsuccessful build, because now after reboot it works.

In addition to my issue, today I have programmed my FPGA with 2-cored vortex, but faced another problem when running make (as your instruction states, I should follow original instruction, which says to run simple make):

user4ferrum@ferrum:~/Vlasenco/vortex/build$ make
...

/home/user4ferrum/Vlasenco/2/vortex/runtime/xrt/vortex.cpp:69:9: error: ‘xrtDeviceHandle’ does not name a type; did you mean ‘xclDeviceHandle’?
   69 | typedef xrtDeviceHandle xrt_device_t;
  	|     	^~~~~~~~~~~~~~~
  	|     	xclDeviceHandle
compilation terminated due to -Wfatal-errors.
...

And again, make software finishes without errors and xrt driver is working. I've met the above problem earlier, but I don't know the reason for it, because I've sourced both settings64.sh and setup.sh from Xilinx, and vortex's ci/toolchain_env.sh. All the xrt libraries seem to be present on my PC. In the past, I tried to change include guards or copy definitions of xrtDeviceHandle and other types that caused errors into vortex.cpp, but a lot of other things went wrong, so I didn't elaborate further. I'm really confused with this error and I didn't find any patterns of when it occurs.

[spisladqo]

Also, @Udit8348, can you please take a look at my other issue #232? It would help me a lot for the small experiment I described in #205.

[Udit8348]

Thank you for the info! Glad that you resolved the nohup issue too

Here is an example make command that demonstrates how to set warp and thread count using CONFIG

CONFIGS="-DNUM_WARPS=4 -DNUM_THREADS=16" PREFIX=1C_4W_16T_64b_U50 PLATFORM=xilinx_u50_gen3x16_xdma_5_202210_1 TARGET=hw NUM_CORES=1 nohup make > 1c-4w-16t-64b-u50.log 2>&1 &

I have seen that issue before, but I forget exactly how I resolved it. Is your branch pulled upto date? I think that fixed it for me

Yes, I took a look at #232, however it's slightly out of my domain at the moment. I need to read more into it, but in the mean time I can ask my labmates

[spisladqo]

Thank you for your reply.

I will try to configure vortex using the command you provided tomorrow. It's a shame I didn't try to use CONFIGS parameter for that earlier.

My branch is, indeed, out of date. I will try updating it and see if I face this problem again.

I am very grateful for your help. If there is an opportunity, please ask your labmates.