Problem with simulating executable linked with multiple objectives

[macto94]

I'm new to Accel-Sim. I've recently been setting up accel-sim and trying various things. I have a question.

I'm starting to simulate simple toy examples.

Q1)
And I've confirmed that GPGPU-Sim works well with an executable created by compiling a single test.cu.
However, the simple framework (which is extended version of test.cu) involves making several header files, CUDA files, etc., into objectives and then linking them into an executable, but this doesn't seem to work. (Error message said that there is no PTX can be extracted)

Is there a specific way to handle this?

following works :

nvcc test.cu -lcudart -o test

following doesn't work :

EXECUTABLE := test
CU_FILES   := test.cu test2.cu util.cu
CU_DEPS    := test2.cuh utility.cuh
CC_FILES   := main.cpp
###########################################################
ARCH=$(shell uname | sed -e 's/-.*//g')

OBJDIR=objs
CXX=g++ -m64
CXXFLAGS=-O3 -Wall -std=c++17
LDFLAGS=-L/usr/local/cuda/lib64/ -lcudart
NVCC=nvcc
NVCCFLAGS=-O3 -m64 -ccbin /usr/bin/gcc -std=c++17 \
  -gencode arch=compute_60,code=sm_60 \
  -gencode arch=compute_70,code=sm_70 \
  -gencode arch=compute_75,code=sm_75 \
  -gencode arch=compute_80,code=sm_80 \
  -gencode arch=compute_86,code=sm_86

OBJS=$(OBJDIR)/main.o  $(OBJDIR)/test_gemv.o $(OBJDIR)/fast_gemv.o $(OBJDIR)/utility.o

all: $(EXECUTABLE)
default: $(EXECUTABLE)
dirs:
        mkdir -p $(OBJDIR)/
clean:
        rm -rf $(OBJDIR) *.ppm *~ $(EXECUTABLE)

$(EXECUTABLE): dirs $(OBJS)
        $(CXX) $(CXXFLAGS) -o $@ $(OBJS) $(LDFLAGS)
$(OBJDIR)/%.o: %.cpp
        $(CXX) $< $(CXXFLAGS) -c -o $@
$(OBJDIR)/%.o: %.cu $(CU_DEPS)
        $(NVCC) $< $(NVCCFLAGS) -c -o $@

Q2)
besides the benchmarks in gpu-app-collection, can I simulate my custom kernel using accel-sim.out or run_simulations.py?

Sorry if the information is already available somewhere. I'm just starting to study this, and I'm feeling quite overwhelmed, so I'm writing this post. I'll wait for any kind of response, thanks!

[William-An]

For Q1) Can you post the complete error log as well to aid debugging? It could be that you are using g++ to link the overall executable which drops the PTX section (could be related)?

For Q2) Yes you can run your own kernel with accel-sim.out or run_simulations.py. In this case you would need to collect the SASS trace yourself via the NVBit tracing tool and set the commandline arguments properly. With run_simulations.py, you will need to do an extra set and modify the util/job_launching/apps/define-all-apps.yml configuration file to specify the directories.

[macto94]

Q1) Thank you for your response. I'm new to this area, so I think I lack a thorough understanding of compilers like g++ and nvcc. It seems likely that what you mentioned might be the cause. But changing main.cpp to main.cu or adding the -x cu flag as advised in the link you provided doesn't seem to solve the issue. Am I misunderstanding something?
error log is like following: (and I'm not using PyTorch, and CDP is disabled)

*** Initializing Memory Statistics ***
GPGPU-Sim uArch: performance model initialization complete.
self exe links to: /root/accel-sim-framework/gpu-simulator/gpgpu-sim/examples/gemv/gemv
self exe links to: /root/accel-sim-framework/gpu-simulator/gpgpu-sim/examples/gemv/gemv
11.0
GPGPU-Sim PTX: __cudaRegisterFatBinary, fat_cubin_handle = 1, filename=default
self exe links to: /root/accel-sim-framework/gpu-simulator/gpgpu-sim/examples/gemv/gemv
Running md5sum using "md5sum /root/accel-sim-framework/gpu-simulator/gpgpu-sim/examples/gemv/gemv "
dc70fddaa3bc2a5ad69aa1e3c1405340  /root/accel-sim-framework/gpu-simulator/gpgpu-sim/examples/gemv/gemv
self exe links to: /root/accel-sim-framework/gpu-simulator/gpgpu-sim/examples/gemv/gemv
cuobjdump info    : No PTX file found to extract from '/root/accel-sim-framework/gpu-simulator/gpgpu-sim/examples/gemv/gemv'. You may try with -all option.
WARNING: Number of ptx in the executable file are 0. One of the reasons might be
        1. CDP is enabled
        2. When using PyTorch, PYTORCH_BIN is not set correctly
self exe links to: /root/accel-sim-framework/gpu-simulator/gpgpu-sim/examples/gemv/gemv
self exe links to: /root/accel-sim-framework/gpu-simulator/gpgpu-sim/examples/gemv/gemv
11.0
GPGPU-Sim PTX: __cudaRegisterFatBinary, fat_cubin_handle = 2, filename=default
GPGPU-Sim PTX: __cudaRegisterFunction _Z32check_int2_quantized_correctnessP7uint2_4P6__halfS2_S1_S1_i : hostFun 0x0x5adcf9201f70, fat_cubin_handle = 2
GPGPU-Sim PTX: loading globals with explicit initializers...
Segmentation fault (core dumped)

Q2) Thank you very much. Just a few hours ago, I was reading entire source code to study, and I found the part you mentioned. Now, I am trying to gain a deeper understanding of the trace-driven method. Thanks again :)

[William-An]

Did you include cuda function call in the main.cu function? If you don't include that, likely compiler will optimize it out and won't include the PTX code.

[macto94]

Oh, now I see. I organized the code like following:
int main in main.cu : call void test function
void test in test.cu : call some "solver" which contains launching CUDA device kernel

I really appreciate your clear explanation and help. I will try it !!!