[VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files

[ajeyabsfujitsu]

Description

I am exploring the incubator-gluten repository and trying to understand how Java/Scala code interacts with the native C++ code. However, I am having difficulty identifying where and how this handover happens within the codebase.

An explanation on this process would be extremely helpful.

Java/Scala to C++ Handover - I want to know the specific parts of the codebase where Java/Scala calls are handed over to native C++ code. I’m unable to locate the key files or classes where this interaction is defined. If there are specific entry points or patterns used for this handover, please provide details.

Profiling C++ Code - I’m attempting to use perf to profile the execution and see which C++ functions or files are being called. However, the perf output only shows functions from jibjvm.so or libc.so, and no Gluten or Velox-specific C++ functions are visible. Are there additional configurations or profiling tools recommended for this?

[FelixYBW]

https://github.com/apache/incubator-gluten/tree/main/cpp/core/jni is the place where java call c++

if your gluten is running, from perf you can see the hotspot of velox functions

[ajeyabsfujitsu]

Thanks for the response.

In perf call graph everything is just libjvm and libc. Is there something I am missing?

In async profiler, I get the java/scala functions called, but I want to know at what point native functions take over.

In the files in cpp/core/jni folder, should I look out mainly for "JNI_EXPORT" implementations?

[FelixYBW]

you may go through all the files in the jni folder. Any JNI call can trigger the native function.

this folder is for Velox backend:
https://github.com/apache/incubator-gluten/tree/main/cpp/velox/jni

[zhztheplayer]

Java/Scala to C++ Handover

The code used for C++ / Java communication is usually written in files that have "JniWrapper" as names or name suffixes. VeloxJniWrapper.cc, ...JniWrapper.scala, etc.

In perf call graph everything is just libjvm and libc. Is there something I am missing?

Not sure, but perhaps something like this might happen when the debug symbols are missing from the profiler. Note Gluten deletes the used native library files from tmp/ while the JVM is exiting. Could set spark.gluten.sql.debug.keepJniWorkspace=true to avoid the deletion if it's the reason.

[ajeyabsfujitsu]

@FelixYBW Thanks. I will go through that folder to try finding the calls.

[ajeyabsfujitsu]

@zhztheplayer Thanks for the response.

Debug symbols were installed with this -

apt install -yq libc6-dbg libstdc++6-12-dbg openjdk-${JDK_VERSION}-dbg

Is there any other command that you are aware of?

I am trying to run tpch workload runs.

spark.gluten.sql.debug.keepJniWorkspace=true

I tried setting this flag in tpch_parquet.sh but getting an error

java.util.NoSuchElementException: spark.gluten.sql.debug.keepJniWorkspaceDir

Should this flag be added somewhere else?

[VaibhavFRI]

@ajeyabsfujitsu you should set spark.gluten.sql.debug.keepJniWorkspaceDir=<path_to_dir> conf also for spark.gluten.sql.debug.keepJniWorkspace=true to work.

[ajeyabsfujitsu]

Thanks Vaibhav.

[VaibhavFRI]

@zhztheplayer I tried setting the spark.gluten.sql.debug.keepJniWorkspace=true conf, but I still get libjvm.so and no gluten function calls. Is anything else needs to be set in perf?

[rajatma1993]

@zhztheplayer @FelixYBW

Based on your suggestions, I analyzed the file "Jniwrapper" to trace the Java calls that callback to C++. However, I noticed that most of the Gluten functions are pointing to Scala files rather than Java, which should ultimately lead to the Jniwrapper.

I would like to understand the flow of Gluten functions that transition from Java to C++. Could you please guide me on which folders and files I should refer to for better clarity?