[CH] Gluten failed to load META-INF, causing abnormal execution

[gleonSun]

Backend

CH (ClickHouse)

Bug description

[Expected behavior]
Gluten can load the required Components and Backend, and function normally.

[actual behavior]
Gluten failed to load the required Components and Backend during initialization, resulting in abnormal execution.

RootCause

The ResourceUtil#getResources(java.util.regex.Pattern) method loads META-INF from the java.class.path, but it might not be the META-INF from the project's corresponding resource directory. For example, when other projects reference Gluten, it retrieves the META-INF from their resource directories instead, which does not align with expectations.

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

java.lang.UnsupportedOperationException: empty.reduceLeft
        at scala.collection.mutable.ArrayBuffer.scala$collection$IndexedSeqOptimized$$super$reduceLeft(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
        at scala.collection.IndexedSeqOptimized.reduceLeft(IndexedSeqOptimized.scala:77) ~[scala-library-2.12.15.jar:?]
        at scala.collection.IndexedSeqOptimized.reduceLeft$(IndexedSeqOptimized.scala:76) ~[scala-library-2.12.15.jar:?]
        at scala.collection.mutable.ArrayBuffer.reduceLeft(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
        at scala.collection.TraversableOnce.reduce(TraversableOnce.scala:254) ~[scala-library-2.12.15.jar:?]
        at scala.collection.TraversableOnce.reduce$(TraversableOnce.scala:254) ~[scala-library-2.12.15.jar:?]
        at scala.collection.AbstractTraversable.reduce(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
        at org.apache.gluten.extension.columnar.transition.ConventionFunc$$anon$1.<init>(ConventionFunc.scala:78) ~[gluten.jar:?]
        at org.apache.gluten.extension.columnar.transition.ConventionFunc$.newOverride(ConventionFunc.scala:76) ~[gluten.jar:?]
        at org.apache.gluten.extension.columnar.transition.ConventionFunc$.create(ConventionFunc.scala:62) ~[gluten.jar:?]
        at org.apache.gluten.extension.columnar.transition.InsertTransitions.<init>(Transitions.scala:27) ~[gluten.jar:?]
        at org.apache.gluten.extension.columnar.transition.InsertTransitions$.create(Transitions.scala:71) ~[gluten.jar:?]
        at org.apache.gluten.extension.columnar.transition.Transitions$.insert(Transitions.scala:89) ~[gluten.jar:?]
        at org.apache.gluten.extension.GlutenColumnarRule.org$apache$gluten$extension$GlutenColumnarRule$$$anonfun$postColumnarTransitions$1(GlutenColumnarRule.scala:100) ~[gluten.jar:?]
        at org.apache.gluten.extension.GlutenColumnarRule$$anonfun$postColumnarTransitions$2.apply(GlutenColumnarRule.scala:85) ~[gluten.jar:?]
        at org.apache.gluten.extension.GlutenColumnarRule$$anonfun$postColumnarTransitions$2.apply(GlutenColumnarRule.scala:85) ~[gluten.jar:?]
        at org.apache.gluten.extension.injector.InjectorControl$Disabler$DisablerOps$$anon$5$$anon$7.apply(InjectorControl.scala:149) ~[gluten.jar:?]
        at org.apache.gluten.extension.injector.InjectorControl$Disabler$DisablerOps$$anon$5$$anon$7.apply(InjectorControl.scala:144) ~[gluten.jar:?]

[zhztheplayer]

@gleonSun Thanks for reporting!

Would you like to post the classpath, namely the value of java.class.path of your program? Which will make me better understand the issue. Thanks!

[gleonSun]

@zhztheplayer hi
In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar. This prevents Gluten from locating the configuration files under META-INF/gluten-components, leading to an empty Components list being loaded. Consequently, when ConventionFunc#newOverride attempts to operate on the Components list, it triggers an exception.

[zhztheplayer]

In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar

Thanks for the detailed explanation. So gluten's compiled classes and shared libraries are included directedly into xxx/kylin.jar however the component files with META-INF/gluten-components are omitted in your case? Am I understanding correctly?

[gleonSun]

In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar

Thanks for the detailed explanation. So gluten's compiled classes and shared libraries are included directedly into xxx/kylin.jar however the component files with META-INF/gluten-components are omitted in your case? Am I understanding correctly?

Exactly

[zhztheplayer]

In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar

Thanks for the detailed explanation. So gluten's compiled classes and shared libraries are included directedly into xxx/kylin.jar however the component files with META-INF/gluten-components are omitted in your case? Am I understanding correctly?

Exactly

I see. Before #8271 we use standard service file META-INF/services/org.apache.gluten.backend.Backend for backend's discovery. How did the previous way work in your case given that with current solution META-INF/gluten-components is excluded?