[FEA] Access test files in resources from Spark source code when running Spark UT

[thirtiseven]

Is your feature request related to a problem? Please describe.
Some Spark UT will try to read files in resources from Spark's code. So when we introduce Spark UT to plugin, we can't read those files directly.

For example, "SPARK-31716: inferring should handle malformed input" in RapidsJsonSuite got following error (if included):

- SPARK-31716: inferring should handle malformed input *** FAILED ***
  org.apache.spark.sql.AnalysisException: Path does not exist: file:/home/haoyangl/spark-rapids/tests/src/test/resources/test-data/malformed_utf8.json
  at org.apache.spark.sql.errors.QueryCompilationErrors$.dataPathNotExistError(QueryCompilationErrors.scala:1011)
  at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4(DataSource.scala:785)
  at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$4$adapted(DataSource.scala:782)
  at org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:372)
  at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
  at scala.util.Success.$anonfun$map$1(Try.scala:255)
  at scala.util.Success.map(Try.scala:213)
  at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
  at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
  at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
  ...

I hope we can find a way to read test files in Spark's resources so we can really test related Spark UT cases.

Describe the solution you'd like

Gluten overrides testFile

  /** Returns full path to the given file in the resource folder */
  override protected def testFile(fileName: String): String = {
    getWorkspaceFilePath("sql", "core", "src", "test", "resources").toString + "/" + fileName
  }

where getWorkspaceFilePath in Spark

/**
 * Get a Path relative to the root project. It is assumed that a spark home is set.
 */
protected final def getWorkspaceFilePath(first: String, more: String*): Path = {
  if (!(sys.props.contains("spark.test.home") || sys.env.contains("SPARK_HOME"))) {
    fail("spark.test.home or SPARK_HOME is not set.")
  }
  val sparkHome = sys.props.getOrElse("spark.test.home", sys.env("SPARK_HOME"))
  java.nio.file.Paths.get(sparkHome, first +: more: _*)
}

Gluten is leveraging the system property "spark.test.home" (check here). For running Spark UT on shim 3.x.y, Gluten will prepare the docker container with a source folder filled with 3.x.y's code.

To do the same thing, we need to set up a Spark source folder before running Spark UT CI, and noted how to set up the env when running spark UT locally.

Describe alternatives you've considered
Those resource files are also included in jars, it is possible to read them from the jars if we know where the files are located.

Additional context
copy files to plugin pr: #10864
Spark UT RapidsJsonSuite issue: #10773

[mattahrens]

Preference is to use a test jar artifact to include rather than depending on full Spark source.

[gerashegalov]

Closed by #10946