Shimplify: Simplify shim source code management

[gerashegalov]

Problem Statement

We currently support 14 unique Spark builds

$ mvn -q help:evaluate -pl dist \
  -Dexpression=included_buildvers \
  -PsnapshotsWithDatabricks -DforceStdout | wc -l
14

with the source code spread over 51 directories

$ ls -ald sql-plugin/src/main/* | grep -v 'scala\|java' | wc -l
51

representing version ranges with numerous exceptions

This is a proposal to eliminate hard-to-manage version-range directories.

Instead each source code file typically resides in the earliest version directory it applies to. Every shim source code file
has a special comment in the header with the explicit list of all shims the code applies to in the format TBD like JSON Lines:

/*
 {"spark":"323","delta":["210","211"], ...}
 {"spark":"330"}
*/

these headers are processed to generate symlinks under

sql-plugin/target/spark${buildver}/src

We have an option to create copies instead of symlinks but symlinks allow to immediately see the changes tracked by git.

Motivation

First, currently it's hard to see what source directory set constitutes the shim? It was easy with submodules before #3381

With this approach we can easily query git without even building:

git grep '"spark":"312"' '*.java' '*.scala'

Second, IDEs support, however often capriciously, dynamically (via build-helper:add-source) added source roots. There are way too many to chose from https://github.com/NVIDIA/spark-rapids/tree/branch-22.12/sql-plugin/src/main and we lack intuitive-enough naming to fix it manually

With this approach, we have simple per-shim source roots to fix manually in the IDE available after running the build, or after running a script with git grep.

Third, vendor patchwork distributions are really hard to squeeze into the version range scheme. See #7152.
With this approach it's easy to decide where to place a physical shim file. And update the header whatever other shims it applies to without re-engineering the build.

Task List:

• Centralize source-related properties in parent pom for consistent usage in submodules #7600
• Maven goal to convert an old-style shim to a shimple shim #7528
• Make SparkShims Scala Test name follow an easier pattern to make it harder to miss Add missing spark shim test suites  #7673
• Verify/document IntelliJ IDEA code navigation, test, debugging works
• Verify/document VS Code navigation, test, debugging works
• Add proper formatting and validation of the applicable distros comment
• Add scripts for adding a skeleton of a new shim based on a previous shim,
• Automate removing shims
• Run a full conversion across all shims Automatic conversion to shimplified directory structure [databricks] #7222

Signed-off-by: Gera Shegalov [email protected]