[FEA] Shim followup - investigate version matching logic

[tgravescs]

Shimloader logic for matching versions is currently very strict and must match exactly and we throw an exception when a shim layer is not found.

Investigate if we want to relax that in case users modify the version. Like 3.0.0.1 but its really compatible with 3.0.0 for instance. Do we want to allow snapshots? Generally should be ok unless we update our code and they use older snapshot.

[jlowe]

I think minimally we need a plugin config that can override the detection logic and specify the shim that should be used. This helps cover the case where the user is using a custom version of Spark that we've never tested/seen but has a chance of working with a specific shim. At that point the user is a bit on their own since we've not tested that combination, but at least they have a possible path to move forward. cc: @sameerz for his thoughts on this.

[sameerz]

@jlowe agreed we can let the plugin run on an untested version of Spark, as long as we have a message indicating we are running on a version of Spark we have not tested the plugin with.

[tgravescs]

seems like the override has been working we can close for now until becomes issue