[FEA] Enhance databricks CI support

[tgravescs]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I wish the RAPIDS Accelerator for Apache Spark would [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context, code examples, or references to existing implementations about the feature request here.

Make the CI Support for Databricks more robust.

• Make sure failure handling working properly. When test fails build should once we have support to xfail the tests that have known issues.
• make sure if build stopped Databricks cluster shutdown
• Better parameterization to pass things around then hardcoded
• Push new docker image and update jenkinsfile

[tgravescs]

Note see specific comments as well: #221