Upgrade databricks provider dependency

[dcmshi]

continuation of #42626
closes: #39274
suppress: #42626
patching some of the mypy test failures

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: [email protected]
  Slack: https://s.apache.org/airflow-slack

[dcmshi]

provider checks are good locally

hatch run pre-commit run mypy-providers --all-files

[dcmshi]

reran with manual mypy-providers check

hatch run pre-commit run --color always --verbose --hook-stage manual  mypy-providers --all-files

[dcmshi]

just syncing my fork, but the manual checks were passing before this so I think this should be ready for review?

[potiuk]

Approved the workflow: 👀

[dcmshi]

sorry I think I needed to rebase again, there were quite a few test failures in the test_backfill_endpoint module, but I think not related to my changes, I do see it was removed in this PR: https://github.com/apache/airflow/pull/43649/files . Could you please help me trigger checks whenever you get chance again 🙇

CI run test failures: https://github.com/apache/airflow/actions/runs/11671655535/job/32506671351?pr=43272

[potiuk]

Still there seem to be databricks imports in snowflake ?

[potiuk]

You need to rebase, rebuild image and it should work then.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.