Rename databricks task type

[pingsutw]

Tracking issue

NA

Why are the changes needed?

For now, Spark tasks can only run on either k8s or Databricks, but they cannot run on both at the same time.

What changes were proposed in this pull request?

The databricks task's type (spark) should be renamed to support running both Spark and Databricks in the same workflow.

• Task: To keep backward compatible, we add a new task config (DatabricksV2), and the new task type for this config is databricks.

from flytekitplugins.spark import Databricks # Old user

from flytekitplugins.spark import DatabricksV2 as Databricks # New user

• Agent: Always register two agetns, these agent supports spark and databricks respectively.

AgentRegistry.register(DatabricksAgent())
AgentRegistry.register(DatabricksAgentV2())

How was this patch tested?

local/remote

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

NA

Docs link

NA

[codecov]

Codecov Report

Attention: Patch coverage is 20.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 76.16%. Comparing base (ccade5a) to head (17a47b8).
Report is 1 commits behind head on master.

Files	Patch %	Lines
flytekit/extend/backend/base_agent.py	20.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2574      +/-   ##
==========================================
- Coverage   76.23%   76.16%   -0.07%     
==========================================
  Files         183      184       +1     
  Lines       18745    18783      +38     
  Branches     3699     3706       +7     
==========================================
+ Hits        14290    14306      +16     
- Misses       3825     3844      +19     
- Partials      630      633       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.