Feature: Automate some of the migration steps for Spark Infrastructure v2 chart

[csun-cpointe]

Description

To smooth out the Spark Infrastructure v2 chart, we want to automate some of the migration steps for user. We don't want to remove files for users so we will leave those steps for users to manually migrate but automate the steps that require user repeatedly adding content to files.

DOD (DRAFT)

Acceptance criteria required to realize the requested feature

• Write Baton migration to remove hive-metastore-service-deploy and hive-metastore-db-deploy from the downstream project deploy pom fermenter-mda executions.
• Write Baton migration to replace the aissemble-spark-infrastructure-deploy profile with the aissemble-spark-infrastructure-deploy-v2 profile ithe downstream project deploy pom fermenter-mda executions.
• Write Baton migration script to remove the hive-metastore-db, hive-metatstore-service and spark-infrastructure argocd templates
• Write Baton migration script to remove references to hive-metastore-db and hive-metastore-service from the Tiltfile
• Write Baton migration script to update the downstream project each data-delivery pipeline's resources/apps/<pipeline>-*.yaml file to below the spakConf

sparkApp:
  spec:
    sparkConf:
      spark.hadoop.fs.s3a.endpoint: "http://s3-local:4566"
      spark.eventLog.dir: "/opt/spark/spark-events"
      spark.hive.metastore.warehouse.dir: "s3a://spark-infrastructure/warehouse"

BDD Scenario:

Feature: Migration Spark Infrastructure v2 chart

• Scenario: Migrate spark pipeline template without sparkConf configuration
• Scenario Outline: Migrate spark pipeline template with sparkConf configuration
• Scenario: Migrate spark pipeline template file indent with 4 spaces?
• Scenario: The s3a endpoint config is added to spark pipeline values-dev file
• Scenario: The event log config is added to base file if it's not present to point to local file system to all values file
• Scenario: The warehouse dir config is updated to point to local file system to all values file

Test Strategy/Script

How will this feature be verified?

References/Additional Context

A clear and concise description of any alternative solutions or features you've considered.
Add any other context, links, or screenshots about the feature request here.