[Spark runner] Removal of Spark 2 runner support

[mosche]

The runner for Spark 2 was deprecated quite a while back in August 2022 with the release of Beam 2.41.0.

This PR finally removes support for Spark 2 (beam-runners-spark) to only maintain support for Spark 3 (beam-runners-spark-3) going forward.

Closes #25259

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[mosche]

R: @aromanenko-dev

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[codecov]

Codecov Report

Merging #25263 (86d464b) into master (88da381) will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@           Coverage Diff           @@
##           master   #25263   +/-   ##
=======================================
  Coverage   72.95%   72.95%           
=======================================
  Files         745      745           
  Lines       99191    99191           
=======================================
+ Hits        72361    72369    +8     
+ Misses      25465    25456    -9     
- Partials     1365     1366    +1     

Flag	Coverage Δ
python	82.46% <0.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...dks/python/apache_beam/options/pipeline_options.py	93.97% <ø> (ø)
...on/apache_beam/runners/portability/spark_runner.py	67.34% <0.00%> (ø)
...m/runners/portability/spark_uber_jar_job_server.py	84.32% <0.00%> (ø)
sdks/go/pkg/beam/core/metrics/dumper.go	49.20% <0.00%> (-4.77%)	⬇️
sdks/python/apache_beam/transforms/combiners.py	93.05% <0.00%> (-0.39%)	⬇️
...on/apache_beam/runners/dataflow/dataflow_runner.py	81.88% <0.00%> (+0.14%)	⬆️
...ks/python/apache_beam/runners/worker/sdk_worker.py	89.24% <0.00%> (+0.16%)	⬆️
...hon/apache_beam/runners/worker/bundle_processor.py	93.61% <0.00%> (+0.36%)	⬆️
...eam/runners/portability/fn_api_runner/execution.py	93.13% <0.00%> (+0.63%)	⬆️
...python/apache_beam/runners/worker/worker_status.py	75.33% <0.00%> (+0.66%)	⬆️
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[mosche]

R: @je-ik
R: @JozoVilcek

[mosche]

Run Python_Runners PreCommit

[mosche]

Run Python_Runners PreCommit

[mosche]

Run Java_Pulsar_IO_Direct PreCommit

[JozoVilcek]

Hey @mosche , LGTM

[mosche]

@aromanenko-dev Could you have a look, pls. Any objections moving ahead with the removal?

[aromanenko-dev]

Thanks, LGTM, just several minor questions, ptal

Before merging, please, run all related pre- and post- commit jobs to make sure that nothing is broken.

[mosche]

Run Python Spark ValidatesRunner

[mosche]

Run Java PostCommit

[mosche]

Run SQL PostCommit

[mosche]

Run Go PostCommit

[mosche]

Before merging, please, run all related pre- and post- commit jobs to make sure that nothing is broken.

@aromanenko-dev This pretty much triggered all pre jobs automatically in addition to the post jobs run above. Anything else you have in mind?

[aromanenko-dev]

@mosche Nope, I think it should be enough

[aromanenko-dev]

@mosche Could you rebase a feature branch against master instead of merging?

[mosche]

@aromanenko-dev That's what github does if resolving a conflict in the UI. It absolutely doesn't make a difference if squashed before merging.

[mosche]

Run Java_Kafka_IO_Direct PreCommit