[BUG] Migrate AzureML SDK from v1 to v2

[SimonYansenZhao]

Description

The AzureML SDK used in the testing for Recommenders is of version v1 (named azureml-core):

recommenders/.github/actions/azureml-test/action.yml

Lines 75 to 77 in 310370b

	- name: Install azureml-core and azure-cli on a GitHub hosted server
	shell: bash
	run: pip install --quiet "azureml-core>1,<2" "azure-cli>2,<3"

recommenders/tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py

Lines 42 to 50 in 310370b

	from azureml.core.authentication import AzureCliAuthentication
	from azureml.core import Workspace
	from azureml.core import Experiment
	from azureml.core.runconfig import RunConfiguration
	from azureml.core.conda_dependencies import CondaDependencies
	from azureml.core.script_run_config import ScriptRunConfig
	from azureml.core.compute import ComputeTarget, AmlCompute
	from azureml.core.compute_target import ComputeTargetException
	from azureml.core.workspace import WorkspaceException

Although v1 is still supported, It's better to migrate to version v2 (named azure-ai-ml). See Upgrade to v2 -- Azure Machine Learning

In which platform does it happen?

Azure Machine Learning

How do we replicate the issue?

The SDK is used in all tests for Recommenders

Expected behavior (i.e. solution)

AzureML SDK needs to be upgraded from v1 to v2, and all code in tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py needs to be rewritten using v2.

[SimonYansenZhao]

In addition, an unresolved issue with the AzureML SDK v1 affects the upgrade of Recommenders is mentioned at #1937 (comment).

[SimonYansenZhao]

The following items need to be considered:

• Review test readme and define best practices #2130 (comment)
• Support of Python 3.12 #2098 (comment)

[daviddavo]

It EOLs in September 2025. While more than a year remains, perhaps migrating to v2 would make Python 3.12 support easier too.