[Packaging] Build MSI/Homebrew packages with dependecies resolved from requirements.txt

[fengzhou-msft]

Description

This PR finishes the Packaging deployment part for #9781 to build with requirements.*.txt files for all packages. To resolve the issue, we need to change azdev and CI to use requirements.*.txt to resolve depedencies as well. @haroldrandom for awareness.

I have also submitted the PR Azure/azure-cli-dev-tools#194 for azdev setup to use requirements file.

Testing Guide

---

This checklist is used to make sure that common guidelines for a pull request are followed.

• The PR title and description has followed the guideline in Submitting Pull Requests.

• I adhere to the Command Guidelines.

[yonzhan]

Packaging

[haroldrandom]

What do azdev need to do?

[fengzhou-msft]

What do azdev need to do?

I have submitted the PR Azure/azure-cli-dev-tools#194 in azdev. azdev setup will use requirements file to download dependencies by default. But we still provide the option with --deps setup to download dependencies based on setup.py. The azdev PR currently has failed CI because some dependencies in the requirements file is old and not compatible with dependencies of azdev. I updated them in this PR. After merging this PR, the azdev PR should pass CI.

After merging the azdev PR, we need to modify pipeline script to test both setup.py and requirements file. I'm thinking about using setup.py for python 3.8 tests and requirements file for python 3.6 since we use requirements file for all packages except pip install with this PR and some packages have a bundled python 3.6.

What do you think?

[arrownj]

should we consider remove all six dependency as we only support python 3 now ?

[fengzhou-msft]

Good point. I will work on cleaning up dependencies in another PR. Feel free to suggest any other dependencies that could be removed.

[haroldrandom]

@fengzhou-msft

What do you think?

I could be done.

I have one question. How do we keep up with the version uprgade of those dependencies? How often we update requirements.txt?

[fengzhou-msft]

@fengzhou-msft

What do you think?

I could be done.

I have one question. How do we keep up with the version uprgade of those dependencies? How often we update requirements.txt?

I think the current practice is when we need to use new features or when customers requested to do so, we update the versions.

[haroldrandom]

we can also make setup.py use requirements.txt too.
https://stackoverflow.com/questions/14399534/reference-requirements-txt-for-the-install-requires-kwarg-in-setuptools-setup-py

[jiasli]

Private wheels under /privates are copied to TEMP_SCRATCH_FOLDER first:

azure-cli/build_scripts/windows/scripts/build.cmd

Lines 50 to 52 in 868decc

	if exist %REPO_ROOT%\privates (
	copy %REPO_ROOT%\privates\*.whl %TEMP_SCRATCH_FOLDER%
	)

Then Azure CLI's wheels are also built into TEMP_SCRATCH_FOLDER:

azure-cli/build_scripts/windows/scripts/build.cmd

Lines 93 to 97 in 868decc

	for %%a in (%CLI_SRC%\azure-cli %CLI_SRC%\azure-cli-core %CLI_SRC%\azure-cli-nspkg %CLI_SRC%\azure-cli-telemetry) do (
	pushd %%a
	%BUILDING_DIR%\python.exe setup.py bdist_wheel -d %TEMP_SCRATCH_FOLDER%
	popd
	)

ALL_MODULES represents all wheels under TEMP_SCRATCH_FOLDER:

azure-cli/build_scripts/windows/scripts/build.cmd

Lines 102 to 105 in 868decc

	set ALL_MODULES=
	for %%i in (%TEMP_SCRATCH_FOLDER%\*.whl) do (
	set ALL_MODULES=!ALL_MODULES! %%i
	)

They are installed by

azure-cli/build_scripts/windows/scripts/build.cmd

Line 108 in 868decc

%BUILDING_DIR%\python.exe -m pip install --no-warn-script-location --no-cache-dir %ALL_MODULES%

As now packages are not installed from TEMP_SCRATCH_FOLDER, but requirements.py3.windows.txt, the /privates private wheel injection mechanism is broken.