az extension add -n ml failed with Pip failed with status code 1

[ZhiliangWu]

Describe the bug

When using azure-cli from the official image (sha256:88e032ed2623fa2ce4e5cba8cba5eb095147a9d4929c477a4e2a5aa53d41600d, the installation of the ml extension failed with

An error occurred. Pip failed with status code 1. Use --debug for more information.

Related command

docker run -it --rm mcr.microsoft.com/azure-cli:latest /bin/bash
az extension add -n ml 

Errors

Default enabled including preview versions for extension installation now. Disabled in May 2024. Use '--allow-preview true' to enable it specifically if needed. Use '--allow-preview false' to install stable version only.
An error occurred. Pip failed with status code 1. Use --debug for more information.

Issue script & Debug output

running build_ext
  building 'psutil._psutil_linux' extension
  creating build/temp.linux-x86_64-cpython-311
  creating build/temp.linux-x86_64-cpython-311/psutil
  creating build/temp.linux-x86_64-cpython-311/psutil/arch
  creating build/temp.linux-x86_64-cpython-311/psutil/arch/linux
  gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=598 -DPy_LIMITED_API=0x03060000 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -DPSUTIL_LINUX=1 -I/usr/local/include/python3.11 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-cpython-311/psutil/_psutil_common.o
  psutil could not be installed from sources because gcc is not installed. Try running:
    sudo apk add gcc python3-dev musl-dev linux-headers
  error: command 'gcc' failed: No such file or directory
  error: subprocess-exited-with-error

  × Building wheel for psutil (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/local/bin/python /usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpmtbwlall
  cwd: /tmp/pip-install-1b1ouzfe/psutil_dca6a20dba644533a2cf9d19e2da704d
  Building wheel for psutil (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for psutil
Failed to build psutil
ERROR: Could not build wheels for psutil, which is required to install pyproject.toml-based projects
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 429, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for psutil, which is required to install pyproject.toml-based projects
Removed build tracker: '/tmp/pip-build-tracker-t3vv_e7w'

cli.azure.cli.core.extension.operations: Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--target', '/root/.azure/cliextensions/ml', '/tmp/tmplkup8dgc/ml-2.22.0-py3-none-any.whl', '-vv', '--disable-pip-version-check', '--no-cache-dir']' returned non-zero exit status 1.
cli.azure.cli.core.extension.operations: Pip failed so deleting anything we might have installed at /root/.azure/cliextensions/ml
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/extension/custom.py", line 16, in add_extension_cmd
    return add_extension(cli_ctx=cmd.cli_ctx, source=source, extension_name=extension_name, index_url=index_url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/extension/operations.py", line 349, in add_extension
    extension_name = _add_whl_ext(cli_ctx=cmd_cli_ctx, source=source, ext_sha256=ext_sha256,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/extension/operations.py", line 172, in _add_whl_ext
    raise CLIError('An error occurred. Pip failed with status code {}. '
knack.util.CLIError: An error occurred. Pip failed with status code 1. Use --debug for more information.

cli.azure.cli.core.azclierror: An error occurred. Pip failed with status code 1. Use --debug for more information.
az_command_data_logger: An error occurred. Pip failed with status code 1. Use --debug for more information.
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fbb895e0400>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 26.585 seconds (init: 0.293, invoke: 26.292)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3398 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/local/bin/python /usr/local/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /root/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

ml extension installed.

Environment Summary

{
"azure-cli": "2.56.0",
"azure-cli-core": "2.56.0",
"azure-cli-telemetry": "1.1.0",
"extensions": {}
}

Additional context

using the docker image

[azure-client-tools-bot-prd]

Hi @ZhiliangWu
Find similar issue #22140.

Issue title	azure-cli-ml installation fails on MacOS arm
Create time	2022-04-21
Comment number	13

---

Please confirm if this resolves your issue.

[yonzhan]

Thank you for opening this issue, we will look into it.

[ZhiliangWu]

@yonzhan Thanks for your quick reply. I also tested different tags with it, the problems occur also on 2.54.0, 2.55.0 and 2.56.0 (the latest one is 2.56.0 if I understand correctly). It starts to disappear on 2.53.1.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

[bebound]

This is related to this pr: #27567
The psutil does not provide wheel for Alpine, you need to run apk add gcc musl-dev linux-headers libffi-dev before installing ml extension.

[ZhiliangWu]

This is related to this pr: #27567 The psutil does not provide wheel for Alpine, you need to run apk add gcc musl-dev linux-headers libffi-dev before installing ml extension.

@bebound Thanks for your reply. Would this be fixed in new published images, or these steps are suggested to be done everytime we install the extension?

[bebound]

No plan to revert it. You need to run these steps manually.

[ZhiliangWu]

No plan to revert it. You need to run these steps manually.

@bebound
Any reason why there is no plan to solve this issue? IMO it makes the usage of azure-cli image in danger when one wants to use ml extension.

[bebound]

It's explained in the #27567 PR description.