[aiohttp==3.8.0] pip cannot install with `--no-index` #6228

addyess · 2021-11-02T17:54:26Z

Describe the bug

In this issue in setuptools project, it's shows that using

version = attr: mypackage.__version__

must import the mypackage module in order to find the version. If that module imports dependencies which are not yet installed, the install process fails.

To Reproduce

download version 3.8.0 of aiohttp and all downstream dependents
create a virtualenv env
pip install the package with --find-links pointing at the package directory containing all the dependant packages

Expected behavior

The expectation is that setuptools could install aiohttp because it lists it requirements in setup.cfg, but setuptools cannot read setup.cfg due to trying to load the package version during _parse_version

Logs/tracebacks

(venv) ➜  /tmp pip freeze
(venv) ➜  /tmp ls wheelhouse 
aiohttp-3.8.0.tar.gz        cached-property-1.5.2.tar.gz     frozenlist-1.2.0.tar.gz  Jinja2-2.10.1.tar.gz                 multidict-5.2.0.tar.gz               pbr-5.6.0.tar.gz      requests-2.26.0.tar.gz        tenacity-5.0.3.tar.gz
aiosignal-1.2.0.tar.gz      certifi-2021.10.8.tar.gz         gunicorn-20.1.0.tar.gz   loadbalancer_interface-1.1.1.tar.gz  netaddr-0.7.19.tar.gz                pip-18.1.tar.gz       setuptools-41.6.0.zip         typing_extensions-3.10.0.2.tar.gz
asynctest-0.13.0.tar.gz     charmhelpers-0.20.23.tar.gz      hvac-0.11.2.tar.gz       MarkupSafe-1.1.1.tar.gz              netifaces-0.11.0.tar.gz              psutil-5.8.0.tar.gz   setuptools_scm-1.17.0.tar.gz  urllib3-1.26.7.tar.gz
async-timeout-4.0.0.tar.gz  charms.reactive-1.4.1.tar.gz     idna-3.3.tar.gz          marshmallow-3.14.0.tar.gz            ops-1.2.0.tar.gz                     pyaml-21.10.1.tar.gz  six-1.16.0.tar.gz             wheel-0.33.6.tar.gz
attrs-21.2.0.tar.gz         charset-normalizer-2.0.7.tar.gz  idna-ssl-1.1.0.tar.gz    marshmallow-enum-1.5.1.tar.gz        ops_reactive_interface-1.0.1.tar.gz  PyYAML-5.2.tar.gz     Tempita-0.4.tar.gz            yarl-1.7.2.tar.gz
(venv) ➜  /tmp pip install --upgrade setuptools pip
Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (58.4.0)
Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.3.1)
(venv) ➜  /tmp pip install -U -f wheelhouse --no-index --no-cache-dir wheelhouse/aiohttp-3.8.0.tar.gz
Looking in links: wheelhouse
Processing ./wheelhouse/aiohttp-3.8.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /tmp/venv/bin/python /tmp/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpygjsimvg
       cwd: /tmp/pip-req-build-3l4pzn_m
  Complete output (53 lines):
  *********************
  * Accelerated build *
  *********************
  Traceback (most recent call last):
    File "/tmp/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/tmp/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 146, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 127, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 236, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 54, in <module>
      setup(**setup_kwargs)
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.9/distutils/core.py", line 121, in setup
      dist.parse_config_files()
    File "/tmp/venv/lib/python3.9/site-packages/_virtualenv.py", line 21, in parse_config_files
      result = old_parse_config_files(self, *args, **kwargs)
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 700, in parse_config_files
      parse_configuration(self, self.command_options,
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/config.py", line 121, in parse_configuration
      meta.parse()
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/config.py", line 426, in parse
      section_parser_method(section_options)
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/config.py", line 399, in parse_section
      self[name] = value
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/config.py", line 184, in __setitem__
      value = parser(value)
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/config.py", line 514, in _parse_version
      version = self._parse_attr(value, self.package_dir)
    File "/tmp/pip-build-env-46387wng/overlay/lib/python3.9/site-packages/setuptools/config.py", line 349, in _parse_attr
      module = import_module(module_name)
    File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
    File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 850, in exec_module
    File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
    File "/tmp/pip-req-build-3l4pzn_m/aiohttp/__init__.py", line 5, in <module>
      from . import hdrs as hdrs
    File "/tmp/pip-req-build-3l4pzn_m/aiohttp/hdrs.py", line 8, in <module>
      from multidict import istr
  ModuleNotFoundError: No module named 'multidict'
  ----------------------------------------
WARNING: Discarding file:///tmp/wheelhouse/aiohttp-3.8.0.tar.gz. Command errored out with exit status 1: /tmp/venv/bin/python /tmp/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpygjsimvg Check the logs for full command output.
ERROR: Command errored out with exit status 1: /tmp/venv/bin/python /tmp/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpygjsimvg Check the logs for full command output.

Python Version

$ python --version
Python 3.9.7

aiohttp Version

file:///tmp/wheelhouse/aiohttp-3.8.0.tar.gz

multidict Version

file:///tmp/wheelhouse/multidict-5.2.0.tar.gz

yarl Version

file:///tmp/wheelhouse/yarl-1.7.2.tar.gz

OS

Ubuntu 21.10

(venv) ➜  /tmp uname -a
Linux addyesshp-lnx 5.13.0-20-generic #20-Ubuntu SMP Fri Oct 15 14:21:35 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
(venv) ➜  /tmp cat /etc/os-release 
PRETTY_NAME="Ubuntu 21.10"
NAME="Ubuntu"
VERSION_ID="21.10"
VERSION="21.10 (Impish Indri)"
VERSION_CODENAME=impish
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=impish

Related component

Client

Additional context

This first fails in version 3.8.0 with the switch to defining more bits and bobs in setup.cfg.

The same process works fine with 3.7.4.post

Code of Conduct

I agree to follow the aio-libs Code of Conduct

The text was updated successfully, but these errors were encountered:

addyess · 2021-11-02T18:12:45Z

Even your projects own linter proves my point:
https://github.com/aio-libs/aiohttp/runs/4083895995?check_suite_focus=true

webknjaz · 2021-11-02T19:26:57Z

I'll look into it tomorrow, it's probably not directly related to --no-index and one issue has been fixed earlier today by declaring the PEP 518 build section. The CI failure is not necessarily related, though. @asvetlov texted me a few hours ago saying he'd broken the deps in the CI and will fix them tomorrow so that may be unrelated.
Meanwhile, could you verify the problem with the recent commit on the 3.8 branch and the latest pip?

asvetlov · 2021-11-03T12:48:24Z

CI is green now on master, 3.8, 3.9

asvetlov · 2021-11-03T12:50:23Z

@webknjaz we can return to regex-based version detection in setup.py if attr: doesn't work for us. Not a big issue from my perspective.

addyess · 2021-11-03T12:54:22Z

I was looking into how to build the same source distro as listed here on pypi but if you think you can adjust the version field

version = attr: aiohttp.__version__

so that it doesn't require an import aiohttp to get to that info -- i'd be willing to test an updated tar.gz installs.

addyess · 2021-11-03T13:36:58Z

I cloned the repo,
checked out the 3.8 branch
built a new sdist using

python setup.py sdist

copied that tar.gz to /tmp/testbed
added the now required setuptools==46.4.0 (was using 41.6.0)
installed in purepython mode

AIOHTTP_NO_EXTENSIONS=1 pip install -U -f wheelhouse --no-index --no-cache-dir wheelhouse/aiohttp-3.8.0.tar.gz

Seems to be working: https://paste.ubuntu.com/p/hHkR82hbrW/
It seems the presence of setuptools-46.4.0.zip in wheelhouse/ is enough to make this work

addyess · 2021-11-03T14:08:55Z

Its likely this commit(f8b07dd) will resolve my issue. Feel free to close if you agree

i look forward to the quick release of aiohttp==3.8.1 to unblock our builds. I could work around in the meanwhile i guess with this, but that seems unnecessary if a release is coming soon

setuptools>=46.4.0,<47.0.0

webknjaz · 2021-11-03T15:23:14Z

Its likely this commit(f8b07dd) will resolve my issue. Feel free to close if you agree

Yep, I suspected that #6205 would address this. Resolved.

3. built a new sdist using
python setup.py sdist

Don't do this. Instead, use python -m build --sdist to use PEP 517: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html.

asvetlov · 2021-11-03T15:23:24Z

Cool! Please wait for a few days, we have a list of minor bugs that should be fixed as well.

As a workaround you might try pip install --upgrade setuptools before installing aiohttp

addyess added the bug label Nov 2, 2021

addyess mentioned this issue Nov 2, 2021

addyess 6228 #6229

Closed

5 tasks

webknjaz self-assigned this Nov 2, 2021

addyess mentioned this issue Nov 2, 2021

Test that containerd config toml generates under integration tests charmed-kubernetes/charm-containerd#58

Merged

webknjaz closed this as completed Nov 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[aiohttp==3.8.0] pip cannot install with `--no-index` #6228

[aiohttp==3.8.0] pip cannot install with `--no-index` #6228

addyess commented Nov 2, 2021 •

edited

Loading

addyess commented Nov 2, 2021

webknjaz commented Nov 2, 2021

asvetlov commented Nov 3, 2021

asvetlov commented Nov 3, 2021

addyess commented Nov 3, 2021

addyess commented Nov 3, 2021 •

edited

Loading

addyess commented Nov 3, 2021

webknjaz commented Nov 3, 2021

asvetlov commented Nov 3, 2021

[aiohttp==3.8.0] pip cannot install with --no-index #6228

[aiohttp==3.8.0] pip cannot install with --no-index #6228

Comments

addyess commented Nov 2, 2021 • edited Loading

Describe the bug

To Reproduce

Expected behavior

Logs/tracebacks

Python Version

aiohttp Version

multidict Version

yarl Version

OS

Related component

Additional context

Code of Conduct

addyess commented Nov 2, 2021

webknjaz commented Nov 2, 2021

asvetlov commented Nov 3, 2021

asvetlov commented Nov 3, 2021

addyess commented Nov 3, 2021

addyess commented Nov 3, 2021 • edited Loading

addyess commented Nov 3, 2021

webknjaz commented Nov 3, 2021

asvetlov commented Nov 3, 2021

[aiohttp==3.8.0] pip cannot install with `--no-index` #6228

[aiohttp==3.8.0] pip cannot install with `--no-index` #6228

addyess commented Nov 2, 2021 •

edited

Loading

addyess commented Nov 3, 2021 •

edited

Loading