azure-nspkg update breaks pipenv compatibility for some packages

[jafreck]

This is related to #3447. The latest change & release of azure-nspkg broke pipenv compatibility for some of the packages.

For example, for azure-mgmt-batch, here is repro:

• Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
azure-mgmt-batch = "==5.0.1"

[requires]
python_version = "3.5"

• pipenv install output:

Pipfile.lock (b83976) out of date, updating to (fb9739)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches azure-nspkg>=3.0.0
Tried: 1.0.0, 1.0.0, 2.0.0, 2.0.0
Skipped pre-versions: 1.0.0rc1, 1.0.0rc1, 1.0.0rc2, 1.0.0rc2
There are incompatible versions in the resolved dependencies.

The error indicates that somehow azure-nspkg>=3.0.0 is required, yet not found. It's likely not found since that version specifies python2 only, but I'm not sure why it is required at all.

I checked a number of the azure-mgmt packages including azure-mgmt-applicationinsights and azure-mgmt-documentdband this occurred behavior occurred for those packages as well.

Installing using pip seems to work for python3. I can see that azure-nspkg==2.0.0 is pulled. I'm not sure why it is being installed considering it was not specified in the azure-mgmt-batch setup.py. Is it being injected somehow?

Here is the output from a pip install in a clean virtualenv:

• python3.5 -m venv env && source env/bin/activate && pip install --upgrade pip==18.0.0 && pip install azure-mgmt-batch==5.0.1:

Collecting pip==18.0.0
  Using cached https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Uninstalling pip-8.1.1:
      Successfully uninstalled pip-8.1.1
Successfully installed pip-18.0
Collecting azure-mgmt-batch==5.0.1
  Using cached https://files.pythonhosted.org/packages/97/81/a9eb3fd2ab070159105b4cfe9640c24410ac8195286729d62bfdf871de94/azure_mgmt_batch-5.0.1-py2.py3-none-any.whl
Collecting azure-mgmt-nspkg>=2.0.0 (from azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/c9/e3/48b98f929290b0cd0aaa4707ecabb8aaeb267e8ef628af907f1d1c506ec7/azure_mgmt_nspkg-2.0.0-py2.py3-none-any.whl
Collecting msrestazure<2.0.0,>=0.4.27 (from azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/a4/48/cfa54772365b4d0b8f3e7af16354db768b0317b3273265b37c14f71b705b/msrestazure-0.5.0-py2.py3-none-any.whl
Collecting azure-common~=1.1 (from azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/ac/d3/055ce7ad06459a415ff9ca210e04c6cbb51bd6564815b7c8ac34bf5a1c39/azure_common-1.1.16-py2.py3-none-any.whl
Collecting azure-nspkg>=2.0.0 (from azure-mgmt-nspkg>=2.0.0->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/cd/a0/76ca6659ae9afd7567fdbb5b9c85e9c9a0b48238cfcacd92525484408f66/azure_nspkg-2.0.0-py2.py3-none-any.whl
Collecting msrest<2.0.0,>=0.4.28 (from msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/ad/2a/37b2481fa1a7cd727c07d20cbcb0a3f9bf9dd994930f44dd9770df15c28d/msrest-0.5.5-py2.py3-none-any.whl
Collecting adal<2.0.0,>=0.6.0 (from msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/15/2b/8f674c2a20bb2a55f8f1c8fb7a458c9b513409b2cfc42f73e4cbc1ee757e/adal-1.1.0-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from msrest<2.0.0,>=0.4.28->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl
Collecting requests-oauthlib>=0.5.0 (from msrest<2.0.0,>=0.4.28->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/94/e7/c250d122992e1561690d9c0f7856dadb79d61fd4bdd0e598087dce607f6c/requests_oauthlib-1.0.0-py2.py3-none-any.whl
Collecting requests~=2.16 (from msrest<2.0.0,>=0.4.28->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl
Collecting isodate>=0.6.0 (from msrest<2.0.0,>=0.4.28->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl
Collecting cryptography>=1.1.0 (from adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/59/32/92cade62c645756a83598edf56289e9b19aae5370642a7ce690cd06bc72f/cryptography-2.3.1-cp34-abi3-manylinux1_x86_64.whl
Collecting PyJWT>=1.0.0 (from adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/93/d1/3378cc8184a6524dc92993090ee8b4c03847c567e298305d6cf86987e005/PyJWT-1.6.4-py2.py3-none-any.whl
Collecting python-dateutil>=2.1.0 (from adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl
Collecting oauthlib>=0.6.2 (from requests-oauthlib>=0.5.0->msrest<2.0.0,>=0.4.28->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/e6/d1/ddd9cfea3e736399b97ded5c2dd62d1322adef4a72d816f1ed1049d6a179/oauthlib-2.1.0-py2.py3-none-any.whl
Collecting idna<2.8,>=2.5 (from requests~=2.16->msrest<2.0.0,>=0.4.28->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests~=2.16->msrest<2.0.0,>=0.4.28->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3<1.24,>=1.21.1 (from requests~=2.16->msrest<2.0.0,>=0.4.28->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl
Collecting six (from isodate>=0.6.0->msrest<2.0.0,>=0.4.28->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.7 (from cryptography>=1.1.0->adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/59/cc/0e1635b4951021ef35f5c92b32c865ae605fac2a19d724fb6ff99d745c81/cffi-1.11.5-cp35-cp35m-manylinux1_x86_64.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.1.0->adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography>=1.1.0->adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.27->azure-mgmt-batch==5.0.1)
  Using cached https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz
Installing collected packages: azure-nspkg, azure-mgmt-nspkg, certifi, idna, chardet, urllib3, requests, oauthlib, requests-oauthlib, six, isodate, msrest, pycparser, cffi, asn1crypto, cryptography, PyJWT, python-dateutil, adal, msrestazure, azure-common, azure-mgmt-batch
  Running setup.py install for pycparser ... done
Successfully installed PyJWT-1.6.4 adal-1.1.0 asn1crypto-0.24.0 azure-common-1.1.16 azure-mgmt-batch-5.0.1 azure-mgmt-nspkg-2.0.0 azure-nspkg-2.0.0 certifi-2018.8.24 cffi-1.11.5 chardet-3.0.4 cryptography-2.3.1 idna-2.7 isodate-0.6.0 msrest-0.5.5 msrestazure-0.5.0 oauthlib-2.1.0 pycparser-2.19 python-dateutil-2.7.3 requests-2.19.1 requests-oauthlib-1.0.0 six-1.11.0 urllib3-1.23

This output indicates that azure-mgmt-nspkg is a direct dependency of azure-mgmt-batch.

[lmazuel]

Hi @jafreck
Yes, we inject the dependency azure-mgmt-nspkg>=2.0 (we stopped that in recent packages, since one of the update of setuptools broke the system). And azure-mgmt-nspkg 2.0 has a dependency on "azure-nspkg>=2.0".

azure-mgmt-nspkg 3.0 has a dependency on azure-nspkg 3.0, BUT they are tagged to be not installable on Python 3.

So there is definitely a pipenv bug here: my guess is pipenv download azure-mgmt-nspkg 3.0 by mistake (it shouldn't, see bug #3447 ), and then this one requires azure-nspkg>=3.0 (which is not compatible with Python 3.0, hence the bug). So pipenv deduces from azure-mgmt-nspkg 3.0 that it needs azure-ngpkt==3.0, and deduces at the same time from python_requires that it can't install 3.0 (Tried: 1.0.0, 1.0.0, 2.0.0, 2.0.0)

I'm amazed honestly by the number of re-implementation of the PyPI protocol that does not respect official PEPs.... I would have not expected that at all...

Side note, I'm curious to see if Poetry is better here (they claim to have the best dependency solver, at least better than pipenv).

[lmazuel]

Released azure-nspkg 3.0.1 and azure-mgmt-nspkg 3.0.1 that does not use python_requires, but install an empty Python 3 package with no file instead.

There is too many people impacted, even if I did nothing wrong, and it's third-party issues, to just do nothing. Hopefully this fixes all issues, so closing here.

[jafreck]

Thanks for the update @lmazuel. I think I understand the issue now, but just to be clear:

• azure-nspkg 3.0.0 used a python_requires identifier inline with PEP508 to prevent installations with python3
• pipenv, despite being invoked with python3, found azure-nspkg 3.0.0 and specified it as required (as azure-mgmtnspkg>=2.0.0` was injected into the dependencies)
• pipenv later found that azure-nspkg 3.0.0 could not be used with python3 (evidenced by Tried: 1.0.0, 1.0.0, 2.0.0, 2.0.0)

So, pipenv seems to respect that PEP only partially. Probably when it finds the dependencies, it does not respect that identifier, but when it goes to actually generate the lock file & install, it does.

This seems like a major issue since pipenv is "standard tooling" under github.com/pypa. I will try to investigate this and raise it with them.

[niren777]

@lmazuel I have updated the azure library from 2.0.0a1 to 4.0.0, the azure-nspkg is also updated from 2.0.0 to 3.0.2. The packages are installed properly without any issue, but when I tried to start the celery service, celery is not starting and gives the following error. I'm using Celery 3.1.24, python 3.4.9 and django 1.8.2.

root@niren:/app# C_FAKEFORK=1 sh +x /etc/init.d/accounts_celeryd start

celery init v10.1.
Using config script: /etc/default/accounts_celeryd
celery multi v3.1.24 (Cipater)
> Starting nodes...
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.4/site-packages/celery/__main__.py", line 54, in <module>
    main()
  File "/usr/local/lib/python3.4/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/usr/local/lib/python3.4/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/usr/local/lib/python3.4/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/local/lib/python3.4/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/usr/local/lib/python3.4/site-packages/celery/bin/celery.py", line 785, in handle_argv
    return self.execute(command, argv)
  File "/usr/local/lib/python3.4/site-packages/celery/bin/celery.py", line 717, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/usr/local/lib/python3.4/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/usr/local/lib/python3.4/site-packages/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/usr/local/lib/python3.4/site-packages/celery/worker/__init__.py", line 95, in __init__
    self.app.loader.init_worker()
  File "/usr/local/lib/python3.4/site-packages/celery/loaders/base.py", line 128, in init_worker
    self.import_default_modules()
  File "/usr/local/lib/python3.4/site-packages/celery/loaders/base.py", line 116, in import_default_modules
    signals.import_modules.send(sender=self.app)
  File "/usr/local/lib/python3.4/site-packages/celery/utils/dispatch/signal.py", line 166, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/local/lib/python3.4/site-packages/amqp/utils.py", line 42, in __call__
    self.set_error_state(exc)
  File "/usr/local/lib/python3.4/site-packages/amqp/utils.py", line 39, in __call__
    **dict(self.kwargs, **kwargs) if self.kwargs else kwargs
  File "/usr/local/lib/python3.4/site-packages/celery/app/base.py", line 330, in _autodiscover_tasks
    self.loader.autodiscover_tasks(packages, related_name)
  File "/usr/local/lib/python3.4/site-packages/celery/loaders/base.py", line 252, in autodiscover_tasks
    related_name) if mod)
  File "/usr/local/lib/python3.4/site-packages/celery/loaders/base.py", line 273, in autodiscover_tasks
    return [find_related_module(pkg, related_name) for pkg in packages]
  File "/usr/local/lib/python3.4/site-packages/celery/loaders/base.py", line 273, in <listcomp>
    return [find_related_module(pkg, related_name) for pkg in packages]
  File "/usr/local/lib/python3.4/site-packages/celery/loaders/base.py", line 295, in find_related_module
    _imp.find_module(related_name, pkg_path)
  File "/usr/local/lib/python3.4/imp.py", line 271, in find_module
    "not {}".format(type(name)))
RuntimeError: 'list' must be None or a list, not <class 'str'>
	> accounts_worker@niren: * Child terminated with errorcode 1
FAILED

If I downgrade the azure-nspkg to 2.0.0(pip install azure-nspkg==2.0.0) and keep azure at 4.0.0, the celery service is starting without any issue. But I'm not sure if this is a proper way to handle.