dependency error in repo

[oindrillac]

Describe the bug
there is some mismatch in dependency versions which causes pipenv install to fail

To Reproduce
Steps to reproduce the behavior:

1. clone repo : git clone [email protected]:aicoe-aiops/mailing-list-analysis-toolkit.git
2. run pipenv install .
3. gives following error

Gives

[1000600000@jupyterhub-nb-ochatter-40redhat-2ecom mailing-list-analysis-toolkit]$ pipenv install .
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing .…
Adding src to Pipfile's [packages]…
✔ Installation Succeeded
Pipfile.lock (a864ef) out of date, updating to (66307a)…
Locking [dev-packages] dependencies…
✔ Success!
Locking [packages] dependencies…
✘ Locking Failed!
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/resolver.py", line 69, in resolve
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches black
[pipenv.exceptions.ResolutionFailure]:       Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3,18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches black
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1,18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies.
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches black
[pipenv.exceptions.ResolutionFailure]:       Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3,18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches black
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1,18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies.

Expected behavior
pipenv install . shyould succesfully install all dependencies without any error

[tumido]

@oindrillac the error you see is because you're trying to install a new "local" package (see the . at the end of pipenv install . command you're executing).

To install all the dependencies from the lock file you should run pipenv install instead (without the .). This command works just fine:

$ git rev-parse --short HEAD
3e4b84d

$ pipenv install
Installing dependencies from Pipfile.lock (66307a)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 29/29 — 00:00:11
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Adding the local package to dependencies is not recommended, since it may lead to multitude of problems, from being unable to use thamos instead of pipenv, to pipenv cache bloating, or locking problems you're facing in here... If you need to use the local package, please import it directly and don't lock it as a dependency please.

[goern]

/sig docs
/kind question

@fridex @pacospace

[fridex]

Related: thoth-station/thoth-application#816

[oindrillac]

@tumido thanks for the advice, I was using the commands interchangeably. When I try pipenv install, I am still getting an error

sh-4.2$ git rev-parse --short HEAD
79f823f
sh-4.2$ pipenv install
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment andcreate its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Pipfile.lock (66307a) out of date, updating to (8a2567)...
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/resolver.py", line 709, in resolve_packages
[ResolutionFailure]:       requirements_dir=requirements_dir,
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/resolver.py", line 692, in resolve
[ResolutionFailure]:       req_dir=requirements_dir
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 1403, in resolve_deps
[ResolutionFailure]:       req_dir=req_dir,
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 833, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches black (from papermill==2.3.2->-r /tmp/pipenvnwdiqy8prequirements/pipenv-3lht5csv-constraints.txt (line 4))
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2,18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies:

on running pipenv lock --clear I get the following error

sh-4.2$ pipenv lock --clear
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment andcreate its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/resolver.py", line 709, in resolve_packages
[ResolutionFailure]:       requirements_dir=requirements_dir,
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/resolver.py", line 692, in resolve
[ResolutionFailure]:       req_dir=requirements_dir
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 1403, in resolve_deps
[ResolutionFailure]:       req_dir=req_dir,
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 833, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches black (from papermill==2.3.2->-r /tmp/pipenvwbhwseoyrequirements/pipenv-mxm60r4y-constraints.txt (line 3))
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2,18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies:

[oindrillac]

pipenv install --skip-lock works for me

sh-4.2$ pipenv install --skip-lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment andcreate its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing dependencies from Pipfile...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00

[tumido]

I was using the commands interchangeably

The commands are not interchangible at all though.

1. pipenv install . is trying to add and install a new dependency at the local path .. This command is editing both Pipfile and Pipfile.lock.
2. pipenv install is ensuring the dependencies listed in the lock file are present. If there's no lock file, it will generate it for you. This command is not editing the Pipfile, though it may edit the Pipfile.lock if it doesn't exist or in case the hash in this file doesn't match the Pipfile.

If you run the pipenv install . first, it's gonna add the local package to the Pipfile as its first action and then tries to generate the lock file. Which fails, because of transitive dependency on a package pre-release (see the thoth-station/thoth-application#816, #36 issues). Then, any consecutive pipenv install call (without .) fails as well, because the Pipfile was already changed and it needs to re-lock the lock file.

Please reset/checkout your Pipfile back to the master branch and run pipenv install without . only. This will install the dependencies for you.

When you want to set up a project workspace, you want to install the dependencies prescribed in the lock file. You should never be locking or re-locking dependencies in such case. Locking should happen only if the dependency graph changes somehow. The fact that you are trying to lock the Pipfile again means something went wrong and you're no more installing the original set of dependencies.

See here, I'm running it all from scratch:

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

$ git rev-parse --short HEAD
79f823f

$ pipenv install
Creating a virtualenv for this project...
Pipfile: /home/tcoufal/Programming/AI-CoE/aiops/mailing-list-analysis-toolkit/Pipfile
Using /home/tcoufal/.pyenv/versions/3.8.6/bin/python3.8 (3.8.6) to create virtualenv...
⠧ Creating virtual environment...created virtual environment CPython3.8.6.final.0-64 in 413ms
  creator CPython3Posix(dest=/home/tcoufal/.local/share/virtualenvs/mailing-list-analysis-toolkit-fOywof3C, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/tcoufal/.local/share/virtualenv)
    added seed packages: pip==20.3.3, setuptools==51.1.1, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/tcoufal/.local/share/virtualenvs/mailing-list-analysis-toolkit-fOywof3C
Installing dependencies from Pipfile.lock (66307a)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 88/88 — 00:01:23
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

I'm running on clean master branch, install the dependencies and end up with a clean master branch again. And that's how it's supposed to behave.

[oindrillac]

Thanks for saving the day @tumido 🙌 that makes perfect sense and on a clean master, this works for me.