pipenv install fails #68

oindrillac · 2021-04-13T00:11:26Z

Describe the bug
Pipenv install of hatesonar and scikit-learn==0.20.3 fails.

To Reproduce

Trying out similar workflow from #51 (comment) fails.

Spawn up JupyterHub mailing-list-analysis-toolkit image

$ git rev-parse --short  HEAD
f911c7b

$ pipenv lock --pre
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Locking [dev-packages] dependencies…
✔ Success!
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (0f8e7c)!

$ pipenv install --pre scikit-learn==0.20.3
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing scikit-learn==0.20.3…
Adding scikit-learn to Pipfile's [packages]…
✔ Installation Succeeded
Pipfile.lock (0897b7) out of date, updating to (0f8e7c)…
Locking [dev-packages] dependencies…
✔ Success!
Locking [packages] dependencies…
✘ Locking Failed!
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/resolver.py", line 69, in resolve
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version thatmatches black
[pipenv.exceptions.ResolutionFailure]:       Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches black
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1,18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies.
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/opt/app-root/lib/python3.6/site-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version thatmatches black
[pipenv.exceptions.ResolutionFailure]:       Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches black
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1,18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies.

@tumido any recommendations?

cc: @cdolfi
related to : #54

The text was updated successfully, but these errors were encountered:

MichaelClifford · 2021-04-15T15:50:43Z

@oindrillac why are you forcing scikit-learn==0.20.3 I'm trying to run this on the s2i-elyra-v6 image and it looks like both the language packages, hatesonar and flair require newer versions of sk-learn

MichaelClifford · 2021-04-15T15:53:43Z

By not specifying the versions for hatesonar or scikit-learn in the Pipfile, running pipenv lock --pre then pipenv install seems to work.

oindrillac · 2021-04-15T15:59:04Z

@MichaelClifford hatesonar with a scikit-learn > 0.20.3 was giving this error

The problem above is not only with installing scikit-learn==0.20.3.

Noticed the same when I tried

pipenv install --pre hatesonar as well on top of the Pipfile on upstream.

I'm trying to run this on the s2i-elyra-v6 image

Doesnt the elyra v6 image support 3.8? The repo is on version python 3.6, and hence I've been trying this on the mailing-list image.

oindrillac · 2021-04-15T16:16:08Z

@MichaelClifford on a version greater than 0.20.3

sonar = Sonar() gives the following error

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-9-b474975b2fe6> in <module>
----> 1 sonar = Sonar()

/opt/app-root/lib/python3.6/site-packages/hatesonar/api.py in __init__(self)
     19         model_file = os.path.join(BASE_DIR, 'model.joblib')
     20         preprocessor_file = os.path.join(BASE_DIR, 'preprocess.joblib')
---> 21         self.estimator = joblib.load(model_file)
     22         self.preprocessor = joblib.load(preprocessor_file)
     23 

/opt/app-root/lib/python3.6/site-packages/joblib/numpy_pickle.py in load(filename, mmap_mode)
    583                     return load_compatibility(fobj)
    584 
--> 585                 obj = _unpickle(fobj, filename, mmap_mode)
    586     return obj

/opt/app-root/lib/python3.6/site-packages/joblib/numpy_pickle.py in _unpickle(fobj, filename, mmap_mode)
    502     obj = None
    503     try:
--> 504         obj = unpickler.load()
    505         if unpickler.compat_mode:
    506             warnings.warn("The file '%s' has been generated with a "

/usr/lib64/python3.6/pickle.py in load(self)
   1048                     raise EOFError
   1049                 assert isinstance(key, bytes_types)
-> 1050                 dispatch[key[0]](self)
   1051         except _Stop as stopinst:
   1052             return stopinst.value

/usr/lib64/python3.6/pickle.py in load_global(self)
   1336         module = self.readline()[:-1].decode("utf-8")
   1337         name = self.readline()[:-1].decode("utf-8")
-> 1338         klass = self.find_class(module, name)
   1339         self.append(klass)
   1340     dispatch[GLOBAL[0]] = load_global

/usr/lib64/python3.6/pickle.py in find_class(self, module, name)
   1386             elif module in _compat_pickle.IMPORT_MAPPING:
   1387                 module = _compat_pickle.IMPORT_MAPPING[module]
-> 1388         __import__(module, level=0)
   1389         if self.proto >= 4:
   1390             return _getattribute(sys.modules[module], name)[0]

ModuleNotFoundError: No module named 'sklearn.linear_model.logistic'

oindrillac · 2021-04-15T16:22:14Z

since the hatesonar approach is experimental and is most likely not the final approach, is it a good idea not to track the dependencies in the pipfile and lock

chauhankaranraj · 2021-04-15T18:24:58Z

ERROR: ERROR: Could not find a version that matches black
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1,18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies.

@oindrillac From the error message, it looks like the issue is not hatesonar or sklearn, but rather black. What happens when you explicitly mention black version in the pipfile like this:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
ipython = "*"

[packages]
pandas = "*"
boto3 = "*"
wget = "*"
python-dotenv = "*"
sklearn = "*"
papermill = "*"
ibm-cloud-sdk-core = "*"
ibm-watson = "*"
scikit-learn = "==0.20.3"
hatesonar = "*"
black = "==20.8b1"

[requires]
python_version = "3.6"

[pipenv]
allow_prereleases = false

[thoth.allow_prereleases]
black = true

The above works fine for me on operate-first JH, on the mailing-list-analysis-toolkit image.

oindrillac · 2021-04-21T15:04:27Z

@chauhankaranraj thanks for the tip

black seems to stop throwing an error after this.

however, i think the scikit-learn version is still throws a conflict when flair is added

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
ipython = "*"

[packages]
pandas = "*"
boto3 = "*"
wget = "*"
python-dotenv = "*"
sklearn = "*"
papermill = "*"
ibm-cloud-sdk-core = "*"
ibm-watson = "*"
flair = "*"
scikit-learn = "==0.20.3"
hatesonar = "*"
black = "==20.8b1"

[requires]
python_version = "3.6"

[pipenv]
allow_prereleases = false

[thoth.allow_prereleases]
black = true

getting the version mismatch issue

✘ Locking Failed! 
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 702, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 684, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 1395, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 833, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches scikit-learn==0.20.3,>=0.19.1,>=0.21.3 (from -r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 4))
Tried: 0.9, 0.10, 0.11, 0.12, 0.12.1, 0.13, 0.13.1, 0.14, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.17, 0.17.1, 0.18, 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.21.1, 0.21.2, 0.21.3, 0.22, 0.22, 0.22.1, 0.22.1, 0.22.2, 0.22.2.post1, 0.22.2.post1, 0.23.0, 0.23.0, 0.23.1, 0.23.1, 0.23.2, 0.23.2, 0.24.0, 0.24.0, 0.24.1, 0.24.1, 0.24.1
Skipped pre-versions: 0.15.0b1, 0.15.0b2, 0.16b1, 0.17b1, 0.19b2, 0.20rc1, 0.21rc2, 0.22rc2.post1, 0.22rc3, 0.22rc3, 0.23.0rc1, 0.23.0rc1, 0.24.dev0, 0.24.dev0, 0.24.0rc1, 0.24.0rc1
There are incompatible versions in the resolved dependencies:
  scikit-learn==0.20.3 (from -r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 4))
  scikit-learn (from sklearn==0.0->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 8))
  scikit-learn>=0.19.1 (from hatesonar==0.0.7->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 9))
  scikit-learn>=0.21.3 (from flair==0.8.0.post1->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 3))

and I am not sure how we can overcome this other than not tracking the scikit-learn pinned dependency in the Pipfile.

MichaelClifford · 2021-04-29T12:09:38Z

@pacospace do you think we could mitigate this issue by using "overlays" and having notebook specific dependencies? If so, we should look into using those here. This is a good first use case for a Elyra/KFpipeline that actually requires unique runtime images for individual notebooks. How does the aicoe-ci/build-check handle repo's with out a global Pipfile?

pacospace · 2021-04-29T12:39:32Z

@pacospace do you think we could mitigate this issue by using "overlays" and having notebook specific dependencies? If so, we should look into using those here. This is a good first use case for a Elyra/KFpipeline that actually requires unique runtime images for individual notebooks. How does the aicoe-ci/build-check handle repo's with out a global Pipfile?

Thanks @MichaelClifford, I agree could be a very nice use case.

The tutorial we have shows how the repo will look like: https://github.com/thoth-station/elyra-aidevsecops-tutorial

Basically each notebook will have its own kernel and overlays/ repo matches kernel name selected for the notebook. If you use jupyterlab-requirements all kernels and overlays directory will be updated automatically. And the dependecies and kernel name will be present in notebook metadata as well.

So steps are:

for each notebook/step have a software stack in overlays/ directory: https://github.com/thoth-station/elyra-aidevsecops-tutorial/tree/master/overlays
update .aicoe.yaml to use overlays builds, you can have a look here: https://github.com/thoth-station/elyra-aidevsecops-tutorial/blob/master/.aicoe-ci.yaml

Then when you open a release the aicoe-ci pipeline will do the magic and each overlays will have an image on your registry depending on the name you specify in the .aicoe.yaml. Then your images are ready to be used in the Elyra Pipeline.

oindrillac mentioned this issue Apr 21, 2021

Hatesonar Analysis Notebook #54

Merged

oindrillac closed this as completed Apr 21, 2021

MichaelClifford mentioned this issue Apr 29, 2021

Apply overlays to manage notebook specific dependencies #71

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pipenv install fails #68

pipenv install fails #68

oindrillac commented Apr 13, 2021 •

edited

Loading

MichaelClifford commented Apr 15, 2021

MichaelClifford commented Apr 15, 2021

oindrillac commented Apr 15, 2021 •

edited

Loading

oindrillac commented Apr 15, 2021

oindrillac commented Apr 15, 2021

chauhankaranraj commented Apr 15, 2021

oindrillac commented Apr 21, 2021

MichaelClifford commented Apr 29, 2021

pacospace commented Apr 29, 2021

pipenv install fails #68

pipenv install fails #68

Comments

oindrillac commented Apr 13, 2021 • edited Loading

MichaelClifford commented Apr 15, 2021

MichaelClifford commented Apr 15, 2021

oindrillac commented Apr 15, 2021 • edited Loading

oindrillac commented Apr 15, 2021

oindrillac commented Apr 15, 2021

chauhankaranraj commented Apr 15, 2021

oindrillac commented Apr 21, 2021

MichaelClifford commented Apr 29, 2021

pacospace commented Apr 29, 2021

oindrillac commented Apr 13, 2021 •

edited

Loading

oindrillac commented Apr 15, 2021 •

edited

Loading