Version conflict in docker

[beckstefan]

This is the full traceback when calling in latest docker ocrd/all:maximum

Traceback (most recent call last):
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 567, in _build_master
    ws.require(__requires__)
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 884, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 775, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (numpy 1.19.2 (/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages), Requirement.parse('numpy<1.19.0,>=1.16.0'), {'tensorflow'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/local/sub-venv/headless-tf22/bin/ocrd-anybaseocr-crop", line 33, in <module>
    sys.exit(load_entry_point('ocrd-anybaseocr', 'console_scripts', 'ocrd-anybaseocr-crop')())
  File "/usr/local/local/sub-venv/headless-tf22/bin/ocrd-anybaseocr-crop", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 105, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/build/ocrd_anybaseocr/ocrd_anybaseocr/cli/ocrd_anybaseocr_cropping.py", line 42, in <module>
    from ocrd.decorators import ocrd_cli_options, ocrd_cli_wrap_processor
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd/__init__.py", line 17, in <module>
    from ocrd.processor.base import run_processor, run_cli, Processor
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd/processor/__init__.py", line 1, in <module>
    from .base import (
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd/processor/base.py", line 9, in <module>
    from ocrd_utils import VERSION as OCRD_VERSION, MIMETYPE_PAGE
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd_utils/__init__.py", line 71, in <module>
    from .constants import (
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd_utils/constants.py", line 4, in <module>
    from pkg_resources import get_distribution
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3238, in <module>
    @_call_aside
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3222, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3251, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 569, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 582, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 775, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (numpy 1.19.2 (/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages), Requirement.parse('numpy<1.19.0,>=1.16.0'), {'tensorflow'})

[kba]

numpy is required in a few projects

cor-asv-ann/requirements.txt
cor-asv-fst/requirements.txt
core/ocrd_utils/requirements.txt
dinglehopper/requirements.txt
ocrd_anybaseocr/requirements.txt
ocrd_calamari/requirements.txt
ocrd_keraslm/requirements.txt
ocrd_segment/requirements.txt
ocrd_wrap/requirements.txt

But none of them define upper-bounds for the version. Would introducing numpy < 1.19.0 in core fix this issue? It's a workaround but if that's what it takes to get anybaseocr running again @bertsky @stweil ?

Why did this happen now? Did tensorflow 2.2 change it's requirements?

[stweil]

My latest local installation has these numpy versions:

venv-20200925/local/sub-venv/headless-tf1/lib/python3.7/site-packages/numpy-1.19.2.dist-info
venv-20200925/local/sub-venv/headless-tf21/lib/python3.7/site-packages/numpy-1.18.5.dist-info
venv-20200925/local/sub-venv/headless-tf22/lib/python3.7/site-packages/numpy-1.19.2.dist-info
venv-20200925/local/sub-venv/headless-torch14/lib/python3.7/site-packages/numpy-1.19.2.dist-info

ocrd_pc_segmentation already has a hard requirement for numpy==1.18.5.

[stweil]

I just checked the TF versions and was surprised to see the result:

venv-20200925/local/sub-venv/headless-tf21/lib/python3.7/site-packages/tensorflow-2.3.1.dist-info
venv-20200925/local/sub-venv/headless-tf22/lib/python3.7/site-packages/tensorflow-2.1.2.dist-info

So tf21 uses the latest TensorFlow 2.3 while tf22 uses TensorFlow 2.1. That's strange and unexpected.

[stweil]

All versions of TensorFlow seem to have identical requirements for numpy:

grep numpy venv-20200925/local/sub-venv/headless-tf*/lib/python3.7/site-packages/tensorflow*.dist-info/METADATA 
venv-20200925/local/sub-venv/headless-tf1/lib/python3.7/site-packages/tensorflow_gpu-1.15.4.dist-info/METADATA:Requires-Dist: numpy (<1.19.0,>=1.16.0)
venv-20200925/local/sub-venv/headless-tf21/lib/python3.7/site-packages/tensorflow-2.3.1.dist-info/METADATA:Requires-Dist: numpy (<1.19.0,>=1.16.0)
venv-20200925/local/sub-venv/headless-tf22/lib/python3.7/site-packages/tensorflow-2.1.2.dist-info/METADATA:Requires-Dist: numpy (<1.19.0,>=1.16.0)

[stweil]

Would introducing numpy < 1.19.0 in core fix this issue?

It looks like that would be a working solution for all sub venvs with TensorFlow, although it unnecessarily enforces older versions of numpy also for the venvs without TensorFlow.

A side note: did I ever say that TensorFlow is a nightmare?

[bertsky]

So this has become OCR-D/core#620.

[bertsky]

BTW, here's a temporary workaround for existing Docker images or native venvs:

cd /usr/local # only in Docker installation
. venv/bin/activate && cd venv # only in native installation
pip install "numpy<1.19"
. local/sub-venv/headless-tf1/bin/activate && pip install "numpy<1.19"
. local/sub-venv/headless-tf21/bin/activate && pip install "numpy<1.19"
. local/sub-venv/headless-tf22/bin/activate && pip install "numpy<1.19"