Python package issues: proposed resolution - upgrade to Python 3.8

[lintool]

Currently, we've standardized on Python 3.6, with the rationale to be as generous as possible in supporting older versions of Python. However, this is causing issues in the dependency chain.

Here are two related issues: #907 and #911

I just upgraded my mac to macOS 12.1 Monterey and had to rebuild my environment. I came across this issue when running unit tests:

    from . import _swigfaiss
ImportError: dlopen(/Users/jimmylin/opt/anaconda3/envs/python36/lib/python3.6/site-packages/faiss/_swigfaiss.so, 0x0002): symbol not found in flat namespace '_mkl_blas_cgemm3m'

This is the same issue reported in #907

I can also confirm that basing conda off Python 3.8 addresses the issue.

Here's my minimal env setup to get unit tests to pass:

conda create -n python38 python=3.8
pip install -r requirements.txt

pip install torch==1.8.1 torchvision==0.9.1 torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
conda install faiss-cpu -c pytorch

So, my proposal is to make Python 3.8 the supported Python version for Pyserini (e.g., forcing everyone to use this version).

Thoughts?

[tyao-t]

I think that works. Why not?

[MXueguang]

So, my proposal is to make Python 3.8 the supported Python version for Pyserini (e.g., forcing everyone to use this version).

I usually use Python 3.8 as my python environment for Pyserini etc and it work well.
I think it make sense to set 3.8 as supported python version.

p.s. maybe not that important, but can anyone confirm if python3.7 has same error as 3.6 or not?
@tyao-t @lingwei-gu

[tyao-t]

I can give it a try but I think @lintool was determined to use 3.8 as the version. I am busy with a few things recently as well. @lingwei-gu if you do have time, you can consider giving it a try.