Haystack REST API not starting

[ajainvivek]

Describe the bug
I am unable to start the haystack API using docker. I am getting infinite error loop

Error message

Expected behavior

I have tried multiple versions of python image.

Increased the docker memory to 6 GB and swap to 4 GB

System:

• OS: Mac M1
• GPU/CPU: CPU
• Haystack version (commit or version number): 0.9.0

[oryx1729]

Hi @ajainvivek,

Are you using your own Dockerfile or the one provided in the repo? If you're using a custom Dockerfile, can you share the code?

Did you test if the API works without Docker?

[ajainvivek]

I haven't changed anything in the docker file.

I have commented out ray and Gaiss-cpu in the requirements file.

[oryx1729]

Hi @ajainvivek, can you try if you get the same issue with docker-compose: docker-compose pull && docker-compose up?

[ajainvivek]

I have tried both, the issue still persists. I can pair with someone in the team because the issue occurs only in Mac M1 machine.

[julian-risch]

I think you would at least need to make some small changes in docker-compose.yml. For example, the elasticsearch image needs to be changed to image: "elasticsearch:7.9.2" (default is image: "deepset/elasticsearch-game-of-thrones"). The default image uses an old elasticsearch image that is not available on ARM architecture (link, link). Did you make that change and did you make any other changes to the docker-compose.yml file @ajainvivek ?

[julian-risch]

@ajainvivek Does docker-compose up --build work on your machine or do you already see errors there during the installation of dependencies?

[julian-risch]

I can reproduce this issue on my Apple M1 machine. However, to get to the point where i see the reported error messages I first had to make changes to the dockerfile so that the tokenizers dependency is correctly installed with a rust compiler:

# install as a package
COPY setup.py requirements.txt README.md /home/user/
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
RUN pip install --upgrade pip setuptools wheel
RUN ["sh", "-c", "export PATH=$HOME/.cargo/bin:$PATH && pip install -r requirements.txt"]
RUN pip install -e .

In addition, I changed the python image to python:3.9.7-buster

[julian-risch]

I found this issue that reports the same error: googleapis/python-pubsub#414
It suggest that the problem is caused by a bug in an older versions of grpcio (older than 1.38.1). Here is the issue in the grpc repo: grpc/grpc#26279
grpcio==1.37.1 is a dependency of pymilvus: https://github.com/milvus-io/pymilvus/blob/f27df7c00a393a537f72836b1a70c8c0c3c81346/requirements.txt#L4
Unfortunately, removing milvus from haystack's requirements.txt is no workaround at the moment because it's required by an import statement in haystack's code:

haystack/haystack/document_store/__init__.py

Line 4 in 05da7f7

from haystack.document_store.milvus import MilvusDocumentStore

[julian-risch]

I got it working on Apple M1 by removing pymilvus from the requirements.txt and removing the line from haystack.document_store.milvus import MilvusDocumentStore from haystack/document_store/__init__.py.
@ajainvivek could you please check whether removing the two lines also works on your machine and report back here?
If the next release of pymilvus bumps the grpcio dependency to >=1.38.1 that step would become obsolete.

[ajainvivek]

Sure, I will have a look today😁

[julian-risch]

As an alternative, we have uploaded a haystack-m1 docker image: https://hub.docker.com/repository/docker/deepset/haystack-m1/

[julian-risch]

Hi @ajainvivek could you please give an update whether one of the suggested solutions worked for you? Either the uploaded haystack-m1 docker image or making the suggested changes to the docker file and running the listed commands? 🙂

[ZanSara]

Closing as I'm assuming the suggested solution solved the problem. If it was not the case, please feel free to re-open or open a new issue.