Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better Error Messaging for Users when docker-compose up Fails #2399

Closed
brandenchan opened this issue Apr 11, 2022 · 2 comments
Closed

Better Error Messaging for Users when docker-compose up Fails #2399

brandenchan opened this issue Apr 11, 2022 · 2 comments
Labels
P2 Medium priority, add to the next sprint if no P1 available topic:docker topic:rest_api type:documentation Improvements on the docs type:feature New feature or request

Comments

@brandenchan
Copy link
Contributor

brandenchan commented Apr 11, 2022

Is your feature request related to a problem? Please describe.
I was running docker-compose up on a Macbook Pro. Docker Desktop was allotted 4GB of memory. The REST API would continually fail to load the ElasticsearchDocumentStore and hence also the Retriever. The Elasticsearch instance would keep starting, crashing and restarting. There was no error message as to why this was happening.

After some deep exploration with @tstadel , we found that this machine was able to start up the REST API and the Elasticsearch containers separately. But when started up together using docker-compose up, the Elasticsearch container would crash because of insufficient memory. Since the restart: on-failure is set in docker-compose.yml it would keep trying to restart.

After we changed the restart policy to:

deploy:
  restart_policy:
    condition: none

Error code 137 would be returned when ES crashed.

Calling docker container inspect haystack_elasticsearch_1 after crash might show "State"/"OOMKilled": true

Describe the solution you'd like
It would be great to have some kind of error message displayed to the user about why docker-compose up is not working in this situation. It would also be good to explain that you might need to allot more memory to Docker to ensure the containers don't crash, and that you can also change the number of workers to reduce memory consumption.

@masci
Copy link
Contributor

masci commented Nov 28, 2022

A fix like it's described in this issue is not possible: we can't deduce why one of the containers orchestrated is killed or stuck in crashloop.

What we can do is providing hardware requirements for running the demo, adding the documentation label.

@masci masci added type:documentation Improvements on the docs P2 Medium priority, add to the next sprint if no P1 available and removed topic:elasticsearch topic:document_store labels Nov 28, 2022
@masci
Copy link
Contributor

masci commented Jan 25, 2023

We've since added an healthcheck for the ES container, the crash-loop can't happen anymore and the failure would be now evident, closing.

@masci masci closed this as completed Jan 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 Medium priority, add to the next sprint if no P1 available topic:docker topic:rest_api type:documentation Improvements on the docs type:feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants