DEXTR not working in GUI (waitpid returned 9)

[brixxx]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

Hi, thank you very much for this great tool.
In my installation Dextr does not work. Has anyone an idea what the problem could be?
The "Make AI Polygon"-option shows up in the GUI but using it, so drawing an intelligent polygon leads to "Could not create annotations Error: Error: Work-horse process was terminated unexpectedly (waitpid returned 9)".

Steps to Reproduce (for bugs)

I installed CVAT [1.0.0] - 2020-05-29 and also openvino to be able to use DEXTR.
For the later I downloaded l_openvino_toolkit_p_2020.2.120.tgz and put it in cvat/components/openvino.
I checked the file cvat/components/openvino/eula.cfg and it was already accepted.

Then I did: "docker-compose -f docker-compose.yml -f components/openvino/docker-compose.openvino.yml -f cvat/apps/dextr_segmentation/docker-compose.dextr.yml build"
followed by "docker-compose -f docker-compose.yml -f components/openvino/docker-compose.openvino.yml -f cvat/apps/dextr_segmentation/docker-compose.dextr.yml up -d".

My Environment

CVAT [1.0.0] - 2020-05-29
l_openvino_toolkit_p_2020.2.120.tgz
Ubuntu 18.04.4 LTS

Docker:
Client: Docker Engine - Community
Version: 19.03.10
API version: 1.40
Go version: go1.13.10
Git commit: 9424aeaee9
Built: Thu May 28 22:16:49 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.10
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 9424aeaee9
Built: Thu May 28 22:15:20 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

LOG:

2020-05-30 16:14:45,819 DEBG 'rqworker_default_0' stderr output:
INFO:rq.worker:default: cvat.apps.dextr_segmentation.views._dextr_thread(<Data: Data object (2)>, 0, [{'x': 265.6708984375, 'y': 431.490234375}, {'x': 511.24028029441797, 'y': 615.0707823753364}, {'x': 492.16693892478906, 'y': 338.5073325157173}, {'x': 648.3299213886257, 'y': 472.0207221031196}]) (dextr.create/2/kitz)

2020-05-30 16:14:46,795 DEBG 'runserver' stderr output:
[Sat May 30 16:14:46.792753 2020] [wsgi:error] [pid 705:tid 139973466679040] [remote 172.18.0.6:41832] [2020-05-30 16:14:46,791] INFO cvat.server.task_2: check dextr request for the JOB: 2 by the USER: kitz

2020-05-30 16:14:46,795 DEBG 'runserver' stderr output:
[Sat May 30 16:14:46.795903 2020] [wsgi:error] [pid 705:tid 139973466679040] [remote 172.18.0.6:41832] INFO:cvat.server.task_2:check dextr request for the JOB: 2 by the USER: kitz

2020-05-30 16:14:48,646 DEBG 'runserver' stderr output:
[Sat May 30 16:14:48.645263 2020] [wsgi:error] [pid 705:tid 139973458286336] [remote 172.18.0.6:41838] [2020-05-30 16:14:48,644] INFO cvat.server.task_2: check dextr request for the JOB: 2 by the USER: kitz
[Sat May 30 16:14:48.645414 2020] [wsgi:error] [pid 705:tid 139973458286336] [remote 172.18.0.6:41838] INFO:cvat.server.task_2:check dextr request for the JOB: 2 by the USER: kitz

2020-05-30 16:14:48,647 DEBG 'rqworker_default_0' stderr output:
WARNING:rq.worker:Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 9)    

[bsekachev]

@brixxx

Hi, could you please attach also logs of the task where you have tried to run DEXTR?

This commands could help:

docker exec cvat bash -c 'cat data/tasks/<TASK_ID>/logs/task.log' > task.log

The file task.log will be in the directory where you ran the command.

[brixxx]

Hi, thanks!

Here is the desired log:

[2020-05-30 15:40:34,390] INFO cvat.server.task_2: get repository request
[2020-05-30 15:41:48,620] INFO cvat.server.task_2: create dextr request for the JOB: 2 by the USER: kitz on the FRAME: 0
[2020-05-30 15:41:52,884] INFO cvat.server.task_2: check dextr request for the JOB: 2 by the USER: kitz
[2020-05-30 16:06:20,853] INFO cvat.server.task_2: create dextr request for the JOB: 2 by the USER: kitz on the FRAME: 0
[2020-05-30 16:06:22,001] INFO cvat.server.task_2: check dextr request for the JOB: 2 by the USER: kitz
[2020-05-30 16:06:24,659] INFO cvat.server.task_2: check dextr request for the JOB: 2 by the USER: kitz
[2020-05-30 16:14:45,738] INFO cvat.server.task_2: create dextr request for the JOB: 2 by the USER: kitz on the FRAME: 0
[2020-05-30 16:14:46,791] INFO cvat.server.task_2: check dextr request for the JOB: 2 by the USER: kitz
[2020-05-30 16:14:48,644] INFO cvat.server.task_2: check dextr request for the JOB: 2 by the USER: kitz

[bsekachev]

@brixxx

Hi, in your case process with Deep Extreme Cut is died, but from logs it isn't clear why.
I can suggest something wrong lower than python level.
Need more time to investigate and probably your help to resolve the issue.

[brixxx]

@bsekachev
Hi,
thank you very much for your effort.
If you need my help, just tell me what to do.
I have basic shell and python knowledge, but I am not familiar with docker and the development of CVAT at all.

[bsekachev]

This patch #1767 probably will resolve the issue

[brixxx]

@bsekachev
Thank you very much!

I will wait until it is merged into development branch and then give it a try.

[nmanovic]

Should be fixed by #1767