Error during synchronization between servers: Unexpected end of file from server executing GET http://compreface-core:3000/status

[kanemari]

Describe the bug
Getting this error message when logging in for the first time and trying to add a service to a new application in the UI

To Reproduce
Steps to reproduce the behavior:

1. Go to localhost:8000
2. Log in
3. Create a new application
4. Click on it and wait for service synchronisation (I assume, because it has never gone further than this)
5. Error message appears

Expected behavior
I would expect that a list of services would appear - e.g. face recognition, face detection, etc

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• Browser Chrome
• Version 92.0.4515.159 (Official Build) (64-bit)

Additional context
the compreface-core docker image doesnt seem to be listening on port 3000. docker compose pulls and extracts all containers with no errors.
this is the SubCenter-ArcFace-r100-gpu custom-build

[kanemari]

Have noticed this in the docker compose logs:-

compreface-core | Respawned uWSGI worker 1 (new pid: 397) compreface-core | DAMN ! worker 1 (pid: 397) died, killed by si gnal 4 :( trying respawn ... compreface-core | Respawned uWSGI worker 1 (new pid: 405) compreface-core | DAMN ! worker 1 (pid: 405) died, killed by si gnal 4 :( trying respawn ... compreface-core | worker respawning too fast !!! i have to slee p a bit (2 seconds)...

doesnt really give any details as to why?

have checked and although my CPU is old, it is supposed to support avx
$ lscpu | grep avx Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibrs ibpb stibp tsc_adjust arat md_clear flush_l1d arch_capabilities $

[pospielov]

InsightFace library requires AVX2 instructions...
https://github.com/exadel-inc/CompreFace/tree/master/custom-builds
We mentioned it here. Sorry, but your CPU supports only the default FaceNet version

[kanemari]

hmm thats a shame - had thought the majority of the work would be done by the GPU so it wouldnt matter so much. is there any way to reduce the requirement to avx only? unsupported of course, but would rather try that than give up entirely. the CPU isnt good enough to process the load by itself, so I need something that supports GPU without requiring avx2.

[pospielov]

I have two ideas:

1. Try to build compreface-core with intel optimization:
   https://github.com/exadel-inc/CompreFace/tree/master/embedding-calculator#optimization
   It could be possible that Intel MKL can work without AVX, but I'm not sure
2. Try to build compreface-core with mxnet without AVX. You can set it here:
   https://github.com/exadel-inc/CompreFace/blob/master/embedding-calculator/src/services/facescan/plugins/insightface/__init__.py

[kanemari]

first option worked - took a bit of messing around in docker renaming the container and getting it into the correct host network for the URL from within compreface-ui to resolve to compreface-core properly, but its running successfully and reporting available services so far. am running facenet with GPU support - insightface wouldnt run, not sure why but will try again later.

thanks for your help!

[mczarsty]

hi @kanemari! i am one of the designers for CompreFace. i am currently working on a ux survey - i'd love to chat quickly about your thoughts on where we could potentially improve CompreFace, if you are interested? if you are i can set up a skype or whatsapp chat :)

[milkplus27]

first option worked - took a bit of messing around in docker renaming the container and getting it into the correct host network for the URL from within compreface-ui to resolve to compreface-core properly, but its running successfully and reporting available services so far. am running facenet with GPU support - insightface wouldnt run, not sure why but will try again later.

thanks for your help!

hi @kanemari i'm encountering the same issue and was wondering if you could explain how you managed to get Compareface to work? I'm not particularly well versed in Docker, but perhaps if you would be able to share your compose file I might be able to figure it out. thanks :)

[kanemari]

hi @kanemari i'm encountering the same issue and was wondering if you could explain how you managed to get Compareface to work? I'm not particularly well versed in Docker, but perhaps if you would be able to share your compose file I might be able to figure it out. thanks :)

Hi @milkplus27 - I cheated and installed portainer so I could do all the docker management through a UI. But basically, I dont have a compose file. the docker build commands in the compreface doco will create a new docker container for compreface-core (that is named embedding-calculator) that is detached from the other compreface components. I essentially removed the 'standard' existing compreface-core container, renamed the embedding-calculator container to compreface-core, and added that container into the existing compreface network bridge that all the other containers were a member of (which is set up as part of the standard install).

there may be an easier way, but this worked for me. I was limited to the standard facenet and couldnt compile insightface because I am missing avx2 support. tried using the noavx version of mxnet, which then complained about other missing dependencies, and I gave up. if you can work that out please share :)

[pospielov]

@kanemari do you want to participate in hacktoberfest? :)
#607
I have a great task for you :)

[kanemari]

@pospielov I'll bite. what task would that be?

[pospielov]

Here is the link:
#607
Just write any comment in there, so I'll be able to assign an issue to you
And if you have any questions you also are free to write them into that issue

[Dodoooh]

As info, I use Proxmox to virtualize my Docker instance.

Here it is mandatory to set the CPU Type to Host, otherwise I get the same error message.

https://forum.proxmox.com/threads/avx2-and-avx-flags-on-vm.87808/

[funzie19]

I have two ideas:

1. Try to build compreface-core with intel optimization:
   https://github.com/exadel-inc/CompreFace/tree/master/embedding-calculator#optimization
   It could be possible that Intel MKL can work without AVX, but I'm not sure
2. Try to build compreface-core with mxnet without AVX. You can set it here:
   https://github.com/exadel-inc/CompreFace/blob/master/embedding-calculator/src/services/facescan/plugins/insightface/__init__.py

From my understanding option 1 worked for @kanemari because he has AVX instructions, just not AVX2. I've been trying option 2 since my CPU doesn't have AVX or AVX2. I haven't have much success yet. But will keep trying. Just wanted to add to the discussion.

[felipecrs]

Also tried to build with the mxnet-noavx, but no luck. I installed the pip dependency, and set the __init.py__ as follows:

from src.services.facescan.plugins.dependencies import get_mxnet


requirements = get_mxnet() + ('mxnet-noavx==2.0.3',)

Was this what I was supposed to do?

EDIT: nevermind, I think I understood. Will make some tests here.

[felipecrs]

Well, I built the containers with mxnet-noavx:

docker pull felipecrs/compreface-core:0.6.1-noavx-facenet
docker pull felipecrs/compreface:0.6.1-noavx

But I don't even know what facenet is or if I should be using it. I'm using because that's what make build-images-cpu produces. Any advice?

Anyway, in my HA using the addon, configured to use my image, it didn't work (Intel J4125 Gemini Lake - no AVX or AVX2). But it still fails:

In the container logs I see frequently:

2021-12-16 14:04:25,931 INFO spawned: 'compreface-core' with pid 377
[uWSGI] getting INI configuration from uwsgi.ini
2021-12-16 14:04:25,940 INFO success: compreface-core entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
*** Starting uWSGI 2.0.19 (64bit) on [Thu Dec 16 14:04:25 2021] ***
compiled with version: 10.2.1 20210110 on 16 December 2021 15:33:39
os: Linux-5.10.83 #1 SMP Fri Dec 3 16:43:02 UTC 2021
nodename: local-compreface
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /app/ml
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
setgid() to 33
setuid() to 33
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 0.0.0.0:3000 fd 3
Python version: 3.7.12 (default, Dec  3 2021, 02:42:50)  [GCC 10.2.1 20210110]
Python main interpreter initialized at 0x556f69584b70
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 218712 bytes (213 KB) for 2 cores
*** Operational MODE: preforking ***
2021-12-16 14:04:28,230 INFO exited: compreface-core (terminated by SIGILL (core dumped); not expected)

For testing purpose, this was how I configured the deps:

[pospielov]

CompreFace could be built based on two face recognition libraries: FaceNet and InsightFace.
By default, FaceNet is based on Tensorflow and requires AVX
By default, InsightFace is based on MXNet and requires AVX2
If you want to build CompreFace with a changed MXNet library, it makes sense to build CompreFace with all InsightFace plugins:
https://github.com/exadel-inc/CompreFace/tree/master/embedding-calculator#plugins

[felipecrs]

That's interesting. Thanks a lot for the response. I think I lost a bit of the interest in trying to build it myself, but I will keep watching this and the other issue in case someone makes a finding.

[felipecrs]

@pospielov is there any chance you can help to build a Dockerfile variant with mxnet-noavx with CompreFace configured with the correct libraries for such?

It seems that it would help a lot of people...

[pospielov]

Sorry, we are now focused on adding support for new GPU generation.
As mxnet is not supported anymore (the last release was May 2022), we need to migrate to another deep-learning library.
As soon as we migrate, we will see its capabilities and restrictions.

[felipecrs]

Got it. Thanks for the answer anyway.

[tcpipchip]

if i put the COMPREFACE on AMAZON (VIRTUAL PC), will i have problems ?

[pospielov]

There shouldn't be problems with ec2 instances. At least with x86_64 CPU
https://aws.amazon.com/marketplace/pp/prodview-4dhcez4e7nth6
We even have a simple way to test it. But we haven't updated the version yet.