Unable to start Label Studio from latest docker image due to OpenBLAS Operation not permitted error #3070

Adrian-Rasmussen · 2022-10-11T03:29:32Z

Describe the bug
When attempting to run the latest docker image, it fails to build due to an OpenBLAS error with operation not permitted. It was previously working on 1.5.0.post0.

OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 40: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 40: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
...
OpenBLAS blas_thread_init: pthread_create failed for thread 39 of 40: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
Traceback (most recent call last):
  File "/usr/local/bin/label-studio", line 33, in <module>
    sys.exit(load_entry_point('label-studio', 'console_scripts', 'label-studio')())
  File "/label-studio/label_studio/server.py", line 290, in main
    _setup_env()
  File "/label-studio/label_studio/server.py", line 40, in _setup_env
    application = get_wsgi_application()
  File "/usr/local/lib/python3.10/dist-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/usr/local/lib/python3.10/dist-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/label-studio/label_studio/core/settings/label_studio.py", line 47, in <module>
    versions = collect_versions()
  File "/label-studio/label_studio/core/utils/common.py", line 468, in collect_versions
    import label_studio_converter
  File "/usr/local/lib/python3.10/dist-packages/label_studio_converter/__init__.py", line 3, in <module>
    from .converter import Converter
  File "/usr/local/lib/python3.10/dist-packages/label_studio_converter/converter.py", line 6, in <module>
    import pandas as pd
  File "/usr/local/lib/python3.10/dist-packages/pandas/__init__.py", line 11, in <module>
    __import__(dependency)
  File "/usr/local/lib/python3.10/dist-packages/numpy/__init__.py", line 140, in <module>
    from . import core
  File "/usr/local/lib/python3.10/dist-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/usr/local/lib/python3.10/dist-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/usr/local/lib/python3.10/dist-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
KeyboardInterrupt
Exception ignored in atexit callback: <function AtexitIntegration.setup_once.<locals>._shutdown at 0x7f6166f76f80>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/integrations/atexit.py", line 62, in _shutdown
    client.close(callback=integration.callback)
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/client.py", line 449, in close
    self.flush(timeout=timeout, callback=callback)
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/client.py", line 471, in flush
    self.transport.flush(timeout=timeout, callback=callback)
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/transport.py", line 487, in flush
    self._worker.submit(lambda: self._flush_client_reports(force=True))
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/worker.py", line 113, in submit
    self._ensure_thread()
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/worker.py", line 42, in _ensure_thread
    self.start()
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/worker.py", line 70, in start
    self._thread.start()
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/integrations/threading.py", line 54, in sentry_start
    return old_start(self, *a, **kw)
  File "/usr/lib/python3.10/threading.py", line 935, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

To Reproduce
Steps to reproduce the behavior:

Run docker run -it -p 8080:8080 heartexlabs/label-studio:latest
See docker image fail to start

Expected behavior
Docker image should start up Label Studio as usual

Environment (please complete the following information):
Latest docker tag, 914038b9316d
Docker setup:

currently installed docker packages:

docker-compose/focal,focal,now 1.25.0-1 all [installed]
docker.io/now 20.10.2-0ubuntu1~20.04.3 amd64 [installed,upgradable to: 20.10.12-0ubuntu2~20.04.1]
python3-docker/focal,focal,now 4.1.0-1 all [installed,automatic]
python3-dockerpty/focal,focal,now 0.4.1-2 all [installed,automatic]


version:

Client:
Version:           20.10.2
API version:       1.41
Go version:        go1.13.8
Git commit:        20.10.2-0ubuntu1~20.04.3
Built:             Fri Jul 23 21:06:26 2021
OS/Arch:           linux/amd64
Context:           default
Experimental:      true

Server:
Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.2-0ubuntu1~20.04.3
  Built:            Fri Jul 23 19:35:35 2021
  OS/Arch:          linux/amd64
  Experimental:     false
containerd:
  Version:          1.5.2-0ubuntu1~20.04.2
  GitCommit:
runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.2
  GitCommit:
docker-init:
  Version:          0.19.0
  GitCommit:

ulimit -a from host system:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 380299
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 380299
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

The text was updated successfully, but these errors were encountered:

farioas · 2022-10-18T12:06:09Z

OpenMathLib/OpenBLAS#3403 (comment)

Try to use this command:
docker run -it -e OPENBLAS_NUM_THREADS=1 -p 8080:8080 heartexlabs/label-studio:latest

Adrian-Rasmussen · 2022-10-19T03:30:25Z

Unfortunately that doesn't work either:

$ docker run -it -e OPENBLAS_NUM_THREADS=1 -p 8081:8080 heartexlabs/label-studio:latest
root
./deploy/docker-entrypoint.sh: Looking for init scripts in /label-studio/deploy/docker-entrypoint.d/
./deploy/docker-entrypoint.sh: Launching /label-studio/deploy/docker-entrypoint.d/20-wait-for-db.sh
=> Skipping wait for database.
./deploy/docker-entrypoint.sh: Launching /label-studio/deploy/docker-entrypoint.d/30-run-db-migrations.sh
=> Skipping run db migrations.
./deploy/docker-entrypoint.sh: Configuration complete; ready for start up
=> Database and media directory: /label-studio/data
=> Static URL is set to: /static/
=> Database and media directory: /label-studio/data
=> Static URL is set to: /static/
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /pypi/label-studio/json HTTP/1.1" 200 52124

   ╔═══════════════════════════════════════╗
   ║                                       ║
   ║   Update available 1.6.0rc5 → 1.6.0   ║
   ║   Run pip install -U label-studio     ║
   ║                                       ║
   ╚═══════════════════════════════════════╝

Initializing database..
Traceback (most recent call last):
  File "/usr/local/bin/label-studio", line 33, in <module>
    sys.exit(load_entry_point('label-studio', 'console_scripts', 'label-studio')())
  File "/label-studio/label_studio/server.py", line 421, in main
    start_browser(url, input_args.no_browser)
  File "/label-studio/label_studio/core/utils/common.py", line 310, in start_browser
    threading.Timer(2.5, lambda: webbrowser.open(browser_url)).start()
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/integrations/threading.py", line 54, in sentry_start
    return old_start(self, *a, **kw)
  File "/usr/lib/python3.10/threading.py", line 935, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
Exception ignored in atexit callback: <function AtexitIntegration.setup_once.<locals>._shutdown at 0x7f15ede7af80>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/integrations/atexit.py", line 62, in _shutdown
    client.close(callback=integration.callback)
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/client.py", line 449, in close
    self.flush(timeout=timeout, callback=callback)
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/client.py", line 471, in flush
    self.transport.flush(timeout=timeout, callback=callback)
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/transport.py", line 487, in flush
    self._worker.submit(lambda: self._flush_client_reports(force=True))
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/worker.py", line 113, in submit
    self._ensure_thread()
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/worker.py", line 42, in _ensure_thread
    self.start()
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/worker.py", line 70, in start
    self._thread.start()
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/integrations/threading.py", line 54, in sentry_start
    return old_start(self, *a, **kw)
  File "/usr/lib/python3.10/threading.py", line 935, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

Adrian-Rasmussen · 2022-10-19T09:03:32Z

Updating to a newer docker version (20.10.12 in my case) has resolved the issue. It may be beneficial to add a note to the install docs with a minimum supported version

jerkhouri · 2022-10-21T09:02:45Z

In my case, the docker update didn't solve the problem.

I have to add this in the docker compose:

app:
    ...
    security_opt:
      - seccomp:unconfined
    ...

The problem was about the glibc version. see=>Forum

aixuedegege · 2023-06-02T02:20:28Z

下面的命令解决了这个问题，启动docker 容器时增加选项：--security-opt seccomp:unconfined

docker run -it -p 8080:8080 -v /data:/label-studio/data --security-opt seccomp:unconfined heartexlabs/label-studio:latest bash

然后进到容器里面用 “label-studio start” 启动

xkcode · 2023-07-20T14:15:40Z

下面的命令解决了这个问题，启动docker 容器时增加选项：--security-opt seccomp:unconfined
docker run -it -p 8080:8080 -v /data:/label-studio/data --security-opt seccomp:unconfined heartexlabs/label-studio:latest bash
然后进到容器里面用 “label-studio start” 启动

牛逼Plus

kimdw9983 · 2023-08-09T08:06:22Z

I've managed to resolve this issue by changing the Python base image to older Debian release.
In my case, 3.11.4-bookworm to 3.11.4-buster, which is compatible for Ubuntu 18.04

I think it is better to upgrade OS version to latest or use base image that compatibles older OS version rather than disabling default seccomp profile for security.

zengqingfu1442 · 2024-05-21T10:02:37Z

need to upgrade docker version to > 20.10.9, https://medium.com/nttlabs/ubuntu-21-10-and-fedora-35-do-not-work-on-docker-20-10-9-1cd439d9921

Adrian-Rasmussen assigned makseq Oct 11, 2022

makseq added problem bug or something isn't working installation labels Oct 11, 2022

Adrian-Rasmussen closed this as completed Oct 19, 2022

IJtLJZ8Rm4Yr mentioned this issue Dec 15, 2022

[misc] walk around label studio start up issue IndustryEssentials/ymir#1438

Merged

farioas mentioned this issue Jan 30, 2023

How to use docker to deploy label-studio offline? #3561

Closed

mematt mentioned this issue Feb 17, 2023

Error running docker compose up using the branch blaze_update medizininformatik-initiative/Projectathon6-miracum1#22

Closed

pfrenzel mentioned this issue May 4, 2023

ImportError: numpy.core.multiarray failed to import on 2.4.2 cvat-ai/cvat#6038

Closed

2 tasks

timothydereuse mentioned this issue Aug 2, 2023

PROD - numpy error in rodan-main container DDMAL/Rodan#1032

Closed

ajhai mentioned this issue Sep 6, 2023

[v0.0.7] Unable to get started trypromptly/LLMStack#24

Closed

vlizanae mentioned this issue Nov 28, 2023

ImportError: PyCapsule_Import could not import module "datetime" numpy/numpy#14474

Closed

dmarteau mentioned this issue Nov 29, 2023

Test with -w in docker-entrypoint.sh gives error 3liz/py-qgis-server#72

Closed

george-kuanli-peng mentioned this issue Mar 18, 2024

Upgrades to 24.02 base image templates myelintek/templates#20

Merged

marcusschiesser mentioned this issue May 30, 2024

Error - not able to run ragapp successfully ragapp/ragapp#22

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to start Label Studio from latest docker image due to OpenBLAS Operation not permitted error #3070

Unable to start Label Studio from latest docker image due to OpenBLAS Operation not permitted error #3070

Adrian-Rasmussen commented Oct 11, 2022

farioas commented Oct 18, 2022

Adrian-Rasmussen commented Oct 19, 2022

Adrian-Rasmussen commented Oct 19, 2022

jerkhouri commented Oct 21, 2022

aixuedegege commented Jun 2, 2023 •

edited

Loading

xkcode commented Jul 20, 2023

kimdw9983 commented Aug 9, 2023

zengqingfu1442 commented May 21, 2024

Unable to start Label Studio from latest docker image due to OpenBLAS Operation not permitted error #3070

Unable to start Label Studio from latest docker image due to OpenBLAS Operation not permitted error #3070

Comments

Adrian-Rasmussen commented Oct 11, 2022

farioas commented Oct 18, 2022

Adrian-Rasmussen commented Oct 19, 2022

Adrian-Rasmussen commented Oct 19, 2022

jerkhouri commented Oct 21, 2022

aixuedegege commented Jun 2, 2023 • edited Loading

xkcode commented Jul 20, 2023

kimdw9983 commented Aug 9, 2023

zengqingfu1442 commented May 21, 2024

aixuedegege commented Jun 2, 2023 •

edited

Loading